Handle differently-named xmlstarlet binary for *suse

Seems *SUSE sticks to the upstream binary name of 'xml'
for the xmlstarlet package.

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #3657 from ceph/wip-10782

pybind: fixed runtime errors with librbdpy

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3627 from tchaikov/check_output-on-centos6

ceph_objectstore_tool: fix check_output on python2.6

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: fix check_output on python2.6

* backported the subprocess.check_output from python2.7

Fixes: #10756
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3652 from dachary/wip-mailmap

mailmap: UMD is University of Maryland

Merge pull request #3131 from mslovy/fix_init_filejournal_throttle

os: FileJournal:: fix, uninitialization of FileJournal throttle

Reviewed-by: Sage Weil <sage@redhat.com>

mailmap: UMD is University of Maryland

And not University of Mississippi

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/hammer'

Merge pull request #3468 from ceph/wip-cephfs-snap1

Wip cephfs snap1

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3649 from ceph/wip-opath-setattr

client: handle O_PATH fds on setattr too

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3633 from sponce/wip-10758

Fixed issue #10758 : write_full behavior in libradosstriper

Reviewed-by: Loic Dachary <ldachary@redhat.com>

libradosstriper: fixed write_full when ENOENT

Fixes: #10758
write_full was returning ENOENT when the file did not exists, while it should just have created it without complaining.

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #3624 from ktdreyer/wip-doc-submitting-patches-branches

SubmittingPatches: clarify the use of branches in PRs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: handle O_PATH fds on setattr too

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3626 from majianpeng/misc

rbd.py: Add rbd_read2/rbd_write2 funcion which can handle fadvise flags.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd.py: Add rbd_read2/rbd_write2 funcion which can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3642 from vuhuong/wip-hammer

Wip xio hammer

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

Merge pull request #3639 from ceph/wip-10770

rgw: send appropriate op to cancel bucket index pending operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3641 from athanatos/wip-10731

ReplicatedPG: block writes on degraded objects unless all peers support ...

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: send appropriate op to cancel bucket index pending operation

Fixes: #10770
Backport: firefly, giant

Reported-by: baijiaruo <baijiaruo@126.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3618 from tchaikov/add-norebalance-flag

Add norebalance flag

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds: avoid propagrating unnecessary snap rstat to parent

if parent is not snapshotted, avoid propagrating snap rstat to it

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: update client_need_snapflush when splitting snap inode

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix rdlock snap inode

make sure the "sync head inode" code  execute

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: save the oldest snapid of inode

When moving a inode to different snapshot realm, we need to check if
the inode is snapshotted. If the inode is snapshotted, we create a
snapshot realm for it. The snapshot realm records inode's past snapshot
realm.

Currently there is no reliable way to check if a inode is snapshotted.
So add a new field to CInode the store the oldest snapid, it's used to
check if inode is snapshotted.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

qa/workunits/snaps: remove snapshots when test finishes

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: split old_inode when needed

Signed-off-by: Yan, Zheng <zyan@redhat.com>

qa/workunits/snaps: don't assume snapshot is disabled

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph: don't count snap dentries when calculating new fragstat/neststat

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: add snaprealm to open_past_parents after it's opened

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: avoid infinite loop of SnapRealm::_open_parents

handle the case that some past parents are missing

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly commit snaprealm to dirfrag

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't manully create snaprealm during rename

project_past_snaprealm_parent() will do the same job. Besides
MDCache::_snaprealm_create_finish() is buggy, it updates snaprealm's
current_parent_since incorrectly

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't mark dirfrag dirty after dropping stale dentries

now MDS use omap to store dirfrag, and always does partial commit.
marking dirfrag dirty after dropping stale dentries is useless.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: remove snapshot dentry during rmsnap

reply of RMSNAP request has no trace, we need to invalidate the
snapshot dentry manually

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: remove snaprealm from parent when closing it

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: check xattr version independent of inode version

Fixes: #10343
Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove superfluous assertion

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: temporarily disable recovery for snapshotted files

recovery of snapshots is very inefficient, which requires
NR_SNAPS * NR_OBJECTS operations.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: verify backtrace by auth MDS only

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: avoid sending snaptrace for snapshot inodes

snapshot information in snaptrace is only used by head inodes

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: optimize SnapRealm::split_at()

avoid executing the 'split children' code because all snaprealms in
mdsdir has no child snaprealm.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix reference after free

CDir::try_trim_snap_dentry() may free the dentry

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: journal snaprealm for all types of inodes

regular file also can has snaprealm.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: always create multiversion inode when cow inode

When moving a snapshotted regular file to stray directory, we need
to create an snaprealm for it. The snaprealm is required for building
correct snapshot context when purging file objects.

Current code normally create seperate inodes when a snapshotted file
inode is COWed. Each time the file inode is COWed, the head inode's
'first' is updated. So when deleting the file, we can't use head inode's
'first' to check if it was snapshotted.

This commit make MDCache::journal_cow_dentry always create multiversion
inode. multiversion inode's 'first' keeps constant.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: check multiversion inode when purging stray

we can't purge a multiversion directory inode when it is still referenced
by some snapshots. For regular files that are still referenced by some
snapshots, we need to keep the backtrace object.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly project past snaprealm parent while renaming files

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly update trace vector when quiting path traverse

rdlock_path_pin_ref() uses the trace vector to setup request reply's
tracedn

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly update capability's client_follow

handle discontinuous snapid(s)

Fixes: #10323
Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix CInode::remove_need_snapflush

make sure client and snapid exist, then drop CInode::PIN_NEEDSNAPFLUSH

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: provide correct snaps to purge_stale_snap_data()

if a directory inode has snaprealm, its parent directory's snapshot
set can be not equal to its snapshot set

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: set last snapid according to removed snaps in data pools

handle the case that we create cephfs using existing pools and
the existing pools' removed snaps are not empty.

Fixes: #10315
Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: update current_parent_since for newly snapshotted directory

current_parent_since should be set to snapshot seq when the directory
was created.

Fixes: 10312
Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix MDCache::journal_cow_dentry()

Use parent directory's snaprealm to decide if we should cow dentry.
Otherwise we may create orphan dentry when deleting an directory with
snaprealm.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: check snaprealm before drop dentries in deleted directory

There is an optimization that drop dentries in deleted directory,
so that MDS can purge corresponding inode faster. We need to check
if the deleted directory is still referenced by any snapshot. If
it is, we can't drop dentries or skip committing dirfrags to object
store. The simplest method to check if a delted directory is
potentially referenced by snaphosts is checking diri->snaprealm.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly update CInode->first during journal replay

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3638 from dachary/wip-make-check

run-make-check.sh: git --ancestry-path is not backward compatible

Reviewed-by: Dmytro Iurchenko <diurchenko@mirantis.com>

install-deps.sh: check for debian/control existence

If the debian/control file does not exist the error is cryptic.
Improve the situation by checking before running dpkg-checkbuilddeps.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

run-make-check.sh: git --ancestry-path is not backward compatible

The behavior is different on git 1.8 which is the default on centos7.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3630 from hef87/hef87-patch-1

Bug fix : AsyncConnection: fix wrong scope of data blocks

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Bug fix : AsyncConnection: fix wrong scope of data blocks

Signed-off-by: Feng He <fenglife@hotmail.com>

Merge pull request #3609 from hef87/hef87-patch-1

AsyncConnection: fix wrong scope of data blocks

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

xio: Squash the whitespaces

Signed-off-by: Vu Pham <vu@mellanox.com>

Merge remote-tracking branch 'gh/hammer'

Merge remote-tracking branch 'gh/wip-assert-version'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3532 from dachary/wip-run-make-check

run-make-check: use nproc and shorter git expression

Merge pull request #3611 from kylinstorage/wip-rbd-recover-tool

Wip rbd recover tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: Fix CMakeLists.txt to compile librbd

Add source files to CMakeLists.txt to compile librbd

Signed-off-by: Vu Pham <vu@mellanox.com>

xio: Fix release xio message on closed connection issue

Do not call xio_release_msg on closed connection

Signed-off-by: Vu Pham <vu@mellanox.com>

Merge pull request #3544 from vuhuong/xio-spread-portals

Xio spread portals

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

Merge pull request #3628 from SUSE/wip-initradosgw-fixup

Unconditionally chown rados log file.

Reviewed-by: Sage Weil <sage@redhat.com>

Unconditionally chown rados log file.

This fixes bnc#905047 (in a somewhat ad-hoc way). Sadly the log
file gets created from several places, so its existence does not
mean init-radosgw had actually run.

Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #3623 from ceph/wip-opath

client: fix O_PATH on older Linux systems

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3622 from athanatos/wip-10340

Objecter::_op_submit_with_budget: add timeout before call

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3621 from athanatos/wip-10717

ReplicatedPG::do_op: ignore snapset_obc if !exists

Reviewed-by: Sage Weil <sage@redhat.com>

osd: apply affinity if any affinity is not default

osd,mon: add norebalance osdmap flag

Fixes: #10559
Signed-off-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: fix wrong scope of data blocks

Signed-off-by: Feng He <fenglife@hotmail.com>

Merge pull request #3616 from ceph/wip-export-dir

mds: implement export dir asok

SubmittingPatches: clarify the use of branches in PRs

Add some documentation so developers understand what goes into next vs
what goes into master and how to handle backports to stable branches.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rbd_recover_tool: move rbd_recover_tool directory to src/tools subdirectory

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

client: fix O_PATH on older Linux systems

O_PATH was introduced in 2.6.39, but many Linux systems in use
today still have older kernels than that. Check for them.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ReplicatedPG: block writes on degraded objects unless all peers support it

Fixes: #10731
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3607 from ceph/wip-librbd-invalid-map

librbd: verify image's object map validity

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3592 from ceph/wip-librbd-watch-errors

librbd: better handling for image watch errors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

include/encoding: fix an compile warning

./include/encoding.h:317:7: warning: 't' may be used uninitialized in
this function [-Wmaybe-uninitialized]

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3619 from ceph/wip-8903

librbd: removed lingering TODO message from ImageWatcher

Reviewed-by: Sage Weil <sage@redhat.com>

msg: fixup for 2ffacbe (crc configuration in messenger)

* Update the remaining Message::encode() calls, which now expect crc
  flags to be passed as a flags parameter instead of a bool (this e.g.
  fixes the issue with routing messages forwarded from an older client).

* In Message::encode() data crc is calculated when MSG_CRC_DATA is
  set, but in decode_message(), Pipe::read/write_message() data crc is
  calculated when MSG_CRC_HEADER is set. Fix this.

Note, 2ffacbe changed the behavior of Pipe::read_message/write_message():
previously the methods always calculated crc, now they calculate it
only if crc is enabled in the config. This means crc can not be disabled
if there are monitors of older version in the cluster.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3613 from rzarzynski/wip-4920

client: add support for O_NOFOLLOW and O_PATH

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

pybind: fixed runtime errors with librbdpy

There was a typo within the RBD Image get_flags method and
several runtime errors relating to the new fadvise flags.

Fixes: #10782
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: implement export dir asok

This was available as a legacy mds tell command
previously: make it availably via the cleaner
asok interface.  asok rather than new `tell` because
it's a low level/debug thing.

Signed-off-by: John Spray <john.spray@redhat.com>

client: implement support for O_PATH on Linux.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

client: add support for O_NOFOLLOW in Client::open().

Fixes: #4920
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rbd-recover-tool: add usefull information of this tool
include README, FAQ, TODO

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

rbd-recover-tool: add a test case for raw image & snapshot

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

rbd-recover-tool: implement the function framework
add control files & config files

rbd-recover-tool is an offline tool for dead ceph cluster
to recever rbd image in replicated pool from all osds.
it is a simple but usefull policy to keep data secruity on ceph

Sigbed-off-by: Min Chen <minchen@ubuntukylin.com>

rbd-recover-tool: add the basic function files

Signed-off-by: Min Chen <minchen@ubuntukylin.com>

Merge pull request #3603 from dachary/wip-10729-debian-install-deps

install-deps.sh: get lsb_release if needed

Reviewed-by: Yann Dupont <yann@objoo.org>

Merge pull request #3565 from rzarzynski/wip-7264

pybind/rados.py: fix error hiding and inconsistency on librados load.

Reviewed-by: Dan Mick <dan.mick@redhat.com>

doc: Removed references to Inktank suppport.

Signed-off-by: John Wilkins <jowilkin@redhat.com>