Merge pull request #11476 from dachary/wip-17162-jewel

jewel: rgw: multisite doesn't retry RGWFetchAllMetaCR on failed lease

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11470 from dachary/wip-17514-jewel

jewel: rgw：bucket check remove _multipart_ prefix

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10661 from dachary/wip-16871-jewel

jewel: rgw: Have a flavor of bucket deletion in radosgw-admin to bypass garbage collection

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13187 from asheplyakov/jewel-bp-18729

jewel: cli: ceph-disk: convert none str to str before printing it

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13153 from smithfarm/wip-no-firefly-on-centos

jewel: tests: upgrade:hammer-x: install firefly only on Ubuntu 14.04

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13118 from smithfarm/wip-18274-jewel

jewel: Memory leaks in object_list_begin and object_list_end

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13025 from SUSE/wip-18605-jewel

jewel: ceph-disk prepare writes osd log 0 with root owner

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13106 from ceph/wip-cd-dev-option

qa/tasks: jewel backport -  ceph-deploy use of dev option

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #13062 from asheplyakov/jewel-bp-18379

jewel: msg/simple: clear_pipe when wait() is mopping up pipes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12978 from asheplyakov/jewel-18581

jewel: ReplicatedBackend: take read locks for clone sources during recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12755 from Abhishekvrshny/wip-18284-jewel

jewel: Need CLI ability to add, edit and remove omap values with binary keys

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12677 from SUSE/wip-18104-jewel

jewel: mon: ceph osd down detection behaviour

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12291 from asheplyakov/jewel-18108

jewel: msg/simple/Pipe: error decoding addr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11998 from Abhishekvrshny/wip-17877-jewel

jewel: FileStore: fiemap cannot be totally retrieved in xfs when the number of extents > 1364

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11991 from asheplyakov/jewel-17909

jewel: osd: limit omap data in push op

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12045 from Abhishekvrshny/wip-17875-jewel

jewel: rgw: file: remove spurious mount entries for RGW buckets

Merge pull request #13161 from smithfarm/wip-lfn-upgrade-hammer

jewel: tests: add require_jewel_osds before upgrading last hammer node

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12425 from SUSE/wip-18190-jewel

jewel: rbd-mirror: gmock warnings in bootstrap request unit tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13129 from smithfarm/wip-18558-jewel

jewel: rbd: bench-write will crash if --io-size is 4G

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13155 from smithfarm/wip-18494-jewel

jewel: rbd: [rbd-mirror] sporadic image replayer shut down failure

Merge pull request #13128 from smithfarm/wip-18633-jewel

jewel: rbd: [qa] crash in journal-enabled fsx run

http://tracker.ceph.com/issues/13512

Merge pull request #13104 from idryomov/wip-fio-unmap-devices-jewel

qa/tasks: backport rbd_fio fixes to jewel

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13156 from smithfarm/wip-18455-jewel

jewel: rbd: Attempting to remove an image w/ incompatible features results in partial removal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13157 from smithfarm/wip-18434-jewel

jewel: rbd: Improve error reporting from rbd feature enable/disable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13168 from trociny/wip-18550-jewel

jewel: librbd: metadata_set API operation should not change global config setting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12739 from SUSE/wip-18323-jewel

jewel: JournalMetadata flooding with errors when being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12416 from SUSE/wip-18219-jewel

jewel: msg: don't truncate message sequence to 32-bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13115 from smithfarm/wip-18404-jewel

jewel: mon: cache tiering: base pool last_force_resend not respected (racing read got wrong version)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13045 from SUSE/wip-18553-jewel

jewel: mon: peon wrongly delete routed pg stats op before receive pg stats ack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13117 from smithfarm/wip-18280-jewel

jewel: mon: osd flag health message is misleading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12320 from dachary/wip-18061-jewel

jewel: rgw:fix for deleting objects name beginning and ending with underscores of one bucket using POST method of js sdk.

Merge pull request #11759 from Abhishekvrshny/wip-17783-jewel

jewel: rgw: json encode/decode of RGWBucketInfo missing index_type field

Merge pull request #12153 from dachary/wip-18010-jewel

jewel: Cleanly reject session evict command when in replay

Merge pull request #12324 from dachary/wip-18103-jewel

jewel: truncate can cause unflushed snapshot data lose

Merge pull request #12783 from SUSE/wip-18413-jewel

jewel: cephfs: lookup of /.. in jewel returns -ENOENT

Merge pull request #12921 from xiaoxichen/wip-18520-jewel

Jewel: speed up readdir by skipping unwanted dn

Merge pull request #13119 from smithfarm/wip-18565-jewel

jewel: MDS crashes on missing metadata object

Merge pull request #13120 from smithfarm/wip-18551-jewel

jewel: ceph-fuse crash during snapshot tests

Merge pull request #13123 from smithfarm/wip-18282-jewel

jewel: monitor cannot start because of FAILED assert(info.state == MDSMap::STATE_STANDBY)

Merge pull request #13125 from smithfarm/wip-18195-jewel

jewel: cephfs: fix missing ll_get for ll_walk

Merge pull request #13126 from smithfarm/wip-18192-jewel

jewel: standby-replay daemons can sometimes miss events

tests: add require_jewel_osds before upgrading last hammer node

Note: this commit was inspired by
http://github.com/ceph/ceph-qa-suite/commit/50758a4810794d265c5d36a71d1e16799251a00d

As of 10.2.4, when upgrading a cluster from hammer to jewel, after the last
node is upgraded the MON will put the cluster into HEALTH_WARN and say: "all
OSDs are running jewel or later but the 'require_jewel_osds' osdmap flag is not
set". The release notes say:

    This is a signal for the admin to do "ceph osd set require_jewel_osds" – by
    doing this, the upgrade path is complete and no more pre-Jewel OSDs may be
    added to the cluster.

Fixes: http://tracker.ceph.com/issues/18719
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13049 from SUSE/wip-18433-jewel

jewel: rados bench seq must verify the hostname

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13040 from SUSE/wip-fs-thrash-jewel

jewel: tests: run fs/thrash on xfs instead of btrfs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13029 from SUSE/wip-18611-jewel

jewel: cephfs: client segfault on ceph_rmdir path /

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12875 from asheplyakov/jewel-bp-18485

jewel: osd/PG: publish PG stats when backfill-related states change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12789 from SUSE/wip-18417-jewel

jewel: osd: leveldb corruption leads to Operation not permitted not handled and assert

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12761 from SUSE/wip-18402-jewel

jewel: tests: objecter_requests workunit fails on wip branches

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11947 from SUSE/wip-17884-jewel

jewel: mon: OSDs marked OUT wrongly after monitor failover

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11508 from SUSE/wip-17583-jewel

jewel: utime.h: fix timezone issue in round_to_* funcs.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Revert "use the create option during instantiation"

jewel cephfs still uses old Filesystem initializtion method

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use dev option instead of dev-commit

Fixes: http://tracker.ceph.com/issues/18736
Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #13169 from linuxbox2/jewel-mg-errortrans

jewel:rgw: fix interface compliance of RGWCivetWeb::write_data()

Merge pull request #12323 from dachary/wip-18024-jewel

jewel: rbd: FAILED assert(m_processing == 0) while running test_lock_fence.sh

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12649 from xiaoxichen/wip-18278-jewel

jewel: rbd: librbd: use proper snapshot when computing diff parent overlap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12741 from SUSE/wip-18320-jewel

jewel: rbd status: json format has duplicated/overwritten key

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12753 from Abhishekvrshny/wip-18288-jewel

jewel: rbd-mirror: image sync object map reload logs message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12756 from Abhishekvrshny/wip-18276-jewel

jewel: rbd-nbd: invalid error code for "failed to read nbd request" messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12822 from SUSE/wip-18450-jewel

jewel: tests: update rbd/singleton/all/formatted-output.yaml to support ceph-ci

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12909 from dillaman/wip-18290-jewel

jewel: librbd: properly order concurrent updates to the object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12529 from SUSE/wip-18270-jewel

jewel: rbd: add image id block name prefix APIs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12322 from dachary/wip-18110-jewel

jewel: diff calculate can hide parent extents when examining first snapshot in clone

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-disk: convert none str to str before printing it

Error('somethings goes wrong', e) is thrown if exception `e` is caught
in ceph-disk, where e is not a string. so we can not just concat it in
Error's __str__(). so cast it to str before doing so.

introduced by d0e29c7

Fixes: http://tracker.ceph.com/issues/18371
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5e0dd1e7df43a3be589d17878714756a22052d8e)

rbd-mirror: fix gmock warnings in bootstrap request unit tests

Fixes: http://tracker.ceph.com/issues/18156
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 376026d7b24c77a59ef95d0f66686494caf0e9d0)

qa/workunits/rbd: test_status_in_pool_dir: explicitly check grep return value

Otherwise, it does not work as supposed to work in statements like below:

 set -e
 test_status_in_pool_dir ... && ...

(e.g. in wait_for_status_in_pool_dir)

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 7c078eda0fc260f6a885fa43f377b47844867d12)

rbd-mirror: make 'rbd mirror image resync' work after split-brain

Fixes: http://tracker.ceph.com/issues/18051
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 6cb1ed485f89f30fe6dda31e77e16e23f9b5b2ab)

rbd-mirror: split-brain issues should be clearly visible in mirror status

Fixed: http://tracker.ceph.com/issues/16991
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit cccca67d5f898c43207a19a6e029a1abb86efbcb)

qa/workunits/rbd: use image id when probing for image presence

This fixes a race in resync tests leading to false negative results.

Fixes: http://tracker.ceph.com/issues/18048
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit c23f56b9835f4e59d72f20a2218de8236574e65f)

qa/workunits/rbd: check status also in pool dir after asok commands

wait_for_image_replay_stopped returns not when the state is stopped,
but when the state is not replaying. So a race was possible when an
asok command was running when the previos stop command was still in
progress, leading to unexpected results.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 2882f8c01a42ee3cfe22a838b64f21c93cf84d16)

Conflicts:
qa/workunits/rbd/rbd_mirror.sh (tests for not backported features)

qa/workunits/rbd: wait for image deleted before checking health

This is a fixup to the previous commit.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit a5f63f726146c2c230d172f6909d8ca1ad46895a)

qa/workunits/rbd: wait for image deleted before checking health

When a primiry image is being deleted, the mirrored image might
temporary be reported in error state, before deletion is propagated.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 8dbe42d34520d0b44b189ae4d8b96559752a57da)

qa/workunits/rbd: small fixup and improvements for rbd-mirror tests

- log to stderr;
- log status if a `wait_for` function failed;
- don't needlessly sleep in `wait_for` functions after the last
  unsuccessful iteration;
- make `wait_for_pool_images` work for image removal case;
- fix `wait_for_pool_images` reset timeout (last_count set).

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit fdb971a2660111d35f0a3077f17a9ca85ca2ef54)

Merge pull request #13043 from SUSE/wip-18570-jewel

jewel: Python Swift client commands in Quick Developer Guide don't match configuration in vstart.sh

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

PrimaryLogPG::try_lock_for_read: give up if missing

The only users calc_*_subsets might try to read_lock an object which is
missing on the primary.  Returning false in those cases is perfectly
reasonable and avoids the problem.

Fixes: http://tracker.ceph.com/issues/18583
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 3833440adea6f8bcb0093603c3a9d16360ed57ec)

ReplicatedBackend: take read locks for clone sources during recovery

Otherwise, we run the risk of a clone source which hasn't actually
come into existence yet being used if we grab a clone which *just*
got added the the ssc, but has not yet actually had time to be
created (can't rely on message ordering here since recovery messages
don't necessarily order with client IO!).

Fixes: http://tracker.ceph.com/issues/17831
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 68defc2b0561414711d4dd0a76bc5d0f46f8a3f8)

Conflicts:
src/osd/ReplicatedBackend.cc:
  PGBackend::failed_push() takes a single pg_shard_t in jewel
src/osd/ReplicatedPG.h:
  trivial: get_obc() is not declared as override in jewel,
           no pgb_{set,clear}_object_snap_mapping() in jewel

Merge pull request #13166 from smithfarm/wip-drop-ext4-test

tests: Remove ext4 option from rados:thrash tests

rgw: fix interface compliance of RGWCivetWeb::write_data()

Backport of (portions of) civet web error handling bugfixes.

Adapted from 3a9f50c55e0be6733893a7ae1a5b4f504a3b0f61.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: metadata_set API operation should not change global config setting

Fixes: http://tracker.ceph.com/issues/18465
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 27465b5916b55ac3c2846c74b89f4362ad17ff1e)

Conflicts:
    src/librbd/Operations.cc (after jewel set_metadata migrated to Operations)

tests: Remove ext4 option from rados:thrash tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(manual cherry-pick from ceph-qa-suite 1fcc4457144278f77dd8462ecf34948a11dcb2a9)

Merge pull request #13160 from ceph/revert-13068-jewel-rgw-multipart-upload-copy

Revert "jewel: rgw: multipart upload copy"

Reverting, will require further testing before merging. Some tests fail.

Revert "jewel: rgw: multipart upload copy"

Merge pull request #13068 from linuxbox2/jewel-rgw-multipart-upload-copy

DNM: jewel:rgw: multipart upload copy

rbd: enabling/disabling rbd feature should report missing dependency

Currently while enabling or disabling rbd feature command does not
give missing dependency for eg: attempting to enable the journaling
feature on an image that doesn't have the exclusive-lock feature
enabled should give missing dependency error message.

Fixes: http://tracker.ceph.com/issues/16985
Reported-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>
(manual cherry pick of bd023cfec8e9aaa8fb0095a8d9534c21b3209020)

librbd: don't remove an image w/ incompatible features

Fixes: http://tracker.ceph.com/issues/18315
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
(cherry picked from commit f76127b5e617923d14adb62bfb836a635c14f209)

rbd-mirror: avoid processing new events after stop requested

Fixes: http://tracker.ceph.com/issues/18441
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit d30873b269441815b5fc7de14c7d9a1077a17d8d)

tests: upgrade: install firefly only on Ubuntu 14.04

The upgrade:hammer-x/f-h-x-offline test installs firefly, but firefly does not
build on CentOS anymore, just Ubuntu 14.04.

References: http://tracker.ceph.com/issues/18089
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: minor optimization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4919dc9987c6376d3d4e143702c26417449524c5)

rgw: rgw_obj_key use adjustment in multipart copy part

This fixes a case where objects start with double underscore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 29fece3545cc1df404a25eec46706b16f893a5df)

rgw: multipart copy-part handle versionId

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 53521efffb1cb92e5f5ce992d4127bf9498d7c33)

rgw: multipart copy part minor parsing cleanup

no need for range.size()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6e9b824d5d4017239d58b4752ebc43bfad8f698d)

rgw: multipart copy, check for empty bucket, improve logging

also reduce log level for non critical user errors.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 2bcb3d286b230ef917d5ba96c8276a942f544689)

rgw: multipart copy part, chunked read

Don't read the entire range from source object, read it in parts.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4049e47a0cfc1eef6efd502590b68ba7234589d3)

rgw: doc: add multipart uploads copy part feature as supported

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 8a7ebeee2ff3f10ceb23b7fa43e43c3c450efe22)

rgw: multipart uploads copy part support

Add multipart uploads copy part feature.

Fixes: http://tracker.ceph.com/issues/12790
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 949480c2e9760855ed6a0501d364d5e766c8207d)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rbd: bench-write should return error if io-size >= 4G

Currently if user perform bench-write with io-size > 4G
then its crashing because currently during memory allocation
bufferptr taking size of buffer as a unsigned and io-size > 4G
will overflow with unsigned. so during memset operation it will
try to set io_size size of memory area pointed by bufferptr,
(bufferptr area is:  (4G - io_size)), so it will cause
segmentation fault.

Fix is to return error if io-size >= 4G

Fixes: http://tracker.ceph.com/issues/18422
Reported-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>
(cherry picked from commit 6ab73e5f420e89e19b52e39dab28fa4c94e00197)

Conflicts:
src/tools/rbd/action/BenchWrite.cc - jewel has rbd_bencher b(&image)
            while in master this has become rbd_bencher b(&image, io_type,
            io_size)

journal: don't hold future lock during assignment

It's possible that the future raced with its owner and reaches
an empty reference count. This was resulting in the future being
destructed while its lock was still held.

Fixes: http://tracker.ceph.com/issues/18618
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0f21ceef8336e35ca16148a9d58f511037911418)

mds: fix dropping events in standby replay

Ensure that we never drop the last segment during
standby replay -- this avoids the case where we
start ignoring events because we think we're
still waiting to see a subtreemap.

Fixes: http://tracker.ceph.com/issues/17954
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit e3f2fa356f419cbac8d72bd068313f64769ef3af)

cephfs: fix missing ll_get for ll_walk

Fixs: http://tracker.ceph.com/issues/18086

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
(cherry picked from commit ad846d10d5fbab2ded5fddc47475b95a788c223c)

Conflicts:
src/client/Client.cc - jewel has fill_stat(in, attr) instead of
                               fill_statx(in, mask, stx) in master

mon/MDSMonitor: fix iterating over mutated map

If a standby is promoted, this was modifying the
standby_daemons map that we were iterating over
in maybe_promote_standby.

Fixes: http://tracker.ceph.com/issues/18166
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit db3deb80d6e4a8e8ca7a2cf6278d5d5cb24eb616)

mon: use clearer code structure

The failed map is not changed in the first if, so the second if should be an
else.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit adba77e34c3fbf983d221729dd340afcc942778b)