Merge pull request #13104 from idryomov/wip-fio-unmap-devices-jewel

qa/tasks: backport rbd_fio fixes to jewel

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13156 from smithfarm/wip-18455-jewel

jewel: rbd: Attempting to remove an image w/ incompatible features results in partial removal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13157 from smithfarm/wip-18434-jewel

jewel: rbd: Improve error reporting from rbd feature enable/disable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13168 from trociny/wip-18550-jewel

jewel: librbd: metadata_set API operation should not change global config setting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12739 from SUSE/wip-18323-jewel

jewel: JournalMetadata flooding with errors when being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12416 from SUSE/wip-18219-jewel

jewel: msg: don't truncate message sequence to 32-bits

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13115 from smithfarm/wip-18404-jewel

jewel: mon: cache tiering: base pool last_force_resend not respected (racing read got wrong version)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13045 from SUSE/wip-18553-jewel

jewel: mon: peon wrongly delete routed pg stats op before receive pg stats ack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13117 from smithfarm/wip-18280-jewel

jewel: mon: osd flag health message is misleading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12320 from dachary/wip-18061-jewel

jewel: rgw:fix for deleting objects name beginning and ending with underscores of one bucket using POST method of js sdk.

Merge pull request #11759 from Abhishekvrshny/wip-17783-jewel

jewel: rgw: json encode/decode of RGWBucketInfo missing index_type field

Merge pull request #12153 from dachary/wip-18010-jewel

jewel: Cleanly reject session evict command when in replay

Merge pull request #12324 from dachary/wip-18103-jewel

jewel: truncate can cause unflushed snapshot data lose

Merge pull request #12783 from SUSE/wip-18413-jewel

jewel: cephfs: lookup of /.. in jewel returns -ENOENT

Merge pull request #12921 from xiaoxichen/wip-18520-jewel

Jewel: speed up readdir by skipping unwanted dn

Merge pull request #13119 from smithfarm/wip-18565-jewel

jewel: MDS crashes on missing metadata object

Merge pull request #13120 from smithfarm/wip-18551-jewel

jewel: ceph-fuse crash during snapshot tests

Merge pull request #13123 from smithfarm/wip-18282-jewel

jewel: monitor cannot start because of FAILED assert(info.state == MDSMap::STATE_STANDBY)

Merge pull request #13125 from smithfarm/wip-18195-jewel

jewel: cephfs: fix missing ll_get for ll_walk

Merge pull request #13126 from smithfarm/wip-18192-jewel

jewel: standby-replay daemons can sometimes miss events

Merge pull request #13049 from SUSE/wip-18433-jewel

jewel: rados bench seq must verify the hostname

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13040 from SUSE/wip-fs-thrash-jewel

jewel: tests: run fs/thrash on xfs instead of btrfs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13029 from SUSE/wip-18611-jewel

jewel: cephfs: client segfault on ceph_rmdir path /

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12875 from asheplyakov/jewel-bp-18485

jewel: osd/PG: publish PG stats when backfill-related states change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12789 from SUSE/wip-18417-jewel

jewel: osd: leveldb corruption leads to Operation not permitted not handled and assert

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12761 from SUSE/wip-18402-jewel

jewel: tests: objecter_requests workunit fails on wip branches

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11947 from SUSE/wip-17884-jewel

jewel: mon: OSDs marked OUT wrongly after monitor failover

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11508 from SUSE/wip-17583-jewel

jewel: utime.h: fix timezone issue in round_to_* funcs.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13169 from linuxbox2/jewel-mg-errortrans

jewel:rgw: fix interface compliance of RGWCivetWeb::write_data()

Merge pull request #12323 from dachary/wip-18024-jewel

jewel: rbd: FAILED assert(m_processing == 0) while running test_lock_fence.sh

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12649 from xiaoxichen/wip-18278-jewel

jewel: rbd: librbd: use proper snapshot when computing diff parent overlap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12741 from SUSE/wip-18320-jewel

jewel: rbd status: json format has duplicated/overwritten key

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12753 from Abhishekvrshny/wip-18288-jewel

jewel: rbd-mirror: image sync object map reload logs message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12756 from Abhishekvrshny/wip-18276-jewel

jewel: rbd-nbd: invalid error code for "failed to read nbd request" messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12822 from SUSE/wip-18450-jewel

jewel: tests: update rbd/singleton/all/formatted-output.yaml to support ceph-ci

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12909 from dillaman/wip-18290-jewel

jewel: librbd: properly order concurrent updates to the object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12529 from SUSE/wip-18270-jewel

jewel: rbd: add image id block name prefix APIs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12322 from dachary/wip-18110-jewel

jewel: diff calculate can hide parent extents when examining first snapshot in clone

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13043 from SUSE/wip-18570-jewel

jewel: Python Swift client commands in Quick Developer Guide don't match configuration in vstart.sh

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13166 from smithfarm/wip-drop-ext4-test

tests: Remove ext4 option from rados:thrash tests

rgw: fix interface compliance of RGWCivetWeb::write_data()

Backport of (portions of) civet web error handling bugfixes.

Adapted from 3a9f50c55e0be6733893a7ae1a5b4f504a3b0f61.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librbd: metadata_set API operation should not change global config setting

Fixes: http://tracker.ceph.com/issues/18465
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 27465b5916b55ac3c2846c74b89f4362ad17ff1e)

Conflicts:
    src/librbd/Operations.cc (after jewel set_metadata migrated to Operations)

tests: Remove ext4 option from rados:thrash tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(manual cherry-pick from ceph-qa-suite 1fcc4457144278f77dd8462ecf34948a11dcb2a9)

Merge pull request #13160 from ceph/revert-13068-jewel-rgw-multipart-upload-copy

Revert "jewel: rgw: multipart upload copy"

Reverting, will require further testing before merging. Some tests fail.

Revert "jewel: rgw: multipart upload copy"

Merge pull request #13068 from linuxbox2/jewel-rgw-multipart-upload-copy

DNM: jewel:rgw: multipart upload copy

rbd: enabling/disabling rbd feature should report missing dependency

Currently while enabling or disabling rbd feature command does not
give missing dependency for eg: attempting to enable the journaling
feature on an image that doesn't have the exclusive-lock feature
enabled should give missing dependency error message.

Fixes: http://tracker.ceph.com/issues/16985
Reported-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>
(manual cherry pick of bd023cfec8e9aaa8fb0095a8d9534c21b3209020)

librbd: don't remove an image w/ incompatible features

Fixes: http://tracker.ceph.com/issues/18315
Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
(cherry picked from commit f76127b5e617923d14adb62bfb836a635c14f209)

rgw: minor optimization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4919dc9987c6376d3d4e143702c26417449524c5)

rgw: rgw_obj_key use adjustment in multipart copy part

This fixes a case where objects start with double underscore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 29fece3545cc1df404a25eec46706b16f893a5df)

rgw: multipart copy-part handle versionId

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 53521efffb1cb92e5f5ce992d4127bf9498d7c33)

rgw: multipart copy part minor parsing cleanup

no need for range.size()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6e9b824d5d4017239d58b4752ebc43bfad8f698d)

rgw: multipart copy, check for empty bucket, improve logging

also reduce log level for non critical user errors.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 2bcb3d286b230ef917d5ba96c8276a942f544689)

rgw: multipart copy part, chunked read

Don't read the entire range from source object, read it in parts.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4049e47a0cfc1eef6efd502590b68ba7234589d3)

rgw: doc: add multipart uploads copy part feature as supported

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 8a7ebeee2ff3f10ceb23b7fa43e43c3c450efe22)

rgw: multipart uploads copy part support

Add multipart uploads copy part feature.

Fixes: http://tracker.ceph.com/issues/12790
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 949480c2e9760855ed6a0501d364d5e766c8207d)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mds: fix dropping events in standby replay

Ensure that we never drop the last segment during
standby replay -- this avoids the case where we
start ignoring events because we think we're
still waiting to see a subtreemap.

Fixes: http://tracker.ceph.com/issues/17954
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit e3f2fa356f419cbac8d72bd068313f64769ef3af)

cephfs: fix missing ll_get for ll_walk

Fixs: http://tracker.ceph.com/issues/18086

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
(cherry picked from commit ad846d10d5fbab2ded5fddc47475b95a788c223c)

Conflicts:
src/client/Client.cc - jewel has fill_stat(in, attr) instead of
                               fill_statx(in, mask, stx) in master

mon/MDSMonitor: fix iterating over mutated map

If a standby is promoted, this was modifying the
standby_daemons map that we were iterating over
in maybe_promote_standby.

Fixes: http://tracker.ceph.com/issues/18166
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit db3deb80d6e4a8e8ca7a2cf6278d5d5cb24eb616)

mon: use clearer code structure

The failed map is not changed in the first if, so the second if should be an
else.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit adba77e34c3fbf983d221729dd340afcc942778b)

client: fix Client::handle_cap_flushsnap_ack() crash

Struct CapSnap holds a reference to its parent inode. So erasing
struct CapSnap from Inode::cap_snaps may drop inode's last reference.
The inode gets freed in the middle of erasing struct CapSnap

Fixes: http://tracker.ceph.com/issues/18460
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 525c52fd491ed1ced385c8047872e3f557f8423f)

Conflicts:
src/client/Client.cc (jewel does in->cap_snaps.erase(follows), master
                             does not; put it after the tmp_ref assignment)

qa/tasks: add test_open_ino_errors

Validate that errors encountered during opening inos are properly
propagated

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 6526ecc084733b34129aa1f21085fa41fb53b785)

mds: propagate error encountered during opening inode by number

Fixes: http://tracker.ceph.com/issues/18179
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 2213cc2dcc0e8fb01bcae3863d0d8a4a1fd8873f)

mon/OSDMonitor: only show interesting flags in health warning

Also add PAUSEREC flag into the list.

Fixes: http://tracker.ceph.com/issues/18175
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 26220f0608f5ed4a7d97fb8d10d0d12a0fcf5583)

mon/OSDMonitor: set last_force_op_resend on overlay pool too

We currently set the last_force_op_resend field on the
base pool when we set or clear the overlay.  Set it on
the cache/overlay pool too.  The Objecter should resend
even with a change only to the base pool, but the OSD
needs to see the change on the overlay pool to correctly
discard the op.

Fixes: http://tracker.ceph.com/issues/18366
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 08c3618771b852840aa88cff1ca98d980d802941)

tests: subst repo and branch in qemu test urls

References: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 2484a5546160560a4050d35400a6edce37ae48ad)

Conflicts:
qa/suites/rbd/qemu/workloads/#qemu_xfstests.yaml#
            (file does not exist in jewel)

tests: subst branch and repo in qa/tasks/qemu.py

References: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 74689df754561e11a3377998840efcea9e780755)

tests: subst repo name in krbd/unmap/tasks/unmap.yaml

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit ed0e8be0b2c4d7a3e6e0716a0211d19e8b93f125)

tests: subst repo name in qa/tasks/cram.py

Inspired by bcbe45d948f1c4da02e27b3be5f29a8b52745e28

Fixes: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 56e37e41f4dddd289dd3c1886b192cd328ed311b)

cram: support fetching from sha1 branch, tag, commit hash

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit d2f0d745987a2e2eee4e1822146aad8da5d42708)

Conflicts:
suites/rbd/singleton/all/formatted-output.yaml (has been moved
           under qa/)

Merge pull request #13103 from dillaman/wip-18672

jewel: qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Fixes: http://tracker.ceph.com/issues/18149
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4314cb945a9c2296e2f7cd357b09015777f233c0)

qa/workunits/rbd: removed qemu-iotest case 077

The test case is not stable due to racing console output. This
results in spurious failures.

Fixes: http://tracker.ceph.com/issues/10773
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c70df978d605a45ff81971b86f5afbefbdaabb6)

tasks/rbd_fio: unmap rbd devices on cleanup

Not doing so leads to issues and can interfere with subsequent jobs.
One example is the invocation of vgs(8) during the inital test setup:
it will issue a read to the left-behind rbd device(s) whose backing
cluster is long gone, locking up the job.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 15be2d29be3e42644b0c541f5c70461f1874f24f)

tasks/rbd_fio: don't use sudo unnecessarily

Creating and cloning images doesn't require sudo.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit ec97445740bbf7848488fbb6213e74d50a355547)

Merge pull request #12137 from jcsp/wip-17974

jewel: client: fix stale entries in command table

Merge pull request #12686 from SUSE/wip-18272-jewel

jewel: tests: Workunits needlessly wget from git.ceph.com

Merge pull request #12836 from SUSE/wip-18462-jewel

jewel: Decode errors on backtrace will crash MDS

Merge pull request #13023 from SUSE/wip-18603-jewel

jewel: cephfs test failures (ceph.com/qa is broken, should be download.ceph.com/qa)

Merge pull request #12155 from dachary/wip-17956-jewel

jewel: Clients without pool-changing caps shouldn't be allowed to change pool_namespace

Merge pull request #12325 from dachary/wip-18026-jewel

jewel: ceph_volume_client.py : Error: Can't handle arrays of non-strings

Merge pull request #13060 from asheplyakov/jewel-bp-18615

jewel: mds: fix null pointer dereference in Locker::handle_client_caps

Merge pull request #11656 from ajarr/wip-17705-jewel

jewel: ceph_volume_client: fix recovery from partial auth update

Merge pull request #12154 from dachary/wip-18008-jewel

jewel: Cannot create deep directories when caps contain path=/somepath

Merge pull request #13085 from jcsp/wip-18361-jewel

jewel: client: populate metadata during mount

qa/tasks/cephfs/filesystem.py: backport _write_data_xattr() function

This is a partial manual backport of 5f77f09b019b607b84e6a8f89ce19065383ca108

It is needed by test_corrupt_backtrace() in qa/tasks/cephfs/test_damage.py

Signed-off-by: Nathan Cutler <ncutler@suse.com>

client: populate metadata during mount

This way we avoid having to over-write the "root"
metadata during mount, and any user-set overrides (such
as bad values injected by tests) will survive.

Because Client instances may also open sessions without
mounting to send commands, add a call into populate_metadata
from mds_command as well.

Fixes: http://tracker.ceph.com/issues/18361
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 1dbff09ad553f9ff07f4f4217ba7ece6c2cdc5d2)

mds: fix null pointer dereference in Locker::handle_client_caps

Locker::handle_client_caps delays processing cap message if the
corresponding inode is freezing or frozen. When the message gets
processed, client can have already closed the session.

Fixes: http://tracker.ceph.com/issues/18306
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit e281a0b9c1fdeaf09f1b01f34cecd62e4f49d02e)

doc: document hostname constraints for rados bench

Fixes: http://tracker.ceph.com/issues/17526
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1d7e8188b9067f4f51958a13a23da1e105d89c63)

mon: do not send duplicated osdmap msg to not sync'ed osd

prior to this change:
a peon may forward the pgstats to leader, and record it locally, but leader will
check if osd has the latest map before process, if not, will use a route op to
indicate peon to send it, then poen will delete routed op when fininaly send
out which make peon cannot send pgstatack when leader has processed the
pgstat update. so osd will always track it util reach a threshold block pgstats
sending, at worst, reopen mon session.
also, both leader and peon will send out the osdmap message to the osd.

after this change:
only the peon will send out the osdmap message. and the pgstatack message
will be routed to the osd as expected. so the osd will not keep track of the
"acked" pg stats in its queue forever before times out.

Fixes: http://tracker.ceph.com/issues/18458
Signed-off-by: Mingxin Liu <mingxin@xsky.com>
(cherry picked from commit 57274488c072ec6912b700288ce5b1ea8372d162)

Doc: Fixes Python Swift client commands

Fixes: http://tracker.ceph.com/issues/17746
Signed-off-by: Ronak Jain <ronakjain@outlook.in>
(cherry picked from commit 8c79959557d60f619adf1a3ed1b5bd1112ceaabb)

tests: run fs/thrash on xfs instead of btrfs

Signed-off-by: Nathan Cutler <ncutler@suse.com>

client/Client.cc: prevent segfaulting

The segfaulting in the rmdir function is caused by calling
filepath::last_dentry() function.
last_dentry() function assumes that the bits vector has always at
least one element, which is not the case for the the filepath object
created with "/" input.
This commit also fixes other functions affected by this bug:
link, unlink, rename, mkdir, mknod and symlink.

Fixes: http://tracker.ceph.com/issues/9935
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>
(cherry picked from commit 6ed7f2364ae5507bab14c60b582929aa7b0ba400)

Conflicts:
src/client/Client.cc (Client.cc - path_walk(), may_create(), and
                              _link() take fewer parameters in jewel)
src/test/libcephfs/test.cc (preceding tests are missing in jewel)

qa: update remaining ceph.com to download.ceph.com

Fixes: http://tracker.ceph.com/issues/18574
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 549d993d3fd8ffffa280ed4a64aca41d1c6f2da1)

Conflicts:
qa/tasks/cram.py (trivial resolution)

Merge pull request #12766 from jtlayton/wip-18408-jewel

client: Fix lookup of "/.." in jewel

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #12147 from dachary/wip-18007-jewel

jewel: ceph-disk: ceph-disk@.service races with ceph-osd@.service

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #12983 from ceph/wip-cherry-pick-4vasu

qa: Wip cherry pick https://github.com/ceph/ceph/pull/12969

Add ceph-create-keys to explicitly create admin/bootstrap keys

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit 68f9b7eb3c0548c88650f67fb72c6ff9bc0f3ead)

Remove debug overrides

the high level of debug for mon/osd is causing remoto to hang during get key

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit f7dcc74cd3f119a2f65584fdb544c08d115f8c39)

use the create option during instantiation

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit be836bb30960000468c79e08fb416ceefd79d7db)