Merge pull request #13115 from smithfarm/wip-18404-jewel

jewel: mon: cache tiering: base pool last_force_resend not respected (racing read got wrong version)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13045 from SUSE/wip-18553-jewel

jewel: mon: peon wrongly delete routed pg stats op before receive pg stats ack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13117 from smithfarm/wip-18280-jewel

jewel: mon: osd flag health message is misleading

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12320 from dachary/wip-18061-jewel

jewel: rgw:fix for deleting objects name beginning and ending with underscores of one bucket using POST method of js sdk.

Merge pull request #11759 from Abhishekvrshny/wip-17783-jewel

jewel: rgw: json encode/decode of RGWBucketInfo missing index_type field

Merge pull request #12153 from dachary/wip-18010-jewel

jewel: Cleanly reject session evict command when in replay

Merge pull request #12324 from dachary/wip-18103-jewel

jewel: truncate can cause unflushed snapshot data lose

Merge pull request #12783 from SUSE/wip-18413-jewel

jewel: cephfs: lookup of /.. in jewel returns -ENOENT

Merge pull request #12921 from xiaoxichen/wip-18520-jewel

Jewel: speed up readdir by skipping unwanted dn

Merge pull request #13119 from smithfarm/wip-18565-jewel

jewel: MDS crashes on missing metadata object

Merge pull request #13120 from smithfarm/wip-18551-jewel

jewel: ceph-fuse crash during snapshot tests

Merge pull request #13123 from smithfarm/wip-18282-jewel

jewel: monitor cannot start because of FAILED assert(info.state == MDSMap::STATE_STANDBY)

Merge pull request #13125 from smithfarm/wip-18195-jewel

jewel: cephfs: fix missing ll_get for ll_walk

Merge pull request #13126 from smithfarm/wip-18192-jewel

jewel: standby-replay daemons can sometimes miss events

Merge pull request #13049 from SUSE/wip-18433-jewel

jewel: rados bench seq must verify the hostname

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13040 from SUSE/wip-fs-thrash-jewel

jewel: tests: run fs/thrash on xfs instead of btrfs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13029 from SUSE/wip-18611-jewel

jewel: cephfs: client segfault on ceph_rmdir path /

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12875 from asheplyakov/jewel-bp-18485

jewel: osd/PG: publish PG stats when backfill-related states change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12789 from SUSE/wip-18417-jewel

jewel: osd: leveldb corruption leads to Operation not permitted not handled and assert

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12761 from SUSE/wip-18402-jewel

jewel: tests: objecter_requests workunit fails on wip branches

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11947 from SUSE/wip-17884-jewel

jewel: mon: OSDs marked OUT wrongly after monitor failover

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11508 from SUSE/wip-17583-jewel

jewel: utime.h: fix timezone issue in round_to_* funcs.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #13169 from linuxbox2/jewel-mg-errortrans

jewel:rgw: fix interface compliance of RGWCivetWeb::write_data()

Merge pull request #12323 from dachary/wip-18024-jewel

jewel: rbd: FAILED assert(m_processing == 0) while running test_lock_fence.sh

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12649 from xiaoxichen/wip-18278-jewel

jewel: rbd: librbd: use proper snapshot when computing diff parent overlap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12741 from SUSE/wip-18320-jewel

jewel: rbd status: json format has duplicated/overwritten key

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12753 from Abhishekvrshny/wip-18288-jewel

jewel: rbd-mirror: image sync object map reload logs message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12756 from Abhishekvrshny/wip-18276-jewel

jewel: rbd-nbd: invalid error code for "failed to read nbd request" messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12822 from SUSE/wip-18450-jewel

jewel: tests: update rbd/singleton/all/formatted-output.yaml to support ceph-ci

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12909 from dillaman/wip-18290-jewel

jewel: librbd: properly order concurrent updates to the object map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12529 from SUSE/wip-18270-jewel

jewel: rbd: add image id block name prefix APIs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12322 from dachary/wip-18110-jewel

jewel: diff calculate can hide parent extents when examining first snapshot in clone

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13043 from SUSE/wip-18570-jewel

jewel: Python Swift client commands in Quick Developer Guide don't match configuration in vstart.sh

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13166 from smithfarm/wip-drop-ext4-test

tests: Remove ext4 option from rados:thrash tests

rgw: fix interface compliance of RGWCivetWeb::write_data()

Backport of (portions of) civet web error handling bugfixes.

Adapted from 3a9f50c55e0be6733893a7ae1a5b4f504a3b0f61.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

tests: Remove ext4 option from rados:thrash tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(manual cherry-pick from ceph-qa-suite 1fcc4457144278f77dd8462ecf34948a11dcb2a9)

Merge pull request #13160 from ceph/revert-13068-jewel-rgw-multipart-upload-copy

Revert "jewel: rgw: multipart upload copy"

Reverting, will require further testing before merging. Some tests fail.

Revert "jewel: rgw: multipart upload copy"

Merge pull request #13068 from linuxbox2/jewel-rgw-multipart-upload-copy

DNM: jewel:rgw: multipart upload copy

rgw: minor optimization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4919dc9987c6376d3d4e143702c26417449524c5)

rgw: rgw_obj_key use adjustment in multipart copy part

This fixes a case where objects start with double underscore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 29fece3545cc1df404a25eec46706b16f893a5df)

rgw: multipart copy-part handle versionId

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 53521efffb1cb92e5f5ce992d4127bf9498d7c33)

rgw: multipart copy part minor parsing cleanup

no need for range.size()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6e9b824d5d4017239d58b4752ebc43bfad8f698d)

rgw: multipart copy, check for empty bucket, improve logging

also reduce log level for non critical user errors.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 2bcb3d286b230ef917d5ba96c8276a942f544689)

rgw: multipart copy part, chunked read

Don't read the entire range from source object, read it in parts.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4049e47a0cfc1eef6efd502590b68ba7234589d3)

rgw: doc: add multipart uploads copy part feature as supported

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 8a7ebeee2ff3f10ceb23b7fa43e43c3c450efe22)

rgw: multipart uploads copy part support

Add multipart uploads copy part feature.

Fixes: http://tracker.ceph.com/issues/12790
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 949480c2e9760855ed6a0501d364d5e766c8207d)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mds: fix dropping events in standby replay

Ensure that we never drop the last segment during
standby replay -- this avoids the case where we
start ignoring events because we think we're
still waiting to see a subtreemap.

Fixes: http://tracker.ceph.com/issues/17954
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit e3f2fa356f419cbac8d72bd068313f64769ef3af)

cephfs: fix missing ll_get for ll_walk

Fixs: http://tracker.ceph.com/issues/18086

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>
(cherry picked from commit ad846d10d5fbab2ded5fddc47475b95a788c223c)

Conflicts:
src/client/Client.cc - jewel has fill_stat(in, attr) instead of
                               fill_statx(in, mask, stx) in master

mon/MDSMonitor: fix iterating over mutated map

If a standby is promoted, this was modifying the
standby_daemons map that we were iterating over
in maybe_promote_standby.

Fixes: http://tracker.ceph.com/issues/18166
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit db3deb80d6e4a8e8ca7a2cf6278d5d5cb24eb616)

mon: use clearer code structure

The failed map is not changed in the first if, so the second if should be an
else.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit adba77e34c3fbf983d221729dd340afcc942778b)

client: fix Client::handle_cap_flushsnap_ack() crash

Struct CapSnap holds a reference to its parent inode. So erasing
struct CapSnap from Inode::cap_snaps may drop inode's last reference.
The inode gets freed in the middle of erasing struct CapSnap

Fixes: http://tracker.ceph.com/issues/18460
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 525c52fd491ed1ced385c8047872e3f557f8423f)

Conflicts:
src/client/Client.cc (jewel does in->cap_snaps.erase(follows), master
                             does not; put it after the tmp_ref assignment)

qa/tasks: add test_open_ino_errors

Validate that errors encountered during opening inos are properly
propagated

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 6526ecc084733b34129aa1f21085fa41fb53b785)

mds: propagate error encountered during opening inode by number

Fixes: http://tracker.ceph.com/issues/18179
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 2213cc2dcc0e8fb01bcae3863d0d8a4a1fd8873f)

mon/OSDMonitor: only show interesting flags in health warning

Also add PAUSEREC flag into the list.

Fixes: http://tracker.ceph.com/issues/18175
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 26220f0608f5ed4a7d97fb8d10d0d12a0fcf5583)

mon/OSDMonitor: set last_force_op_resend on overlay pool too

We currently set the last_force_op_resend field on the
base pool when we set or clear the overlay.  Set it on
the cache/overlay pool too.  The Objecter should resend
even with a change only to the base pool, but the OSD
needs to see the change on the overlay pool to correctly
discard the op.

Fixes: http://tracker.ceph.com/issues/18366
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 08c3618771b852840aa88cff1ca98d980d802941)

tests: subst repo and branch in qemu test urls

References: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 2484a5546160560a4050d35400a6edce37ae48ad)

Conflicts:
qa/suites/rbd/qemu/workloads/#qemu_xfstests.yaml#
            (file does not exist in jewel)

tests: subst branch and repo in qa/tasks/qemu.py

References: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 74689df754561e11a3377998840efcea9e780755)

tests: subst repo name in krbd/unmap/tasks/unmap.yaml

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit ed0e8be0b2c4d7a3e6e0716a0211d19e8b93f125)

tests: subst repo name in qa/tasks/cram.py

Inspired by bcbe45d948f1c4da02e27b3be5f29a8b52745e28

Fixes: http://tracker.ceph.com/issues/18440
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 56e37e41f4dddd289dd3c1886b192cd328ed311b)

cram: support fetching from sha1 branch, tag, commit hash

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit d2f0d745987a2e2eee4e1822146aad8da5d42708)

Conflicts:
suites/rbd/singleton/all/formatted-output.yaml (has been moved
           under qa/)

Merge pull request #13103 from dillaman/wip-18672

jewel: qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Fixes: http://tracker.ceph.com/issues/18149
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4314cb945a9c2296e2f7cd357b09015777f233c0)

qa/workunits/rbd: removed qemu-iotest case 077

The test case is not stable due to racing console output. This
results in spurious failures.

Fixes: http://tracker.ceph.com/issues/10773
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c70df978d605a45ff81971b86f5afbefbdaabb6)

Merge pull request #12137 from jcsp/wip-17974

jewel: client: fix stale entries in command table

Merge pull request #12686 from SUSE/wip-18272-jewel

jewel: tests: Workunits needlessly wget from git.ceph.com

Merge pull request #12836 from SUSE/wip-18462-jewel

jewel: Decode errors on backtrace will crash MDS

Merge pull request #13023 from SUSE/wip-18603-jewel

jewel: cephfs test failures (ceph.com/qa is broken, should be download.ceph.com/qa)

Merge pull request #12155 from dachary/wip-17956-jewel

jewel: Clients without pool-changing caps shouldn't be allowed to change pool_namespace

Merge pull request #12325 from dachary/wip-18026-jewel

jewel: ceph_volume_client.py : Error: Can't handle arrays of non-strings

Merge pull request #13060 from asheplyakov/jewel-bp-18615

jewel: mds: fix null pointer dereference in Locker::handle_client_caps

Merge pull request #11656 from ajarr/wip-17705-jewel

jewel: ceph_volume_client: fix recovery from partial auth update

Merge pull request #12154 from dachary/wip-18008-jewel

jewel: Cannot create deep directories when caps contain path=/somepath

Merge pull request #13085 from jcsp/wip-18361-jewel

jewel: client: populate metadata during mount

qa/tasks/cephfs/filesystem.py: backport _write_data_xattr() function

This is a partial manual backport of 5f77f09b019b607b84e6a8f89ce19065383ca108

It is needed by test_corrupt_backtrace() in qa/tasks/cephfs/test_damage.py

Signed-off-by: Nathan Cutler <ncutler@suse.com>

client: populate metadata during mount

This way we avoid having to over-write the "root"
metadata during mount, and any user-set overrides (such
as bad values injected by tests) will survive.

Because Client instances may also open sessions without
mounting to send commands, add a call into populate_metadata
from mds_command as well.

Fixes: http://tracker.ceph.com/issues/18361
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 1dbff09ad553f9ff07f4f4217ba7ece6c2cdc5d2)

mds: fix null pointer dereference in Locker::handle_client_caps

Locker::handle_client_caps delays processing cap message if the
corresponding inode is freezing or frozen. When the message gets
processed, client can have already closed the session.

Fixes: http://tracker.ceph.com/issues/18306
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit e281a0b9c1fdeaf09f1b01f34cecd62e4f49d02e)

doc: document hostname constraints for rados bench

Fixes: http://tracker.ceph.com/issues/17526
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 1d7e8188b9067f4f51958a13a23da1e105d89c63)

mon: do not send duplicated osdmap msg to not sync'ed osd

prior to this change:
a peon may forward the pgstats to leader, and record it locally, but leader will
check if osd has the latest map before process, if not, will use a route op to
indicate peon to send it, then poen will delete routed op when fininaly send
out which make peon cannot send pgstatack when leader has processed the
pgstat update. so osd will always track it util reach a threshold block pgstats
sending, at worst, reopen mon session.
also, both leader and peon will send out the osdmap message to the osd.

after this change:
only the peon will send out the osdmap message. and the pgstatack message
will be routed to the osd as expected. so the osd will not keep track of the
"acked" pg stats in its queue forever before times out.

Fixes: http://tracker.ceph.com/issues/18458
Signed-off-by: Mingxin Liu <mingxin@xsky.com>
(cherry picked from commit 57274488c072ec6912b700288ce5b1ea8372d162)

Doc: Fixes Python Swift client commands

Fixes: http://tracker.ceph.com/issues/17746
Signed-off-by: Ronak Jain <ronakjain@outlook.in>
(cherry picked from commit 8c79959557d60f619adf1a3ed1b5bd1112ceaabb)

tests: run fs/thrash on xfs instead of btrfs

Signed-off-by: Nathan Cutler <ncutler@suse.com>

client/Client.cc: prevent segfaulting

The segfaulting in the rmdir function is caused by calling
filepath::last_dentry() function.
last_dentry() function assumes that the bits vector has always at
least one element, which is not the case for the the filepath object
created with "/" input.
This commit also fixes other functions affected by this bug:
link, unlink, rename, mkdir, mknod and symlink.

Fixes: http://tracker.ceph.com/issues/9935
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>
(cherry picked from commit 6ed7f2364ae5507bab14c60b582929aa7b0ba400)

Conflicts:
src/client/Client.cc (Client.cc - path_walk(), may_create(), and
                              _link() take fewer parameters in jewel)
src/test/libcephfs/test.cc (preceding tests are missing in jewel)

qa: update remaining ceph.com to download.ceph.com

Fixes: http://tracker.ceph.com/issues/18574
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 549d993d3fd8ffffa280ed4a64aca41d1c6f2da1)

Conflicts:
qa/tasks/cram.py (trivial resolution)

Merge pull request #12766 from jtlayton/wip-18408-jewel

client: Fix lookup of "/.." in jewel

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #12147 from dachary/wip-18007-jewel

jewel: ceph-disk: ceph-disk@.service races with ceph-osd@.service

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #12983 from ceph/wip-cherry-pick-4vasu

qa: Wip cherry pick https://github.com/ceph/ceph/pull/12969

Add ceph-create-keys to explicitly create admin/bootstrap keys

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit 68f9b7eb3c0548c88650f67fb72c6ff9bc0f3ead)

Remove debug overrides

the high level of debug for mon/osd is causing remoto to hang during get key

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit f7dcc74cd3f119a2f65584fdb544c08d115f8c39)

use the create option during instantiation

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit be836bb30960000468c79e08fb416ceefd79d7db)

Merge pull request #12210 from ddiss/tracker18049_ceph_disk_trigger_flock_timeout_jewel

jewel: systemd/ceph-disk: reduce ceph-disk flock contention

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #12959 from SUSE/wip-18545-jewel

jewel: [teuthology] update Ubuntu image url after ceph.com refactor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/qemu: update default image url after ceph.com redesign

Fixes: http://tracker.ceph.com/issues/18542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6d17befb3bbc3d83c9d23d763ad95e1e7b2e4be0)

test_volume_client: remove superfluous arguments

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit bb60e01904187db417e8c7d6e57401823a0072fd)

test_volume_client: check volume size

Check that the total size shown by the df output of a mounted volume
is same as the volume size and the quota set on the volume.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 91c74f4778ce5433968226345ffe26e876eb56a7)

tasks/cephfs: test recovery of partial auth update

... in ceph_volume_client.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit f0134a3db576282ed05d4b94b969b9593297669d)

ceph_volume_client: fix partial auth recovery

... for volumes whose group_id is None.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 0ab8badcf3ffe685135af17dc28b238f6e686922)

ceph_volume_client: check if volume metadata is empty

... when recovering from partial auth updates.

Auth update happens in the following order:
auth metadata update, volume metadata update, and then Ceph auth
update.

A partial auth update can happen such that auth metadata is updated,
but the volume metadata isn't updated and is empty, and the auth
update did not propogate to Ceph. When recovering from such a
scenario, check if volume metadata is empty and if so remove the
partial auth update info in auth metadata.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit a95de7882cdf70e04e3c918ff41fc690d0d9bda3)

ceph_volume_client: fix _recover_auth_meta() method

It needs to be an instance method.

Fixes: http://tracker.ceph.com/issues/17216
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 675cb91b68c1b54698708d604253ab9d1b2abdec)

mds/server: skip unwanted dn in handle_client_readdir

We can skip unwanted dn which  < (offset_key, snap) via map.lower_bound, rather than
iterate across them.

Previously we iterate and skip dn which < (offset_key, dn->last), as dn->last >= snap
 means (offset_key, dn->last) >= (offset_key, snap), and such iterate_and_skip logic
still keep, so this commit doesnt change code logic but an optimization.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
(cherry picked from commit 52fe52baf920c672ac7f63a3087dcd31137891b6)

Merge pull request #12745 from SUSE/wip-18386-jewel

jewel: tests: use ceph-jewel branch for s3tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>