qa/workunits/rbd: small fixup and improvements for rbd-mirror tests

- log to stderr;
- log status if a `wait_for` function failed;
- don't needlessly sleep in `wait_for` functions after the last
  unsuccessful iteration;
- make `wait_for_pool_images` work for image removal case;
- fix `wait_for_pool_images` reset timeout (last_count set).

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit fdb971a2660111d35f0a3077f17a9ca85ca2ef54)

Merge pull request #13043 from SUSE/wip-18570-jewel

jewel: Python Swift client commands in Quick Developer Guide don't match configuration in vstart.sh

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13166 from smithfarm/wip-drop-ext4-test

tests: Remove ext4 option from rados:thrash tests

tests: Remove ext4 option from rados:thrash tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(manual cherry-pick from ceph-qa-suite 1fcc4457144278f77dd8462ecf34948a11dcb2a9)

Merge pull request #13160 from ceph/revert-13068-jewel-rgw-multipart-upload-copy

Revert "jewel: rgw: multipart upload copy"

Reverting, will require further testing before merging. Some tests fail.

Revert "jewel: rgw: multipart upload copy"

Merge pull request #13068 from linuxbox2/jewel-rgw-multipart-upload-copy

DNM: jewel:rgw: multipart upload copy

rgw: minor optimization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4919dc9987c6376d3d4e143702c26417449524c5)

rgw: rgw_obj_key use adjustment in multipart copy part

This fixes a case where objects start with double underscore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 29fece3545cc1df404a25eec46706b16f893a5df)

rgw: multipart copy-part handle versionId

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 53521efffb1cb92e5f5ce992d4127bf9498d7c33)

rgw: multipart copy part minor parsing cleanup

no need for range.size()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6e9b824d5d4017239d58b4752ebc43bfad8f698d)

rgw: multipart copy, check for empty bucket, improve logging

also reduce log level for non critical user errors.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 2bcb3d286b230ef917d5ba96c8276a942f544689)

rgw: multipart copy part, chunked read

Don't read the entire range from source object, read it in parts.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4049e47a0cfc1eef6efd502590b68ba7234589d3)

rgw: doc: add multipart uploads copy part feature as supported

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 8a7ebeee2ff3f10ceb23b7fa43e43c3c450efe22)

rgw: multipart uploads copy part support

Add multipart uploads copy part feature.

Fixes: http://tracker.ceph.com/issues/12790
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
(cherry picked from commit 949480c2e9760855ed6a0501d364d5e766c8207d)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13103 from dillaman/wip-18672

jewel: qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: use more recent qemu-iotests that support Xenial

Fixes: http://tracker.ceph.com/issues/18149
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4314cb945a9c2296e2f7cd357b09015777f233c0)

qa/workunits/rbd: removed qemu-iotest case 077

The test case is not stable due to racing console output. This
results in spurious failures.

Fixes: http://tracker.ceph.com/issues/10773
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2c70df978d605a45ff81971b86f5afbefbdaabb6)

Merge pull request #12137 from jcsp/wip-17974

jewel: client: fix stale entries in command table

Merge pull request #12686 from SUSE/wip-18272-jewel

jewel: tests: Workunits needlessly wget from git.ceph.com

Merge pull request #12836 from SUSE/wip-18462-jewel

jewel: Decode errors on backtrace will crash MDS

Merge pull request #13023 from SUSE/wip-18603-jewel

jewel: cephfs test failures (ceph.com/qa is broken, should be download.ceph.com/qa)

Merge pull request #12155 from dachary/wip-17956-jewel

jewel: Clients without pool-changing caps shouldn't be allowed to change pool_namespace

Merge pull request #12325 from dachary/wip-18026-jewel

jewel: ceph_volume_client.py : Error: Can't handle arrays of non-strings

Merge pull request #13060 from asheplyakov/jewel-bp-18615

jewel: mds: fix null pointer dereference in Locker::handle_client_caps

Merge pull request #11656 from ajarr/wip-17705-jewel

jewel: ceph_volume_client: fix recovery from partial auth update

Merge pull request #12154 from dachary/wip-18008-jewel

jewel: Cannot create deep directories when caps contain path=/somepath

Merge pull request #13085 from jcsp/wip-18361-jewel

jewel: client: populate metadata during mount

qa/tasks/cephfs/filesystem.py: backport _write_data_xattr() function

This is a partial manual backport of 5f77f09b019b607b84e6a8f89ce19065383ca108

It is needed by test_corrupt_backtrace() in qa/tasks/cephfs/test_damage.py

Signed-off-by: Nathan Cutler <ncutler@suse.com>

client: populate metadata during mount

This way we avoid having to over-write the "root"
metadata during mount, and any user-set overrides (such
as bad values injected by tests) will survive.

Because Client instances may also open sessions without
mounting to send commands, add a call into populate_metadata
from mds_command as well.

Fixes: http://tracker.ceph.com/issues/18361
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 1dbff09ad553f9ff07f4f4217ba7ece6c2cdc5d2)

mds: fix null pointer dereference in Locker::handle_client_caps

Locker::handle_client_caps delays processing cap message if the
corresponding inode is freezing or frozen. When the message gets
processed, client can have already closed the session.

Fixes: http://tracker.ceph.com/issues/18306
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit e281a0b9c1fdeaf09f1b01f34cecd62e4f49d02e)

Doc: Fixes Python Swift client commands

Fixes: http://tracker.ceph.com/issues/17746
Signed-off-by: Ronak Jain <ronakjain@outlook.in>
(cherry picked from commit 8c79959557d60f619adf1a3ed1b5bd1112ceaabb)

qa: update remaining ceph.com to download.ceph.com

Fixes: http://tracker.ceph.com/issues/18574
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 549d993d3fd8ffffa280ed4a64aca41d1c6f2da1)

Conflicts:
qa/tasks/cram.py (trivial resolution)

Merge pull request #12766 from jtlayton/wip-18408-jewel

client: Fix lookup of "/.." in jewel

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #12147 from dachary/wip-18007-jewel

jewel: ceph-disk: ceph-disk@.service races with ceph-osd@.service

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #12983 from ceph/wip-cherry-pick-4vasu

qa: Wip cherry pick https://github.com/ceph/ceph/pull/12969

Add ceph-create-keys to explicitly create admin/bootstrap keys

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit 68f9b7eb3c0548c88650f67fb72c6ff9bc0f3ead)

Remove debug overrides

the high level of debug for mon/osd is causing remoto to hang during get key

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit f7dcc74cd3f119a2f65584fdb544c08d115f8c39)

use the create option during instantiation

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit be836bb30960000468c79e08fb416ceefd79d7db)

Merge pull request #12210 from ddiss/tracker18049_ceph_disk_trigger_flock_timeout_jewel

jewel: systemd/ceph-disk: reduce ceph-disk flock contention

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #12959 from SUSE/wip-18545-jewel

jewel: [teuthology] update Ubuntu image url after ceph.com refactor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/qemu: update default image url after ceph.com redesign

Fixes: http://tracker.ceph.com/issues/18542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6d17befb3bbc3d83c9d23d763ad95e1e7b2e4be0)

test_volume_client: remove superfluous arguments

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit bb60e01904187db417e8c7d6e57401823a0072fd)

test_volume_client: check volume size

Check that the total size shown by the df output of a mounted volume
is same as the volume size and the quota set on the volume.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 91c74f4778ce5433968226345ffe26e876eb56a7)

tasks/cephfs: test recovery of partial auth update

... in ceph_volume_client.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit f0134a3db576282ed05d4b94b969b9593297669d)

ceph_volume_client: fix partial auth recovery

... for volumes whose group_id is None.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 0ab8badcf3ffe685135af17dc28b238f6e686922)

ceph_volume_client: check if volume metadata is empty

... when recovering from partial auth updates.

Auth update happens in the following order:
auth metadata update, volume metadata update, and then Ceph auth
update.

A partial auth update can happen such that auth metadata is updated,
but the volume metadata isn't updated and is empty, and the auth
update did not propogate to Ceph. When recovering from such a
scenario, check if volume metadata is empty and if so remove the
partial auth update info in auth metadata.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit a95de7882cdf70e04e3c918ff41fc690d0d9bda3)

ceph_volume_client: fix _recover_auth_meta() method

It needs to be an instance method.

Fixes: http://tracker.ceph.com/issues/17216
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 675cb91b68c1b54698708d604253ab9d1b2abdec)

Merge pull request #12745 from SUSE/wip-18386-jewel

jewel: tests: use ceph-jewel branch for s3tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12912 from liewegas/wip-workunits-jewel

qa/tasks/workunits: backport misc fixes to jewel

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/workunit: clear clone dir before retrying checkout

If we checkout ceph-ci.git, and don't find a branch,
we'll try again from ceph.git. But the checkout will
already exist and the clone will fail, so we'll still
fail to find the branch.

The same can happen if a previous workunit task already
checked out the repo.

Fix by removing the repo before checkout (the first and
second times).  Note that this may break if there are
multiple workunit tasks running in parallel on the same
role.  That is already racy, so if it's happening, we'll
want to switch to using a truly unique clonedir for each
instantiation.

Fixes: http://tracker.ceph.com/issues/18336
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2a7013cd5a033c5be43350505d75f088e831e201)

qa/tasks/workunit: retry on ceph.git if checkout fails

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 72d73b8c8836ae35c518fa09f44805a74038f02a)

qa/tasks/workunit.py: add CEPH_BASE env var

Root of git checkout

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 27b8eac24922f8b4bd065e6e7f0bc8e2ba37b5d5)

qa/tasks/workunit: leave workunits inside git checkout

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4602884ab8f5a256d13091f7239d938990482d95)

Merge pull request #12791 from athanatos/wip-15943-jewel

jewel: crash adding snap to purged_snaps in ReplicatedPG::WaitingOnReplicas (part 2)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12868 from athanatos/wip-17899-jewel

OSDMonitor: only reject MOSDBoot based on up_from if inst matches

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks: add test_corrupt_backtrace

Validate that we get EIO and a damage table entry
when seeing a decode error on a backtrace.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 5f6cdab80f6e2f09af5783c8f616d8ddd6d9f428)

mds: check for errors decoding backtraces

Fixes: http://tracker.ceph.com/issues/18311
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 6f489c74ac0040631fde0ceb0926cbab24d3ad55)

PG: fix cached_removed_snaps bug in PGPool::update after map gap

5798fb3bf6d726d14a9c5cb99dc5902eba5b878a actually made 15943 worse
by always creating an out-of-date cached_removed_snaps value after
a map gap rather than only in the case where the the first map after
the gap did not remove any snapshots.

Introduced: 5798fb3bf6d726d14a9c5cb99dc5902eba5b878a
Fixes: http://tracker.ceph.com/issues/15943
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 5642e7e1b3bb6ffceddacd2f4030eb13a17fcccc)

qa/config/rados.yaml: enable osd_debug_verify_cached_snaps

Also, make map gaps more likely.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit d4b6615a49e4635113f9ba900e9c57147b224357)

PG::handle_advance_map: add debugging option to verify cached_removed_snaps

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit aeb8fef92469831d94f06db457a4ba15b5b0e3c5)

client: don't use special faked-up inode for /..

The CEPH_INO_DOTDOT thing is quite strange. Under most OS (Linux
included), the parent of the root is itself. IOW, at the root, '.' and
'..' refer to the same inode.

Change the ceph client to do the same, as this allows users to get
valid stat info for '..', as well as elimnating some special-casing.

Also in several places, we're checking dn_set.empty as an indicator
of being the root. While that is true for the root, it's also true
for unlinked directories.

This patch has treats them the same. An unlinked directory will
be reparented to itself, effectively acting as a root of its own.

Fixes: http://tracker.ceph.com/issues/18408
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 30d4ca01db0de9a1e12658793ba9bf9faf0331dd)

tests: rbd/test_lock_fence.sh: fix rbdrw.py relative path

This commit fixes a regression introduced by
cf294777ea92f0911813a7132068584d4f73a65a

Fixes: http://tracker.ceph.com/issues/18388
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 91231de16dbe4d0e493ec617165a2b38078d122b)

tests: use ceph-jewel branch for s3tests

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/workunit: clear clone dir before retrying checkout

If we checkout ceph-ci.git, and don't find a branch,
we'll try again from ceph.git. But the checkout will
already exist and the clone will fail, so we'll still
fail to find the branch.

The same can happen if a previous workunit task already
checked out the repo.

Fix by removing the repo before checkout (the first and
second times).  Note that this may break if there are
multiple workunit tasks running in parallel on the same
role.  That is already racy, so if it's happening, we'll
want to switch to using a truly unique clonedir for each
instantiation.

Fixes: http://tracker.ceph.com/issues/18336
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2a7013cd5a033c5be43350505d75f088e831e201)

qa/tasks/workunit: retry on ceph.git if checkout fails

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 72d73b8c8836ae35c518fa09f44805a74038f02a)

qa/workunits: include extension for nose tests

When you have a relative path you have to include the extension.
Weird.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5666fd61d6dbd40be1d79354227cabd562e829ea)
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Conflicts:
qa/workunits/rados/test_python.sh (nosetests instead of nose)

qa/workunits: use relative path instead of wget from git

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cf294777ea92f0911813a7132068584d4f73a65a)

Conflicts:
qa/workunits/rados/test_python.sh (nosetests instead of nose)

qa/tasks/workunit.py: add CEPH_BASE env var

Root of git checkout

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 27b8eac24922f8b4bd065e6e7f0bc8e2ba37b5d5)

qa/tasks/workunit: leave workunits inside git checkout

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4602884ab8f5a256d13091f7239d938990482d95)

Merge remote-tracking branch 'ceph/jewel-next' into jewel

Merge pull request #12591 from jtlayton/wip-18308-jewel

Clear setuid bits on ownership changes

Merge branch 'jewel' into wip-18308-jewel

Merge pull request #12592 from jtlayton/wip-18307-jewel

Fix mount root for ceph_mount users and change tarball format

ceph_disk: fix a jewel checkin test break

Silly python:

    ceph_disk/main.py:173:1: E305 expected 2 blank lines after class or function definition, found 1
    ceph_disk/main.py:5011:1: E305 expected 2 blank lines after class or function definition, found 1

Signed-off-by: Jeff Layton <jlayton@redhat.com>

automake: convert to tar-pax

We hit some recent build issues with the merge of ceph-qa-suite into
the main repo. The ustar format barfs on >100 character symlink
paths.

Convert to using "tar-pax" which should make it use the posix format.
Any build machine that we're reasonably targeting should support it.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: drop setuid/setgid bits on ownership change

When we hold exclusive auth caps, then the client is responsible for
handling changes to the mode. Make sure we remove any setuid/setgid
bits on an ownership change.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 18d2499d6c85a10b4b54f3b8c335cddf86c4588f)

mds: clear setuid/setgid bits on ownership changes

If we get a ownership change, POSIX mandates that you clear the
setuid and setgid bits unless you are "appropriately privileged", in
which case the OS is allowed to leave them intact.

Linux however always clears those bits, regardless of the process
privileges, as that makes it simpler to close some potential races.
Have ceph do the same.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 6da72500882d9749cb2be6eaa2568e6fe6e5ff4d)

client: set metadata["root"] from mount method when it's called with a pathname

Currently, we only set the root properly config file or the
--client_metadata command line option. If a userland client program
tries to call ceph_mount with a pathname, it's not being properly
set.

Since we already hold the mutex, we can just update it directly.

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 9f8810008c82eebe6e354e7e321e33a3dcba8407)

Merge pull request #12454 from liewegas/qa-suite-jewel

jewel: merge ceph-qa-suite

merge ceph-qa-suite

move ceph-qa-suite dirs into qa/

Revert "tasks/workunit.py: depth 1 clone"

This reverts commit e6f61ea9f19d0f1fad4a6547775fa80616eeeb89.

tasks/workunit.py: depth 1 clone

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4faf77a649cb3f8ddf497ca81937b3dbf63a18dc)

tasks/workunit: remove kludge to use git.ceph.com

This was hard-coded to ceph.git (almost) and breaks when
you specify --ceph-repo.  Remove it entirely.  We'll see if
github.com is better at handling our load than it used to
be!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 159c455a0326eef2c017b3e3cf510f918b5ec76c)

tasks/ceph: restore context of osd mount path before mkfs

all newly created files and directories under the mount dir inherit the
SELinux type of their parent directory. so we need to set it before
mkfs.

Fixes: http://tracker.ceph.com/issues/16800
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 53225d5272a1d35d4183fcfa55a139f55f77e122)

10.2.5

Merge pull request #11865 from dachary/wip-17710-jewel

jewel: multisite: race between ReadSyncStatus and InitSyncStatus leads to EIO errors

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12376 from liewegas/wip-msgr-eagain-loop-jewel

msg/simple/Pipe: avoid returning 0 on poll timeout

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

msg/simple/Pipe: avoid returning 0 on poll timeout

If poll times out it will return 0 (no data to read on socket).  In
165e5abdbf6311974d4001e43982b83d06f9e0cc we changed tcp_read_wait from
returning -1 to returning -errno, which means we return 0 instead of -1
in this case.

This makes tcp_read() get into an infinite loop by repeatedly trying to
read from the socket and getting EAGAIN.

Fix by explicitly checking for a 0 return from poll(2) and returning
EAGAIN in that case.

Fixes: http://tracker.ceph.com/issues/18184
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6c3d015c6854a12cda40673848813d968ff6afae)

Merge pull request #12033 from dachary/wip-17926-jewel

jewel: ceph-disk --dmcrypt create must not require admin key

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11968 from ddiss/jewel_next_flush_evict_snaps

jewel: tools: snapshotted RBD extent objects can't be manually evicted from a cache tier

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12151 from dachary/wip-18011-jewel

jewel: test fails due to The UNIX domain socket path

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12296 from SUSE/wip-18133-jewel

jewel: build/ops: fix undefined crypto references with --with-xio

Reviewed-by: Loic Dachary <ldachary@redhat.com>

10.2.4

Merge pull request #11997 from Abhishekvrshny/wip-17876-jewel

jewel: osd: update_log_missing does not order correctly with osd_ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11944 from SUSE/wip-17866-jewel

jewel: osd: Add config option to disable new scrubs during recovery

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11672 from linuxbox2/jewel-17663

jewel: rgw_rest_s3:  apply missed base64 try-catch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11953 from SUSE/wip-17885-jewel

jewel: test: temporarily disable fork()'ing tests

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11884 from SUSE/wip-17754-jewel

jewel: ceph-create-keys loops forever

Reviewed-by: Loic Dachary <ldachary@redhat.com>