rgw: add debug info when comparing bucket metadata

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit ad5a2a2f199f8fcecadf2c91b33c0530a26d8c3d)

Merge pull request #1240 from ceph/wip-17734-jewel

upgrade/hammer-x: wait for osdmaps to propagate

upgrade/hammer-x: wait for osdmaps to propagate

Fixes: http://tracker.ceph.com/issues/17808
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1226 from dachary/wip-17734-jewel

releases/jewel: set require_jewel_osds

upgrade/hammer-x: set require_jewel_osds

And replace infernalis with jewel where relevant.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

rados/singleton-nomsgr/all: set require_jewel_osds

those tests do not exist anymore in master, no backport possible.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #1174 from ceph/wip-16516

jewel: test: Eliminate racey scrubbing in scrub_test.py

test: Eliminate racey scrubbing in scrub_test.py

Enhance get_pgid() to return passed value if it is already a pgid

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 29985efc7cdff252a822d31b5e18b9163849bc53)

Fixes: http://tracker.ceph.com/issues/16516

scrub_test: Fixes need with wip-13507

Accept an attribute if not present as False
Change required due change to dict with "epoch" and "inconsistents"

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4d0c668e9eb50c2149cfe36caee37c531541d1be)

Revert "check for inconsistent objects count instead of all keys"

This reverts commit e94f3901928501586f44c35612bc8f2dc504f465.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #1225 from ceph/wip-inconsistent

assert based on inconsistent objects count instead of all keys

check for inconsistent objects count instead of all keys

epoch keys also exist in the objs dump, count only
inconsistent objects

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1214 from jcsp/wip-generate-caps-jewel

tasks/ceph: move generate_caps from teuthology

tasks/ceph: move generate_caps from teuthology

This was only used in this task, and it is much too
ceph-specific to belong in teuthology.

Fixes: http://tracker.ceph.com/issues/17614
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 298cc8f932a2a327e07f5b73708404588a611322)

Merge pull request #1203 from dachary/wip-17537-jewel

suites/rbd: increase timeout for rbd-mirror stress test

suites/rbd: increase timeout for rbd-mirror stress test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 525e445f5f58388edf1523859f8f5d4d6cf4aea5)

Merge pull request #1201 from dachary/wip-17344-jewel

upgrade: ceph-test is needed for ceph-coverage

upgrade: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d44f583436f958e276e4d4819a722ed39f97c961)

Merge pull request #1191 from ceph/wip-offline-split-jewel

ceph_manager: test offline split via ceph-objectstore-tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_manager: test offline split via ceph-objectstore-tool

When killing an osd, split all pools with a low threshold.
This will slow down tests, but should not impact correctness.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit f28f881bda0ad426bfa9c0e22cc4658b2ce2ec5d)

Merge pull request #1181 from ceph/wip-jewel-release

Add ceph-deploy tests for released builds

Merge pull request #1183 from ceph/wip-librbd-disable-helgrind-jewel

jewel: suites/rbd: remove helgrind test cases

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

suites/rbd: remove helgrind test cases

Helgrind cannot properly handle Ceph's use of std::mutex --
resulting in numerous false positives and potential assertion
failures.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b29bf9253d7d2201f4cf01b0fd6997ed1bae231f)

Merge pull request #1182 from ceph/wip-smoke-krbd-iozone-jewel

jewel: smoke: mask out unsupported image features for krbd iozone test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

smoke: mask out unsupported image features for krbd iozone test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 77949e5cfc1231c96417e90aadc1573137c4b4d7)

Add basic testing of ceph bits

simple test suite to test ceph from download.ceph.com

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1173 from ceph/wip-16717-jewel

jewel: suites/upgrade: override rbd_default_features config back to pre-Jewel

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

suites/upgrade: override rbd_default_features config back to pre-Jewel

Fixes: http://tracker.ceph.com/issues/16717
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 47a41286a5506a2b0e036c3e75c99911f417e74b)

Merge pull request #1167 from ceph/wip-17222-jewel

Backport krbd/unmap fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

krbd/unmap: put client.0 on a separate remote

Otherwise a pre-single-major kernel override is a headache,
particularly with non-standard yaml configs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 311a450163cf31d26cdd585faf8e306b19731bb1)

krbd/unmap: override client.0 only

The unmap test uses one remote, so the end result is the same.
However, overriding the most specific role is nicer and allows
scheduling with

  kernel:
    client:
       branch: testing

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 02460f2156cf507c2be94e6fb1ad8af0b3be245a)

krbd/unmap: set tunables to bobtail

Kernel 3.13, which is used in pre-single-major.yaml test, doesn't
support firefly tunables (default in jewel, up from bobtail tunables).
This went unnoticed for a while because of a kernel task regression -
the pre-single-major override was ignored.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 39883e9d9f7267f92dacb40cee10794503c88539)

krbd/unmap: assert that pre-single-major kernel is installed

Similarly to how single-major-off.yaml and single-major-on.yaml check
the value of /sys/module/rbd/parameters/single_major, assert that it's
not there for pre-single-major.yaml.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit f3bc8695221d777fc84489b724a69c9c5799484f)

Merge pull request #1164 from SUSE/wip-fix-make-rpm-jewel

buildpackages: backport make-rpm.sh improvements

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buildpackages/common.sh: use install-deps.sh from jewel

Significant changes took place between jewel and kraken. This commit ensures
that we use the jewel install-deps.sh to build jewel.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages/make-rpm.sh: install lsb_release dependencies

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages/make-rpm.sh: adjust Source0 spec file line on SUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 67de197ce51613922a9015b554be98b5a7b4c9be)

buildpackages: use unadulterated make-dist

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 29671063f9d2562c4e430ea12ce7e61ddf276f9e)

buildpackages: make make-{deb,rpm}.sh aware of cmake

Fixes: http://tracker.ceph.com/issues/16819
Signed-off-by: Ricardo Dias <rdias@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 65590aa039a9c14933a16654e26fcf47e8420199)

buildpackages/make-rpm.sh: use /etc/os-release

The lsb_release binary is deprecated and requires installation of packages.

The /etc/os-release file is guaranteed to be present.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 44d9acb67742db6f96f7daa5620d69afd657471f)

Merge pull request #1157 from SUSE/wip-suse-ceph-test-jewel

jewel: buildpackages: fix make-rpm.sh for SUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buildpackages: force ceph-test build on SUSE

The ceph-test package is required for teuthology. It is disabled to speed up
the build in OBS, but here we need it enabled unconditionally.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 0b10284caaf9dd00ef7fe66e3966e8e476341a62)

buildpackages: fix RPM generation script to support opensuse spec file

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 4692f995ba6967796d5dd2f5fd85fde8705a1ccc)
(cherry picked from commit 494b623602f0b09f10d788e6de6ddcf8d92700e3)

Merge pull request #1154 from ceph/wip-16539-jewel

jewel: rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 2dfe0852801cf0f90079859c0d67e75ef63de5ec)

Merge pull request #1096 from smithfarm/wip-16413-jewel

11429: wait_for_clean between restarting the osds and starting the bench

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #1147 from smithfarm/wip-17152-openstack-jewel

openstack: add disks where necessary

Reviewed-by: Loic Dachary <ldachary@redhat.com>

openstack: add disks where necessary

Ref: http://tracker.ceph.com/issues/17152

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 52017e6707c32fec38788a52f918480909b78ef5)

11429: wait_for_clean between restarting the osds and starting the bench

ceph.restart now marks the osds down, so the objects are actually being
created while slowest of the osds boots. That causes a ton of 1 byte
objects to be created in a degraded state and causes the cleanup to take
a long time.  Also, reduce length of bench since it's only being used
to ensure the osds came up correctly.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit c4ad6b3b339cc52be04a218fab31af21617146f7)

Merge pull request #1143 from smithfarm/wip-17116-jewel

workunit: allow parallel git clone

Merge pull request #1138 from ceph/wip-cd-jewel-backport

jewel backport for --dev-commit option

Update console checking, and DRY

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit bf797f0b1faed137eaef20b5a37cb683f50b0bff)

Drop unnecessary console handling code

Remote.console is provided by teuthology now
http://tracker.ceph.com/issues/17124
Fixes: 17124
Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 821cf2caa63518314e7b03afc834b32e2a8caaa0)

workunit: allow parallel git clone

When multiple client run in parallel on the same machine and they
try to get workunits from a repository that is not github, they must
git clone in a directory that is suffixed as srcdir. Otherwise they
will conflict with each other.

Fixes: http://tracker.ceph.com/issues/17116
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b9813e490ec8a5bb59e177612f047899397c342e)

Fix for 'branch' overrides to work

teuthology default override template defines devcommit which makes
'branch' overrides in the test useless since it only picksup one option
Remove the devcommit overrides from the template and use that as
default in case no other 'branch' override exist.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1134 from SUSE/wip-17075-jewel

Revert "packages.yaml: reflect python-ceph package split"

Revert "packages.yaml: reflect python-ceph package split"

This reverts commit ec4b0c3d6b057e688363d966a83a037c3585b32f.

References: http://tracker.ceph.com/issues/17075
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1120 from ceph/wip-rgw-jewel-17025

rgw: data_extra_pool is unique per zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: data_extra_pool is unique per zone

Fixes: http://tracker.ceph.com/issues/17025
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit c6cdd0905fbb3a9571400a5227b34b7e2e738830)

Merge pull request #1095 from smithfarm/wip-python-split-jewel

packages.yaml: reflect python-ceph package split

Merge pull request #1101 from SUSE/wip-osd-recovery-jewel

osd_recovery: add delay to ensure maps propogate

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #1100 from ajarr/wip-manila-backports-jewel

jewel: backports for OpenStack Manila

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #1103 from ceph/wip-fix-bad-check-jewel

backport - Wip fix bad check

tox.ini: exclude .tox dir from flake8

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 50f32756a472c29cc6c18a128e5b6cc99c9371e5)

Fix rechecking of health in loop

Use the newer stdout value for checking
ceph health again.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

osd_recovery: add delay to ensure maps propogate

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit ddabdb7f722b192435b87bc31fe587b93cc53924)

tasks/cephfs/test_volume_client: test authentication metadata

Test the usecases for the authentication metadata stored
by the volume client:

* Obtain the list of auth IDs having access to a volume.

* Restrict volume access to auth IDs of a single (OpenStack)
  tenant to enforce strong tenant isolation of volumes.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 08e3a49fa52d49a16585493d05cf51c2b31edece)

tasks/cephfs: test read-only authorization for volumes

Test whether mount access to volumes can be restricted to read-only.

Fixes: http://tracker.ceph.com/issues/15614
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit aab9a03c4c1d5ef98033b654e1a11da7ddf8aa9a)

tasks/cephfs: add function to configure guest credentials

In the test_volume_client module, refactor code to add a function
that configures credentials for a guest to mount a volume. The
function could be used by tests to setup a guest client to mount
a volume with fewer lines of code.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 02fc6d6acc1246881c173d693773658b210f1ad3)

packages.yaml: reflect python-ceph package split

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 0216330568fc1dec2e866bbfd1e49dc42266e1ab)

Merge pull request #1088 from ceph/jewel-mds-snap-failover

fs: add snapshot tests to mds thrashing

Reviewed-by: John Spray <john.spray@redhat.com>

fs: add snapshot tests to mds thrashing

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit b9b18c7b4ec17f1461b335e0c4ef3f36be70f74e)

Merge pull request #1071 from ceph/wip-16551-formatted-output

rbd: pull formatted-output.t from the relevant branch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: pull formatted-output.t from the relevant branch

Fixes http://tracker.ceph.com/issues/16551

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #1070 from smithfarm/wip-drop-ceph-devel-jewel

jewel: packages.yaml: drop ceph-devel

Reviewed-by: Loic Dachary <ldachary@redhat.com>

packages.yaml: drop ceph-devel

There are no "-dev" packages listed in the deb section of this file, so it
serves no purpose to have ceph-devel here.

Also, once https://github.com/ceph/ceph/pull/9744 is merged the presence of
"ceph-devel" in this file will break teuthology for rpm targets.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit af9e05709af76c931e4dd50349cac3a4fc87a575)

Merge pull request #1042 from ceph/wip-li-rados-jewel

Jewel: Backport changes needed for list-inconsistent-* format changes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #1056 from ceph/wip-rbd-nbd-jewel

jewel: rbd: added rbd-nbd fsx test case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: added rbd-nbd fsx test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7425e2d7a64e2962f6a7bc6f6ba09ee9137e7a14)

Merge pull request #1015 from ceph/wip-jewel-update-standby

(jewel) tasks/cephfs: update failover test for standby changes

Merge pull request #1016 from ceph/wip-jewel-multicluster-fixes

(jewel) backport cephfs fixes for test broken by multi-cluster support

Merge pull request #1047 from ceph/wip-16113-jewel

suites/rados: add test for 16113

Initial check in for a complete jewel-x suite

Fixes http://tracker.ceph.com/issues/15754
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 6ad42ef5796baed50c107877133128be4ae267f3)

scrub_test: Shards are not marked in error when osd guesses

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 68400e95524181f6bdd1a250b72b1cb38f543aea)

scrub_test: Handle list-inconsistent-obj changes in pull #8983

Add checking of error in the shards

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5992384700f335a970c1f7cb392d5b71cecfda5c)

Merge pull request #1035 from ceph/wip-sleep-jewel

[jewel]: tasks: fix non-existent sleep function

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

suites/rados: add test for 16113

Signed-off-by: Samuel Just <sjust@redhat.com>

tasks: fix non-existent sleep function

CephManager has no sleep function. Use time.sleep() instead.

Ran into this while testing a branch. Apparently it doesn't happen
much since this hasn't changed in years, but the error was copied
into several tasks.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit cfc48e36da74cd8b840270e388cc4b7e3efc1b58)

Merge pull request #1030 from ceph/wip-manila-backports

jewel: backports needed for Manila

Reviewed-by: John Spray <john.spray@redhat.com>

fs/volume_client: exercise the configurable prefix and ns_prefix.

Also ensure namespace_prefix and vol_prefix are set to default values
if they are not explicitly passed.

Fixes #15417

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
(cherry picked from commit 42f63dc6a6bfbe7667ad2293946fedeb097fabce)

tasks/cephfs: add TestVolumeClient.test_purge

For reproducing error handling paths during purge,
such as #15266 (ASCII exception)

Fixes: http://tracker.ceph.com/issues/15266
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 798c50e9bcd6c62b7e1687c85db718bca6883b3e)

tasks/cephfs: test 'df' output in volumeclient

Fixes: http://tracker.ceph.com/issues/15599
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 4e9635e8558aa69e66ec79b53469b03cd3eb12e6)

tasks/cephfs: make mount point more configurable

The mount points of the test clients were created based on the auth
ID they used. So two clients using the same auth ID could not create
different mount points in a straight forward way. Expose the
directory name of the mount point as a public attribute allowing
client mount points to be easily changed whenever needed.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 64869a3e4d5722e84f791ee3a07bf445d4402d32)
(cherry picked from commit dcdccfcd57aa6061620c8796c7cd90829cae03aa)

tasks/cephfs: test volume client eviction

Test whether the CephFSVolumeClient can evict a client based on its
auth ID and the volume path it has mounted.

Fixes: http://tracker.ceph.com/issues/15855
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit cbadb384f2e6f6fa2ebeab0d60af591a6de252a4)
(cherry picked from commit fab28a97e487ead9cc837ef73bb9798c1b24d99c)

suites: allow four remote clients for fs/recovery

test_volume_client requires four clients.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit b970f97dbc5af6e331282c4eb61c9faaff1e586e)
(cherry picked from commit f539587df992fc1b77196a43c8a54fab345fc0c9)

Merge pull request #1029 from ceph/wip-rbd-openstack

rbd: provision volumes to format as XFS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: provision volumes to format as XFS

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 45bd6cbf57c6772c47906f13eb27bf8586067213)

Merge pull request #1027 from ceph/wip-ctx-manager-jewel

tasks/create_verify_lfn_objects: adjust to new ctx.manager location

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tasks/create_verify_lfn_objects: adjust to new ctx.manager location

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 1cb2681cd02d7782c9dea602c9cf2b0fb80e6fae)

Merge pull request #1025 from ceph/wip-rbd-mirror-stress-jewel

jewel: suites/rbd: added replication stress test workunit

Reviewed-by: Jason Dillaman <dillaman@redhat.com>