tasks/workunit.py: depth 1 clone

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4faf77a649cb3f8ddf497ca81937b3dbf63a18dc)

tasks/workunit: remove kludge to use git.ceph.com

This was hard-coded to ceph.git (almost) and breaks when
you specify --ceph-repo.  Remove it entirely.  We'll see if
github.com is better at handling our load than it used to
be!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 159c455a0326eef2c017b3e3cf510f918b5ec76c)

tasks/ceph: restore context of osd mount path before mkfs

all newly created files and directories under the mount dir inherit the
SELinux type of their parent directory. so we need to set it before
mkfs.

Fixes: http://tracker.ceph.com/issues/16800
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 53225d5272a1d35d4183fcfa55a139f55f77e122)

Merge pull request #1290 from SUSE/wip-18014-jewel

thrashosds: try ceph-objectstore-tool for 10 minutes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

thrashosds: try ceph-objectstore-tool for 10 minutes

If ceph-objectstore-tool binary is not present, it's likely because we're in
the middle of an upgrade. Do not try to run the binary until we verify that
it's really present. If it is absent, spend up to 10 minutes waiting for it to
appear.

Before this patch there was quite a large window for a race to occur. This
patch doesn't entirely eliminate it, but drastically reduces it.

Fixes: http://tracker.ceph.com/issues/18014
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 862b47faac1fc9f05ee3322ee4b65cf3d3d666c5)

Merge pull request #1297 from ceph/wip-14.04

suites/rados: s/trusty/"14.04"/

suites/rados: s/trusty/"14.04"/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1296 from dachary/wip-ceph-coverage-jewel

upgrade: ceph-test is needed for ceph-coverage

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

upgrade/infernalis-client-x: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>

upgrade: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 8122494530589e27df929652e38c74137d6d823a)

Merge pull request #1284 from ceph/jewel-name-limits

drop broken name length config args

Merge pull request #1292 from ceph/jewel-avoid-xenial

rados: avoid xenial for upgrade tests

upgrade/client-upgrade: correct distros/ location

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/client-upgrade: fix distro symlinks

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/client-upgrade: specify centos or trusty (not xenial)

Signed-off-by: Sage Weil <sage@redhat.com>

rados: avoid ubuntu xenial on upgrade tests

Not all of the older package builds are present for xenial.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1152 from ceph/wip-objecstore

rados/objectstore/objectstore.yaml: skip bluestore tests

Merge pull request #1288 from dachary/wip-shec-upgrade-jewel

upgrade/hammer-x: verify shec before the full upgrade

Reviewed-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: verify shec before the full upgrade

The hammer-x/stress-split-erasure-code upgrade sequence comes from
hammer-x/stress-split and was modified to fully upgrade the cluster. It
previously upgraded only half of it. Verifying that the shec plugin is
not available and that trying to set it does not crash the OSD or the
MON must be tried before the upgrade is complete.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1287 from ceph/jewel-failed-to-encode

upgrade/hammer-x: encoding fixes (jewel)

Reviewed-by: Samuel Just <sjust@redhat.com>

upgrade/hammer-x/stress-split-*: disable sighup injection

We already did this for stress-split; do the same
here.  It triggers a File closed exception when the greenlet
is joined.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/parallel: white 'failed to encode'

The problem here has nothing to do with osdmap
encoding, but that hammer -> jewel makes the systemd
transition and installing the package starts
the mons.. before the osds.  I'm not sure what
the workaround for that is but the osdmap issue
appears okay, so ignore this for now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1285 from ceph/jewel-failed-to-encode

upgrade/hammer-x: do not whitelist 'failed to encode map'

upgrade/hammer-x/parallel: upgrade osds first

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: do not whitelist 'failed to encode map'

Well, on parallel.  For the others, keep it in
place because we don't upgrade osds first (we are testing
other things).

Signed-off-by: Sage Weil <sage@redhat.com>

drop broken name length config args

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2a44c3d20de9a75065c271e9ad8dfceeed1186d9)

upgrade/hammer-x: fix symlinks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1273 from ceph/wip-whitelist-crc

whitelist CRC mismatch

Merge pull request #1281 from ceph/wip-jewel-debug-fuse

upgrade/hammer-x: debug mds

upgrade/hammer-x: debug mds

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1280 from ceph/wip-jewel-debug-fuse

upgrade/hammer-x: debug client

upgrade/hammer-x: debug client

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1272 from ceph/wip-rgw-sync-agent-retry-jewel

jewel: rgw: fix some races with radosgw and radosgw-agent startup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tasks.rgw: 'time' imported but unused

flake8 was failing.

Signed-off-by: Owen Synge <osynge@suse.com>
(cherry picked from commit ef1d2a6aabe91282e28dbb6200fb7c2fab816720)

Restart OSDs that belong to first node only

Restart only first half osds as only the first node
is upgraded

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

This is triggering failures like

2016-11-18T01:17:08.865 INFO:tasks.ceph.osd.3:Stopping old one...
2016-11-18T01:17:08.865 DEBUG:tasks.ceph.osd.3:waiting for process to exit
2016-11-18T01:17:08.865 INFO:teuthology.orchestra.run:waiting for 300
2016-11-18T01:17:09.199 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/ceph-qa-suite_wip-whitelist-crc/tasks/ceph_manager.py", line 660, in wrapper
    return func(self)
  File "/home/teuthworker/src/ceph-qa-suite_wip-whitelist-crc/tasks/ceph_manager.py", line 677, in do_sighup
    self.ceph_manager.signal_osd(osd, signal.SIGHUP, silent=True)
  File "/home/teuthworker/src/ceph-qa-suite_wip-whitelist-crc/tasks/ceph_manager.py", line 1865, in signal_osd
    self.cluster).signal(sig, silent=silent)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/daemon.py", line 111, in signal
    self.proc.stdin.write(struct.pack('!b', sig))
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 377, in write
    raise IOError('File is closed')
IOError: File is closed

so trying to avoid this error.

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

whitelist CRC mismatch

whitelisted CRC mismatch and added upgrade for second
half of the cluster

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

rgw: remove unnecessary sleeps

remove the sleeps that were added to address radosgw startup races

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3e68bfdbb29e933edb06f73c88eed176ffacc2e3)

rgw: start_rgw() polls gateway until it accepts connections

resolves various races between radosgw startup and further operations -
both within the rgw task itself (such as the 'radosgw-admin realm pull'),
and in later tasks

Fixes: http://tracker.ceph.com/issues/17794
Fixes: http://tracker.ceph.com/issues/17872
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5e6538e623c0dea46203cd0c74201b7700f4767c)

rgw: add retry/backoff to sync agent requests

resolves an issue where startup of the radosgw-agent races with the
requests we send to it to run sync. uses the requests package with
urllib3 to add retry with backoff to these requests

Fixes: http://tracker.ceph.com/issues/16129
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 69bbafa804993d2a6cec608ac4f2eb4bfbb01753)

Merge pull request #1266 from ceph/wip-add-point-jewel

Added /upgrade/jewel-x/point-to-point-x

Added /upgrade/jewel-x/point-to-point-x

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1265 from ceph/wip-dont-whitelist

DO NOT whitelist CRC mismatch

DO NOT whitelist CRC mismatch

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

Merge pull request #1264 from ceph/jewel-osds-before-mon

Jewel osds before mon

upgrade/hammer-x: osds first

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x/f-h-x-offline: osds first

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1252 from SUSE/wip-17683

upgrade: disable ceph-objectstore-tool test in infernalis-x

Merge pull request #1256 from ceph/wip-17734-jewel

upgrade/hammer-x/stress-split: set require_jewel_osds

upgrade/hammer-x/stress-split: set require_jewel_osds

It was missing and the cluster permanently stays on WARNING state after
the upgrade of the OSDs.

Signed-off-by: Loic Dachary <loic@dachary.org>

upgrade: disable ceph-objectstore-tool test in infernalis-x

Fixes: http://tracker.ceph.com/issues/17683
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1245 from ceph/wip-fix-infernalis-s

Added require_jewel_osds flag

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Added require_jewel_osds flag
Added to point-to-point as well

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

suites/powercycle: no ext4

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2a71071bc132c8d14306f8ef4ba6ab043426c85a)

rgw: add sleep to let the sync agent to init

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit 54d311a991cfe658687d0f5a69f40718b7bea707)

rgw: add debug info when comparing bucket metadata

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit ad5a2a2f199f8fcecadf2c91b33c0530a26d8c3d)

Merge pull request #1240 from ceph/wip-17734-jewel

upgrade/hammer-x: wait for osdmaps to propagate

upgrade/hammer-x: wait for osdmaps to propagate

Fixes: http://tracker.ceph.com/issues/17808
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1226 from dachary/wip-17734-jewel

releases/jewel: set require_jewel_osds

upgrade/hammer-x: set require_jewel_osds

And replace infernalis with jewel where relevant.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

rados/singleton-nomsgr/all: set require_jewel_osds

those tests do not exist anymore in master, no backport possible.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #1174 from ceph/wip-16516

jewel: test: Eliminate racey scrubbing in scrub_test.py

test: Eliminate racey scrubbing in scrub_test.py

Enhance get_pgid() to return passed value if it is already a pgid

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 29985efc7cdff252a822d31b5e18b9163849bc53)

Fixes: http://tracker.ceph.com/issues/16516

scrub_test: Fixes need with wip-13507

Accept an attribute if not present as False
Change required due change to dict with "epoch" and "inconsistents"

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 4d0c668e9eb50c2149cfe36caee37c531541d1be)

Revert "check for inconsistent objects count instead of all keys"

This reverts commit e94f3901928501586f44c35612bc8f2dc504f465.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #1225 from ceph/wip-inconsistent

assert based on inconsistent objects count instead of all keys

check for inconsistent objects count instead of all keys

epoch keys also exist in the objs dump, count only
inconsistent objects

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1214 from jcsp/wip-generate-caps-jewel

tasks/ceph: move generate_caps from teuthology

tasks/ceph: move generate_caps from teuthology

This was only used in this task, and it is much too
ceph-specific to belong in teuthology.

Fixes: http://tracker.ceph.com/issues/17614
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 298cc8f932a2a327e07f5b73708404588a611322)

Merge pull request #1203 from dachary/wip-17537-jewel

suites/rbd: increase timeout for rbd-mirror stress test

suites/rbd: increase timeout for rbd-mirror stress test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 525e445f5f58388edf1523859f8f5d4d6cf4aea5)

Merge pull request #1201 from dachary/wip-17344-jewel

upgrade: ceph-test is needed for ceph-coverage

upgrade: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit d44f583436f958e276e4d4819a722ed39f97c961)

Merge pull request #1191 from ceph/wip-offline-split-jewel

ceph_manager: test offline split via ceph-objectstore-tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph_manager: test offline split via ceph-objectstore-tool

When killing an osd, split all pools with a low threshold.
This will slow down tests, but should not impact correctness.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit f28f881bda0ad426bfa9c0e22cc4658b2ce2ec5d)

Merge pull request #1181 from ceph/wip-jewel-release

Add ceph-deploy tests for released builds

Merge pull request #1183 from ceph/wip-librbd-disable-helgrind-jewel

jewel: suites/rbd: remove helgrind test cases

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

suites/rbd: remove helgrind test cases

Helgrind cannot properly handle Ceph's use of std::mutex --
resulting in numerous false positives and potential assertion
failures.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b29bf9253d7d2201f4cf01b0fd6997ed1bae231f)

Merge pull request #1182 from ceph/wip-smoke-krbd-iozone-jewel

jewel: smoke: mask out unsupported image features for krbd iozone test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

smoke: mask out unsupported image features for krbd iozone test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 77949e5cfc1231c96417e90aadc1573137c4b4d7)

Add basic testing of ceph bits

simple test suite to test ceph from download.ceph.com

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1173 from ceph/wip-16717-jewel

jewel: suites/upgrade: override rbd_default_features config back to pre-Jewel

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

suites/upgrade: override rbd_default_features config back to pre-Jewel

Fixes: http://tracker.ceph.com/issues/16717
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 47a41286a5506a2b0e036c3e75c99911f417e74b)

Merge pull request #1167 from ceph/wip-17222-jewel

Backport krbd/unmap fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

krbd/unmap: put client.0 on a separate remote

Otherwise a pre-single-major kernel override is a headache,
particularly with non-standard yaml configs.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 311a450163cf31d26cdd585faf8e306b19731bb1)

krbd/unmap: override client.0 only

The unmap test uses one remote, so the end result is the same.
However, overriding the most specific role is nicer and allows
scheduling with

  kernel:
    client:
       branch: testing

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 02460f2156cf507c2be94e6fb1ad8af0b3be245a)

krbd/unmap: set tunables to bobtail

Kernel 3.13, which is used in pre-single-major.yaml test, doesn't
support firefly tunables (default in jewel, up from bobtail tunables).
This went unnoticed for a while because of a kernel task regression -
the pre-single-major override was ignored.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 39883e9d9f7267f92dacb40cee10794503c88539)

krbd/unmap: assert that pre-single-major kernel is installed

Similarly to how single-major-off.yaml and single-major-on.yaml check
the value of /sys/module/rbd/parameters/single_major, assert that it's
not there for pre-single-major.yaml.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit f3bc8695221d777fc84489b724a69c9c5799484f)

Merge pull request #1164 from SUSE/wip-fix-make-rpm-jewel

buildpackages: backport make-rpm.sh improvements

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buildpackages/common.sh: use install-deps.sh from jewel

Significant changes took place between jewel and kraken. This commit ensures
that we use the jewel install-deps.sh to build jewel.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages/make-rpm.sh: install lsb_release dependencies

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buildpackages/make-rpm.sh: adjust Source0 spec file line on SUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 67de197ce51613922a9015b554be98b5a7b4c9be)

buildpackages: use unadulterated make-dist

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 29671063f9d2562c4e430ea12ce7e61ddf276f9e)

buildpackages: make make-{deb,rpm}.sh aware of cmake

Fixes: http://tracker.ceph.com/issues/16819
Signed-off-by: Ricardo Dias <rdias@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 65590aa039a9c14933a16654e26fcf47e8420199)

buildpackages/make-rpm.sh: use /etc/os-release

The lsb_release binary is deprecated and requires installation of packages.

The /etc/os-release file is guaranteed to be present.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 44d9acb67742db6f96f7daa5620d69afd657471f)

Merge pull request #1157 from SUSE/wip-suse-ceph-test-jewel

jewel: buildpackages: fix make-rpm.sh for SUSE

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buildpackages: force ceph-test build on SUSE

The ceph-test package is required for teuthology. It is disabled to speed up
the build in OBS, but here we need it enabled unconditionally.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 0b10284caaf9dd00ef7fe66e3966e8e476341a62)

buildpackages: fix RPM generation script to support opensuse spec file

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 4692f995ba6967796d5dd2f5fd85fde8705a1ccc)
(cherry picked from commit 494b623602f0b09f10d788e6de6ddcf8d92700e3)

Merge pull request #1154 from ceph/wip-16539-jewel

jewel: rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 2dfe0852801cf0f90079859c0d67e75ef63de5ec)