rgw: start_rgw() polls gateway until it accepts connections

resolves various races between radosgw startup and further operations -
both within the rgw task itself (such as the 'radosgw-admin realm pull'),
and in later tasks

Fixes: http://tracker.ceph.com/issues/17794
Fixes: http://tracker.ceph.com/issues/17872
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add retry/backoff to sync agent requests

resolves an issue where startup of the radosgw-agent races with the
requests we send to it to run sync. uses the requests package with
urllib3 to add retry with backoff to these requests

Fixes: http://tracker.ceph.com/issues/16129
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1263 from ceph/wip-cephfs-perm

rados/singleton-nomsgr/all/cache-fs-trunc: debug

rados/singleton-nomsgr/all/cache-fs-trunc: debug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1258 from ceph/jewel-x-fixes

rados/upgrade: skip libcephfs-devel install

Reviewed-by: Dan Mick <dmick@redhat.com>

rados/upgrade: skip libcephfs-devel install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1253 from ceph/wip-vive-la-libcephfs-devel

packages: s/libcephfs2-devel/libcephfs-devel/

packages: s/libcephfs2-devel/libcephfs-devel/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1251 from ceph/distros

distros/supported: add xenial

rados/singleton/all/mon-seesaw: give mons more time to stabilize

10s wasn't long enough for one run to get pgs created.

Signed-off-by: Sage Weil <sage@redhat.com>

distros/supported: add xenial

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1249 from ceph/radostool-osds

rados/singleton: need 3 osds for radostool test

rados/singleton: need 3 osds for radostool test

There's a k=2 m=1 ec pool in the test!

Fixes: http://tracker.ceph.com/issues/17775
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1248 from ceph/scrub-vs-recovery

rados/thrash: test osd_scrub_during_recovery=false

rados/thrash: test osd_scrub_during_recovery = false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1246 from ceph/wip-powercycle

suites/powercycle: skip ext4; add bluestore

suites/powercycle: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

suites/powercycle: no ext4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1224 from jcsp/wip-hadoop-on-ubuntu

suites/hadoop: run only on ubuntu

Merge pull request #1238 from ceph/rados-upgrades

rados/singleton-nomsgr: drop hammer, infernalis upgrades

upgrade/jewel-x: skip libcephfs2 install

None of these tests currently cover fs, so this is
sufficient... :/

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr: drop hammer, infernalis upgrades

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1175 from batrick/multimds-thrash

MDS thrasher for multimds

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'origin/pull/1/head' into multimds-thrash

* origin/pull/1/head:
  tasks/vstart_runner: fix default run-all
  tasks/vstart_runner: update for Filesystem changes

rados/upgrade: skip libcephfs2

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/vstart_runner: fix default run-all

This broke when path changed.  It's a bit
weird to run the cephfs tests by default
now that vstart_runner is not cephfs-specific,
but for the moment just restore the old
behaviour.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/vstart_runner: update for Filesystem changes

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mds_thrash: support multimds

This commit amends the MDS thrasher task to also work on multimds
clusters. Main changes:

o New FSStatus class in tasks/cephfs/filesystem.py which gets a snapshot
  of the fsmap (`ceph fs dump`). This allows consecutive operations on
  the same fsmap without repeated fs dumps.

o Only one MDSThrasher is started for each file system.

o The MDSThrasher operates on ranks instead of names (and groups of
  standbys following the initial active).

o The MDSThrasher also will change the max_mds for the cluster to a new
  value [1, current) or (current, starting max_mds]. When reduced,
  randomly selected MDSs other than rank 0 will be deactivated to reach
  the new max_mds. The likelihood of changing max_mds in a given cycle of
  the MDSThrasher is set by the "thrash_max_mds" config.

o The MDSThrasher prints out stats on completion, e.g. number of
  mds deactivated or mds_max changed.

Pre-requisite for: http://tracker.ceph.com/issues/10792
Partially fixes: http://tracker.ceph.com/issues/15134

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1223 from ceph/wip-jlayton-libcephfs2

libcephfs1 -> libcephfs2 rename

Merge pull request #1236 from ceph/wip-upgrade-mgr

rados/upgrade: skip ceph-mgr on jewel install

rados/upgrade: skip ceph-mgr on jewel install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1234 from jcsp/wip-fixup-cephfs

Fix running cephfs python tests without mgr daemon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph_test_case: only construct needed parts

Don't construct Filesystem and MDSCluster if there
are no MDSs in the system, don't construct MgrCluster
if there are no mgrs in the system.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/vstart_runner: fix paths in docstring

Should have been updated when this file moved.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mgr: remove duplicated method definition

This was already defined in parent CephCluster class.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1229 from ceph/wip-jewel-upgrades

fix ceph-mgr install for upgrade tests

upgrade/jewel-x/point-to-point: fix upgrades

skip ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1233 from ceph/wip-17759

Added short pg log entries to rgw test

Reviewed-by: Samuel Just <sjust@redhat.com>

Added short pg log entries to rgw test

This change aims to test rgw with messenger failure injection
and very short logs.

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

objectstore/bluestore: disable bluefs env mirror

We can't atomically append to two env backends, which means
that we can see size mismatches when we inject failures.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1232 from ceph/wip-blackhole

tasks: use new generic objectstore_blackhole option

tasks: use new generic objectstore_blackhole option

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/client-upgrades: exclude ceph-mgr from old installs

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly-hammer-x: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis*: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x: skip ceph-mgr install

Signed-off-by: Sage Weil <sage@redhat.com>

rados/objectstore/objectstore: suppress core dumps

Some of these tests to fork and assertion tests that leave
core behind, and that is normal.  Disable core dumps.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1086 from jcsp/wip-ceph-mgr

Create ceph-mgr tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks: update cephfs_test_runner for mgr

Signed-off-by: John Spray <john.spray@redhat.com>

packages: install ceph-mgr

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: update to run ceph-mgr daemons

Signed-off-by: John Spray <john.spray@redhat.com>

suites: create mgr yaml

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: create ceph-mgr tests

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: refactor test case class

A more generic CephTestCase and CephCluster, for
writeing non-cephfs test cases.

This avoids overloading one class with the functionality
needed by lots of different subsystems.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1219 from ceph/wip-cephfs-scan-links

tasks/cephfs: add TestDataScan.test_scan_links

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fixup test_backtrace_repair

The branches got mixed up and the merged one wasn't
the same one that was tested.  This is the one that
works!

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1220 from ceph/wip-10235

Remove obsolete calamari code

suites/hadoop: run only on ubuntu

Workaround for http://tracker.ceph.com/issues/16640

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1145 from michaelsevilla/mantle

mds: add tests for mantle (programmable balancer)

Reviewed-by: John Spray <john.spray@redhat.com>

Remove obsolete calamari code

It's dead, Jim.

Fixes: http://tracker.ceph.com/issues/10235
Signed-off-by: Dan Mick <dan.mick@redhat.com>

tasks/cephfs: add TestDataScan.test_scan_links

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #1218 from jcsp/wip-backtrace-repair

tasks/cephfs: add TestForwardScrub.test_backtrace_repair

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

libcephfs1 -> libcephfs2 rename

Signed-off-by: Jeff Layton <jlayton@redhat.com>

tasks/cephfs: add TestForwardScrub.test_backtrace_repair

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1216 from ceph/wip-pg-files

tasks/cephfs: add TestDataScan.test_pg_files

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1217 from ceph/wip-17558

mon-seesaw: add mon-seesaw test and mon_seesaw task

Reviewed-by: Sam Just <sjust@redhat.com>

mon-seesaw: add mon-seesaw test and mon_seesaw task

mon_seesaw task replaces a monitor with a newly reployed one, in a
single-mon test including this task OSDs will not be able to connect
to cluster if the tracker#17558 is not fixed on the monitor side.

http://tracker.ceph.com/issues/17558
Signed-off-by: Kefu Chai <kchai@redhat.com>

tasks/cephfs: add TestDataScan.test_pg_files

Fixes: http://tracker.ceph.com/issues/17249
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1213 from jcsp/wip-generate-caps

tasks/ceph: move generate_caps from teuthology

Merge pull request #1202 from ajarr/wip-check-volume-size

test_volume_client: check volume size

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #1166 from ajarr/wip-17216

tasks/cephfs : test recovery of partial auth update

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/ceph: move generate_caps from teuthology

This was only used in this task, and it is much too
ceph-specific to belong in teuthology.

Fixes: http://tracker.ceph.com/issues/17614
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1212 from ceph/wip-osd-recovery

rados/singleton/all/osd-recovery*: disable fast fail

rados/singleton/all/osd-recovery*: disable fast fail

This test relies on being able to kill two OSDs before
they detect the failures on their own.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1210 from ceph/wip-add-jewel-x

Added jewel-x-singleton

Added jewel-x-singleton

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1209 from ceph/wip-rados-upgrade

rados/upgrade: remove hammer-x-singleton

rados/upgrade: remove hammer-x-singleton

Upgrades must stop at jewel now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1208 from ceph/wip-rgw-min-size

rados: move rgw_snaps from thrash to basic

rados: move rgw_snaps from thrash to basic

rgw does not tolerate min_size=2 + size=2 thrashing (slow
rados ops).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1207 from ceph/wip-kraken

more kraken upgrade fixes

Merge pull request #1200 from ceph/wip-rebuild-mondb

tasks/rebuild_mondb.py: fill up the caps in the keyring

upgrade/jewel-x: fix require_kraken_osds flag

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x: drop 'failed to encode' crc mismatch whitelist

We should no longer trigger these warnings now that we
build the OSDMap based on require_*_osds flags.

Signed-off-by: Sage Weil <sage@redhat.com>

rados: fix require_kraken_osds flag

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr: drop 16113 and 13234

A stop at jewel is required, and these are pre-jewel bugs.

Signed-off-by: Sage Weil <sage@redhat.com>

rados: infernalis -> kraken is not supported; stop at jewel

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1206 from ceph/wip-thrash-min-size-2

rados: move readwrite test out of thrash matrix

rados: move readwrite test out of thrash matrix

The thrash matrix includes min_size = 2 and size = 2.  This
is fine and good, except that the inevitable slow rados ops
that generates make readwrite.yaml issue timeouts and
fail.  Since the workload isn't meant to tolerate slow
requests, move it into the basic/ group.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1204 from ceph/wip-jd-testing

suites/upgrade: additional point-to-point Jewel upgrade workaround

Merge pull request #1205 from ceph/wip-kraken

upgrade: set kraken osd flag on upgrade

Reviewed-by: Kefu Chai <kchai@redhat.com>

upgrade/jewel-x: set kraken flag when osds finish

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr: set require_*_osds in upgrade tests

Signed-off-by: Sage Weil <sage@redhat.com>

test_volume_client: remove superfluous arguments

Signed-off-by: Ramana Raja <rraja@redhat.com>

test_volume_client: check volume size

Check that the total size shown by the df output of a mounted volume
is same as the volume size and the quota set on the volume.

Signed-off-by: Ramana Raja <rraja@redhat.com>

suites/upgrade: additional point-to-point Jewel upgrade workaround

Upgrade happens in parallel with cls_rbd test run which can result in
a false-positive test failure.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tasks/rebuild_mondb.py: fill up the caps in the keyring

we will fail 'ceph-monstore-tool' command if the caps is empty so user
won't assign a key without any caps when rebuilding the monstore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

releases/kraken.yaml: set require_kraken_osds

Signed-off-by: Sage Weil <sage@redhat.com>

releases/jewel: set require_jewel_osds

Signed-off-by: Sage Weil <sage@redhat.com>