tasks/cephfs: fix wait_until_true logging

The success log message was in the wrong
place (and unreachable)

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1274 from ceph/wip-16064

Improve scrub analysis, list-inconsistent-obj output and osd-scrub-repair test

Merge pull request #1260 from ceph/wip-17902-master

Added excluded packages "'libcephfs-devel','libcephfs-dev'"

Merge pull request #1227 from athanatos/wip-ec-overwrites

ec overwrites tests (and bonus fix to fast reads)

Merge pull request #1271 from SUSE/wip_flake8_F401_time_imported_but_unused

tasks.rgw: 'time' imported but unused

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks.rgw: 'time' imported but unused

flake8 was failing.

Signed-off-by: Owen Synge <osynge@suse.com>

Added excluded packadges "'libcephfs-devel','libcephfs-dev'"

Fixes http://tracker.ceph.com/issues/17902
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/scrub_test.py: Make test deterministic by updating digests

ceph_test_rados: Get coverage of digest updates in deep-scrub/repair

Signed-off-by: David Zafman <dzafman@redhat.com>

repair_test, scrub_test: Fix whitelists for scrub changes

Signed-off-by: David Zafman <dzafman@redhat.com>

scrub_test: Fix for list-inconsistent-obj output changes

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #1269 from ceph/wip-rgw-sync-agent-retry

rgw: fix some races with radosgw and radosgw-agent startup

(verified by Orit)

rgw: remove unnecessary sleeps

remove the sleeps that were added to address radosgw startup races

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start_rgw() polls gateway until it accepts connections

resolves various races between radosgw startup and further operations -
both within the rgw task itself (such as the 'radosgw-admin realm pull'),
and in later tasks

Fixes: http://tracker.ceph.com/issues/17794
Fixes: http://tracker.ceph.com/issues/17872
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add retry/backoff to sync agent requests

resolves an issue where startup of the radosgw-agent races with the
requests we send to it to run sync. uses the requests package with
urllib3 to add retry with backoff to these requests

Fixes: http://tracker.ceph.com/issues/16129
Signed-off-by: Casey Bodley <cbodley@redhat.com>

objectstore.yaml: raise fd limit

Signed-off-by: Samuel Just <sjust@redhat.com>

fast.yaml: overrides not override

Signed-off-by: Samuel Just <sjust@redhat.com>

thrash-erasure-code: disable ceph-objectstore-tool testing for now

Signed-off-by: Samuel Just <sjust@redhat.com>

rados,thrash-erasure-code: add overwrites workloads

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1263 from ceph/wip-cephfs-perm

rados/singleton-nomsgr/all/cache-fs-trunc: debug

rados/singleton-nomsgr/all/cache-fs-trunc: debug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1258 from ceph/jewel-x-fixes

rados/upgrade: skip libcephfs-devel install

Reviewed-by: Dan Mick <dmick@redhat.com>

rados/upgrade: skip libcephfs-devel install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1253 from ceph/wip-vive-la-libcephfs-devel

packages: s/libcephfs2-devel/libcephfs-devel/

packages: s/libcephfs2-devel/libcephfs-devel/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1251 from ceph/distros

distros/supported: add xenial

rados/singleton/all/mon-seesaw: give mons more time to stabilize

10s wasn't long enough for one run to get pgs created.

Signed-off-by: Sage Weil <sage@redhat.com>

distros/supported: add xenial

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1249 from ceph/radostool-osds

rados/singleton: need 3 osds for radostool test

rados/singleton: need 3 osds for radostool test

There's a k=2 m=1 ec pool in the test!

Fixes: http://tracker.ceph.com/issues/17775
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1248 from ceph/scrub-vs-recovery

rados/thrash: test osd_scrub_during_recovery=false

rados/thrash: test osd_scrub_during_recovery = false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1246 from ceph/wip-powercycle

suites/powercycle: skip ext4; add bluestore

suites/powercycle: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

suites/powercycle: no ext4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1224 from jcsp/wip-hadoop-on-ubuntu

suites/hadoop: run only on ubuntu

Merge pull request #1238 from ceph/rados-upgrades

rados/singleton-nomsgr: drop hammer, infernalis upgrades

upgrade/jewel-x: skip libcephfs2 install

None of these tests currently cover fs, so this is
sufficient... :/

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr: drop hammer, infernalis upgrades

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1175 from batrick/multimds-thrash

MDS thrasher for multimds

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'origin/pull/1/head' into multimds-thrash

* origin/pull/1/head:
  tasks/vstart_runner: fix default run-all
  tasks/vstart_runner: update for Filesystem changes

rados/upgrade: skip libcephfs2

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/vstart_runner: fix default run-all

This broke when path changed.  It's a bit
weird to run the cephfs tests by default
now that vstart_runner is not cephfs-specific,
but for the moment just restore the old
behaviour.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/vstart_runner: update for Filesystem changes

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mds_thrash: support multimds

This commit amends the MDS thrasher task to also work on multimds
clusters. Main changes:

o New FSStatus class in tasks/cephfs/filesystem.py which gets a snapshot
  of the fsmap (`ceph fs dump`). This allows consecutive operations on
  the same fsmap without repeated fs dumps.

o Only one MDSThrasher is started for each file system.

o The MDSThrasher operates on ranks instead of names (and groups of
  standbys following the initial active).

o The MDSThrasher also will change the max_mds for the cluster to a new
  value [1, current) or (current, starting max_mds]. When reduced,
  randomly selected MDSs other than rank 0 will be deactivated to reach
  the new max_mds. The likelihood of changing max_mds in a given cycle of
  the MDSThrasher is set by the "thrash_max_mds" config.

o The MDSThrasher prints out stats on completion, e.g. number of
  mds deactivated or mds_max changed.

Pre-requisite for: http://tracker.ceph.com/issues/10792
Partially fixes: http://tracker.ceph.com/issues/15134

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1223 from ceph/wip-jlayton-libcephfs2

libcephfs1 -> libcephfs2 rename

Merge pull request #1236 from ceph/wip-upgrade-mgr

rados/upgrade: skip ceph-mgr on jewel install

rados/upgrade: skip ceph-mgr on jewel install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1234 from jcsp/wip-fixup-cephfs

Fix running cephfs python tests without mgr daemon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph_test_case: only construct needed parts

Don't construct Filesystem and MDSCluster if there
are no MDSs in the system, don't construct MgrCluster
if there are no mgrs in the system.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/vstart_runner: fix paths in docstring

Should have been updated when this file moved.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mgr: remove duplicated method definition

This was already defined in parent CephCluster class.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1229 from ceph/wip-jewel-upgrades

fix ceph-mgr install for upgrade tests

upgrade/jewel-x/point-to-point: fix upgrades

skip ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1233 from ceph/wip-17759

Added short pg log entries to rgw test

Reviewed-by: Samuel Just <sjust@redhat.com>

Added short pg log entries to rgw test

This change aims to test rgw with messenger failure injection
and very short logs.

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

objectstore/bluestore: disable bluefs env mirror

We can't atomically append to two env backends, which means
that we can see size mismatches when we inject failures.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1232 from ceph/wip-blackhole

tasks: use new generic objectstore_blackhole option

tasks: use new generic objectstore_blackhole option

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/client-upgrades: exclude ceph-mgr from old installs

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly-hammer-x: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis*: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x: skip ceph-mgr install

Signed-off-by: Sage Weil <sage@redhat.com>

rados/objectstore/objectstore: suppress core dumps

Some of these tests to fork and assertion tests that leave
core behind, and that is normal.  Disable core dumps.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1086 from jcsp/wip-ceph-mgr

Create ceph-mgr tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks: update cephfs_test_runner for mgr

Signed-off-by: John Spray <john.spray@redhat.com>

packages: install ceph-mgr

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: update to run ceph-mgr daemons

Signed-off-by: John Spray <john.spray@redhat.com>

suites: create mgr yaml

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: create ceph-mgr tests

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: refactor test case class

A more generic CephTestCase and CephCluster, for
writeing non-cephfs test cases.

This avoids overloading one class with the functionality
needed by lots of different subsystems.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1219 from ceph/wip-cephfs-scan-links

tasks/cephfs: add TestDataScan.test_scan_links

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fixup test_backtrace_repair

The branches got mixed up and the merged one wasn't
the same one that was tested.  This is the one that
works!

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1220 from ceph/wip-10235

Remove obsolete calamari code

suites/hadoop: run only on ubuntu

Workaround for http://tracker.ceph.com/issues/16640

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1145 from michaelsevilla/mantle

mds: add tests for mantle (programmable balancer)

Reviewed-by: John Spray <john.spray@redhat.com>

Remove obsolete calamari code

It's dead, Jim.

Fixes: http://tracker.ceph.com/issues/10235
Signed-off-by: Dan Mick <dan.mick@redhat.com>

tasks/cephfs: add TestDataScan.test_scan_links

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #1218 from jcsp/wip-backtrace-repair

tasks/cephfs: add TestForwardScrub.test_backtrace_repair

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

libcephfs1 -> libcephfs2 rename

Signed-off-by: Jeff Layton <jlayton@redhat.com>

tasks/cephfs: add TestForwardScrub.test_backtrace_repair

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1216 from ceph/wip-pg-files

tasks/cephfs: add TestDataScan.test_pg_files

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #1217 from ceph/wip-17558

mon-seesaw: add mon-seesaw test and mon_seesaw task

Reviewed-by: Sam Just <sjust@redhat.com>

mon-seesaw: add mon-seesaw test and mon_seesaw task

mon_seesaw task replaces a monitor with a newly reployed one, in a
single-mon test including this task OSDs will not be able to connect
to cluster if the tracker#17558 is not fixed on the monitor side.

http://tracker.ceph.com/issues/17558
Signed-off-by: Kefu Chai <kchai@redhat.com>

tasks/cephfs: add TestDataScan.test_pg_files

Fixes: http://tracker.ceph.com/issues/17249
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1213 from jcsp/wip-generate-caps

tasks/ceph: move generate_caps from teuthology

Merge pull request #1202 from ajarr/wip-check-volume-size

test_volume_client: check volume size

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #1166 from ajarr/wip-17216

tasks/cephfs : test recovery of partial auth update

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/ceph: move generate_caps from teuthology

This was only used in this task, and it is much too
ceph-specific to belong in teuthology.

Fixes: http://tracker.ceph.com/issues/17614
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1212 from ceph/wip-osd-recovery

rados/singleton/all/osd-recovery*: disable fast fail

rados/singleton/all/osd-recovery*: disable fast fail

This test relies on being able to kill two OSDs before
they detect the failures on their own.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1210 from ceph/wip-add-jewel-x

Added jewel-x-singleton

Added jewel-x-singleton

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1209 from ceph/wip-rados-upgrade

rados/upgrade: remove hammer-x-singleton

rados/upgrade: remove hammer-x-singleton

Upgrades must stop at jewel now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1208 from ceph/wip-rgw-min-size

rados: move rgw_snaps from thrash to basic

rados: move rgw_snaps from thrash to basic

rgw does not tolerate min_size=2 + size=2 thrashing (slow
rados ops).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1207 from ceph/wip-kraken

more kraken upgrade fixes

Merge pull request #1200 from ceph/wip-rebuild-mondb

tasks/rebuild_mondb.py: fill up the caps in the keyring

upgrade/jewel-x: fix require_kraken_osds flag

Signed-off-by: Sage Weil <sage@redhat.com>