Merge pull request #1301 from ceph/wip-valgrind-centos

do all valgrind runs on centos

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

do all valgrind runs on centos

The xenial valgrind has a bug with generating random
numbers; see http://tracker.ceph.com/issues/18126

Revert this when it is fixed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1299 from athanatos/wip-enable-hacky-flag

ec-data-pool: enable the hacky_overwrites flag

ec-data-pool: enable the hacky_overwrites flag

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1298 from ceph/wip-pre-single-major-branch

krbd/unmap: pre-single-major is now 3.12.z

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rados/thrash/workloads: shorten rados bench a bit

I hit ENOSPC during thrashing on one run.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1295 from dachary/wip-16506-ceph-coverage

upgrade: ceph-test is needed for ceph-coverage

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

upgrade: ceph-test is needed for ceph-coverage

Do not exclude the ceph-test package otherwise the ceph-coverage
executable is not installed.

Fixes: http://tracker.ceph.com/issues/16506
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1294 from ceph/wip-17960-master

Removed xenial + Excluded librgw2

upgrade/jewel-x/point-to-point: avoid xenial

Xenial has newer packages in the distro and we want to
avoid fragile install instructions to avoid them.

Signed-off-by: Sage Weil <sage@redhat.com>

Excluded librgw2
Fixes http://tracker.ceph.com/issues/17960

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

krbd/unmap: pre-single-major is now 3.12.z

3.13 won't build on newer distros, due to lack of gcc5 support.
nightly/pre-single-major (with a slash) is now gone.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1293 from ceph/wip-17977-fix

qemu: Set permissions on /dev/kvm

Reviewed-by: Dan Mick <dmick@redhat.com>

qemu: Set permissions on /dev/kvm

Hack until fix in https://bugzilla.redhat.com/show_bug.cgi?id=1333159
makes it into CentOS repos.

Fixes: http://tracker.ceph.com/issues/17977
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1291 from ceph/wip-18085

rados/thrash*: do no set osd_max_backfills=1

Reviewed-by: Samuel Just <sjust@redhat.com>

rados/thrash*: vary osd_max_backfills, always >1

This can lead to a copy-from vs backfill deadlock; see
http://tracker.ceph.com/issues/18085

This effectively reverts 5e880228fd7f59063d22a51eb5488b369b0c8360
which aimed to catch bugs in recovery reservations.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #894 from ceph/wip-bluestore

rados: test bluestore too

rados/objectstore/objectstore.yaml: do not test kstore

It's flaky.

Signed-off-by: Sage Weil <sage@redhat.com>

rados/objectstore/objectstore.yaml: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: test on filestore and bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

fs: test bluestore and filestore

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: test bluestore and filestore

Signed-off-by: Sage Weil <sage@redhat.com>

rados: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1289 from ceph/wip-16397-workaround

suites/knfs: pin to ubuntu

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #1283 from SUSE/wip-18014

thrashosds: try ceph-objectstore-tool for one minute

Reviewed-by: Loic Dachary <ldachary@redhat.com>

suites/knfs: pin to ubuntu

Workaround for http://tracker.ceph.com/issues/16397, which
manifests when we run bleeding edge kernels against the version
of nfs-utils in centos7.

Signed-off-by: John Spray <john.spray@redhat.com>

thrashosds: try ceph-objectstore-tool for 10 minutes

If ceph-objectstore-tool binary is not present, it's likely because we're in
the middle of an upgrade. Do not try to run the binary until we verify that
it's really present. If it is absent, spend up to 10 minutes waiting for it to
appear.

Before this patch there was quite a large window for a race to occur. This
patch doesn't entirely eliminate it, but drastically reduces it.

Fixes: http://tracker.ceph.com/issues/18014
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1235 from vshankar/wip-rbd-data-pool

suites/rbd: add data pool test cases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1277 from ceph/wip-exclude-upgrade-clients

Excluded 'libcephfs2','libcephfs-devel','libcephfs-dev'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Excluded 'libcephfs2','libcephfs-devel','libcephfs-dev'
Removed firefly and infernalis paths
Removed mds role from hammer (per chat with Jason)
Add "fs: xfs" override

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rbd: add datapool config YAML to rbd suite

- cli
- librbd
- qemu

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #1262 from batrick/cephfs-delete-bugfix

cephfs: get new fsmap after marking clusters down

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #1274 from ceph/wip-16064

Improve scrub analysis, list-inconsistent-obj output and osd-scrub-repair test

Merge pull request #1260 from ceph/wip-17902-master

Added excluded packages "'libcephfs-devel','libcephfs-dev'"

Merge pull request #1227 from athanatos/wip-ec-overwrites

ec overwrites tests (and bonus fix to fast reads)

Merge pull request #1271 from SUSE/wip_flake8_F401_time_imported_but_unused

tasks.rgw: 'time' imported but unused

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks.rgw: 'time' imported but unused

flake8 was failing.

Signed-off-by: Owen Synge <osynge@suse.com>

Added excluded packadges "'libcephfs-devel','libcephfs-dev'"

Fixes http://tracker.ceph.com/issues/17902
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

tasks/scrub_test.py: Make test deterministic by updating digests

ceph_test_rados: Get coverage of digest updates in deep-scrub/repair

Signed-off-by: David Zafman <dzafman@redhat.com>

repair_test, scrub_test: Fix whitelists for scrub changes

Signed-off-by: David Zafman <dzafman@redhat.com>

scrub_test: Fix for list-inconsistent-obj output changes

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #1269 from ceph/wip-rgw-sync-agent-retry

rgw: fix some races with radosgw and radosgw-agent startup

(verified by Orit)

rgw: remove unnecessary sleeps

remove the sleeps that were added to address radosgw startup races

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start_rgw() polls gateway until it accepts connections

resolves various races between radosgw startup and further operations -
both within the rgw task itself (such as the 'radosgw-admin realm pull'),
and in later tasks

Fixes: http://tracker.ceph.com/issues/17794
Fixes: http://tracker.ceph.com/issues/17872
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add retry/backoff to sync agent requests

resolves an issue where startup of the radosgw-agent races with the
requests we send to it to run sync. uses the requests package with
urllib3 to add retry with backoff to these requests

Fixes: http://tracker.ceph.com/issues/16129
Signed-off-by: Casey Bodley <cbodley@redhat.com>

objectstore.yaml: raise fd limit

Signed-off-by: Samuel Just <sjust@redhat.com>

fast.yaml: overrides not override

Signed-off-by: Samuel Just <sjust@redhat.com>

thrash-erasure-code: disable ceph-objectstore-tool testing for now

Signed-off-by: Samuel Just <sjust@redhat.com>

rados,thrash-erasure-code: add overwrites workloads

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1263 from ceph/wip-cephfs-perm

rados/singleton-nomsgr/all/cache-fs-trunc: debug

cephfs: get new fsmap after marking clusters down

Credit to John Spray for identifying the problem/cause.

Fixes: http://tracker.ceph.com/issues/17894
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rados/singleton-nomsgr/all/cache-fs-trunc: debug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1258 from ceph/jewel-x-fixes

rados/upgrade: skip libcephfs-devel install

Reviewed-by: Dan Mick <dmick@redhat.com>

rados/upgrade: skip libcephfs-devel install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1253 from ceph/wip-vive-la-libcephfs-devel

packages: s/libcephfs2-devel/libcephfs-devel/

packages: s/libcephfs2-devel/libcephfs-devel/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1251 from ceph/distros

distros/supported: add xenial

rados/singleton/all/mon-seesaw: give mons more time to stabilize

10s wasn't long enough for one run to get pgs created.

Signed-off-by: Sage Weil <sage@redhat.com>

distros/supported: add xenial

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1249 from ceph/radostool-osds

rados/singleton: need 3 osds for radostool test

rados/singleton: need 3 osds for radostool test

There's a k=2 m=1 ec pool in the test!

Fixes: http://tracker.ceph.com/issues/17775
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1248 from ceph/scrub-vs-recovery

rados/thrash: test osd_scrub_during_recovery=false

rados/thrash: test osd_scrub_during_recovery = false

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1246 from ceph/wip-powercycle

suites/powercycle: skip ext4; add bluestore

suites/powercycle: test bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

suites/powercycle: no ext4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1224 from jcsp/wip-hadoop-on-ubuntu

suites/hadoop: run only on ubuntu

Merge pull request #1238 from ceph/rados-upgrades

rados/singleton-nomsgr: drop hammer, infernalis upgrades

upgrade/jewel-x: skip libcephfs2 install

None of these tests currently cover fs, so this is
sufficient... :/

Signed-off-by: Sage Weil <sage@redhat.com>

rados/singleton-nomsgr: drop hammer, infernalis upgrades

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1175 from batrick/multimds-thrash

MDS thrasher for multimds

Reviewed-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'origin/pull/1/head' into multimds-thrash

* origin/pull/1/head:
  tasks/vstart_runner: fix default run-all
  tasks/vstart_runner: update for Filesystem changes

rados/upgrade: skip libcephfs2

Signed-off-by: Sage Weil <sage@redhat.com>

tasks/vstart_runner: fix default run-all

This broke when path changed.  It's a bit
weird to run the cephfs tests by default
now that vstart_runner is not cephfs-specific,
but for the moment just restore the old
behaviour.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/vstart_runner: update for Filesystem changes

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mds_thrash: support multimds

This commit amends the MDS thrasher task to also work on multimds
clusters. Main changes:

o New FSStatus class in tasks/cephfs/filesystem.py which gets a snapshot
  of the fsmap (`ceph fs dump`). This allows consecutive operations on
  the same fsmap without repeated fs dumps.

o Only one MDSThrasher is started for each file system.

o The MDSThrasher operates on ranks instead of names (and groups of
  standbys following the initial active).

o The MDSThrasher also will change the max_mds for the cluster to a new
  value [1, current) or (current, starting max_mds]. When reduced,
  randomly selected MDSs other than rank 0 will be deactivated to reach
  the new max_mds. The likelihood of changing max_mds in a given cycle of
  the MDSThrasher is set by the "thrash_max_mds" config.

o The MDSThrasher prints out stats on completion, e.g. number of
  mds deactivated or mds_max changed.

Pre-requisite for: http://tracker.ceph.com/issues/10792
Partially fixes: http://tracker.ceph.com/issues/15134

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1223 from ceph/wip-jlayton-libcephfs2

libcephfs1 -> libcephfs2 rename

Merge pull request #1236 from ceph/wip-upgrade-mgr

rados/upgrade: skip ceph-mgr on jewel install

rados/upgrade: skip ceph-mgr on jewel install

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1234 from jcsp/wip-fixup-cephfs

Fix running cephfs python tests without mgr daemon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/ceph_test_case: only construct needed parts

Don't construct Filesystem and MDSCluster if there
are no MDSs in the system, don't construct MgrCluster
if there are no mgrs in the system.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/vstart_runner: fix paths in docstring

Should have been updated when this file moved.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/mgr: remove duplicated method definition

This was already defined in parent CephCluster class.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1229 from ceph/wip-jewel-upgrades

fix ceph-mgr install for upgrade tests

upgrade/jewel-x/point-to-point: fix upgrades

skip ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1233 from ceph/wip-17759

Added short pg log entries to rgw test

Reviewed-by: Samuel Just <sjust@redhat.com>

Added short pg log entries to rgw test

This change aims to test rgw with messenger failure injection
and very short logs.

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

objectstore/bluestore: disable bluefs env mirror

We can't atomically append to two env backends, which means
that we can see size mismatches when we inject failures.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1232 from ceph/wip-blackhole

tasks: use new generic objectstore_blackhole option

tasks: use new generic objectstore_blackhole option

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/client-upgrades: exclude ceph-mgr from old installs

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly-hammer-x: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/hammer-x: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/infernalis*: remove

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/jewel-x: skip ceph-mgr install

Signed-off-by: Sage Weil <sage@redhat.com>

rados/objectstore/objectstore: suppress core dumps

Some of these tests to fork and assertion tests that leave
core behind, and that is normal.  Disable core dumps.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1086 from jcsp/wip-ceph-mgr

Create ceph-mgr tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

tasks: update cephfs_test_runner for mgr

Signed-off-by: John Spray <john.spray@redhat.com>

packages: install ceph-mgr

Signed-off-by: John Spray <john.spray@redhat.com>

tasks: update to run ceph-mgr daemons

Signed-off-by: John Spray <john.spray@redhat.com>