buildpackages: fix RPM generation script to support opensuse spec file

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 4692f995ba6967796d5dd2f5fd85fde8705a1ccc)

Merge pull request #1141 from rjfd/wip-test-buildpackages

buildpackages: make make-{deb,rpm}.sh aware of cmake

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #1142 from dachary/wip-17116-workunit

workunit: allow parallel git clone

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

workunit: allow parallel git clone

When multiple client run in parallel on the same machine and they
try to get workunits from a repository that is not github, they must
git clone in a directory that is suffixed as srcdir. Otherwise they
will conflict with each other.

Fixes: http://tracker.ceph.com/issues/17116
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #1137 from ceph/wip-fix-console

Fix console handling in tasks

Reviewed-by: Dan Mick <dmick@redhat.com>

Update console checking, and DRY

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop unnecessary console handling code

Remote.console is provided by teuthology now
http://tracker.ceph.com/issues/17124
Fixes: 17124
Signed-off-by: Zack Cerza <zack@redhat.com>

buildpackages: make make-{deb,rpm}.sh aware of cmake

Fixes: http://tracker.ceph.com/issues/16819
Signed-off-by: Ricardo Dias <rdias@suse.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1133 from SUSE/wip-17075

Revert "packages.yaml: reflect python-ceph package split"

Merge pull request #1135 from athanatos/wip-stdout

ceph_manager: do not use StringIO instance as default argument

ceph_manager: do not use StringIO instance as default argument

Otherwise, all invokations share the *same* StringIO instance
leading to some very silly bugs.

Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "packages.yaml: reflect python-ceph package split"

This reverts commit 0216330568fc1dec2e866bbfd1e49dc42266e1ab.

Fixes: http://tracker.ceph.com/issues/17075
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1128 from ceph/wip-16805

ceph_manager: lower mon_reweight_min_bytes_per_osd to 10

Reviewed-by: Samuel Just <sjust@redhat.com>

ceph_manager: lower mon_reweight_min_bytes_per_osd to 10

otherwise monitor could reject the command:
```
Refusing to reweight: we only used 588084 kb used across all osds!
```
if the average used space is smaller than
`mon_reweight_min_bytes_per_osd`.

Fixes: http://tracker.ceph.com/issues/16805
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1122 from ceph/wip-fixbiglogs

Fix big logs in rados thrash runs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #1116 from ceph/wip-16529

suites/upgrade: point-to-point Jewel upgrade workaround

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

revive_osd: Suppress output used to wait for OSD to be fully started

Signed-off-by: David Zafman <dzafman@redhat.com>

Thrashosds: Turn off output when doing dump_ops testing

Needed to add stdout option to pass down to run()

Signed-off-by: David Zafman <dzafman@redhat.com>

suites/upgrade: point-to-point Jewel upgrade workaround

The v10.2.x cls_rbd test case will not pass against a v10.2.0
OSD. Disable the offending test.

Fixes: http://tracker.ceph.com/issues/16529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1117 from ceph/wip-16975

jewel-x: use jewel-compatible librados tests for upgraded clients

Reviewed-by: Yuri Weinstein <yuri.weinstein@gmail.com>

Merge pull request #1119 from ceph/wip-rgw-17025

rgw: data_extra_pool is unique per zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: data_extra_pool is unique per zone

Fixes: http://tracker.ceph.com/issues/17025
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

jewel-x: use jewel-compatible librados tests for upgraded clients

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1115 from ceph/wip_add_sterr

Added stderr to the log file in addition to stdout

Reviewed-by: Dan Mick <dmick@redhat.com>

Added stderr to the log file in addition to stdout

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1110 from fullerdj/wip-djf-20160808

cephfs/vstart_runner: add -w option to ps

Reviewed-by: John Spray <john.spray@redhat.com>

cephfs/vstart_runner: add -ww option to ps

vstart_runner can't find arguments to ceph daemons to identify them with
ps -x because commands are cut off at terminal width. Add -ww for wide
output.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #1107 from ceph/wip-16879

suites/fs: fix log whitelist for inotable repair

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #1084 from ceph/wip-16640

task/packages: install libcephfs1-dev

suites/fs: fix log whitelist for inotable repair

Fixes: http://tracker.ceph.com/issues/16879
Signed-off-by: John Spray <john.spray@redhat.com>

task/packages: install libcephfs-dev

The way we currently build the cephfs java
bindings depends on the unversioned .so file
which is in the -dev package, not the
main one.

Fixes: http://tracker.ceph.com/issues/16640
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1082 from ceph/wip-mds-minor

tasks/cephfs: check cache size health warning

tasks/cephfs: check cache size health warning

Fortunately we already have a test that creates the condition,
so just tweak it to exceed the 150% threshold for the health warning,
and check that the health message appears.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #1106 from ceph/wip-flake8

fix flake8 errors

Reviewed-by: Samuel Just <sjust@redhat.com>

cephfs/filesystem: remove unused var

flake8 passes now

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

tox.ini: exclude .tox dir from flake8

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1105 from athanatos/wip-ec-pg-log

update diverge priors tests - this is handled by persisting the missing set now

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

suites: fix asok_dump_tree.yaml

This had an extra ceph and install task.

Signed-off-by: John Spray <john.spray@redhat.com>

cephfs: add test for dump tree admin socket command

Add test_dump_tree to validate the MDS admin socket command
dump tree. Add a parameter for path_to_ino to enable the use of
lstat() instead of stat()

See: http://tracker.ceph.com/issues/11171
See: https://github.com/ceph/ceph/pull/9925
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

suites/fs: log whitelist for inotable repair

Signed-off-by: John Spray <john.spray@redhat.com>

cephfs: Added a inotable repair case to the test

Merge pull request #1099 from ceph/wip-fix-bad-check

Fix rechecking of health in loop

Merge pull request #1098 from smithfarm/wip-16808

packages/packages.yaml: drop ceph-fs-common-dbg

Reviewed-by: John Spray <john.spray@redhat.com>

Fix rechecking of health in loop

Use the newer stdout value for checking
ceph health again.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

packages/packages.yaml: drop ceph-fs-common-dbg

Meant to be merged together with https://github.com/ceph/ceph/pull/10433

Fixes: http://tracker.ceph.com/issues/16808
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1094 from ceph/wip_reduce_rados

Reduce number of jobs for hammer branch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1074 from SUSE/wip-python-split

packages.yaml: reflect python-ceph package split

Reduce number of jobs for hammer branch

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1003 from athanatos/wip-15655

ceph_manager: test [test-]reweight-by-(utilization|pg)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

config: add osd_debug_verify_missing_on_start

Signed-off-by: Samuel Just <sjust@redhat.com>

divergent_priors: we no longer track divergent priors explicitely

The rest of the test is still valuable to ensure that we detect missing
items which are not in the log, but now that the missing set is
explicitely persisted, the divergent priors set isn't a special case
and won't have special log lines to check for.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1091 from athanatos/wip-16413

11429: wait_for_clean between restarting the osds and starting the bench

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1090 from athanatos/wip-16416

ec_lost_unfound: set min_size to 2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1085 from ceph/wip_add_newest

Added option to search for newest good/built commit (7 commits back)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1080 from ajarr/wip-15615

tasks/cephfs/test_volume_client: test authentication metadata

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #1073 from ceph/wip-mds-snap-failover

fs: add snapshot tests to mds thrashing

Added option to search for newest good/built commit (7 commits back)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #1077 from trociny/wip-16386

rbd: tests with rbd_skip_partial_discard option enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: tests with rbd_skip_partial_discard option enabled

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: move copy_on_read overrides to config dir

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

fs: add snapshot tests to mds thrashing

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tasks/cephfs/test_volume_client: test authentication metadata

Test the usecases for the authentication metadata stored
by the volume client:

* Obtain the list of auth IDs having access to a volume.

* Restrict volume access to auth IDs of a single (OpenStack)
  tenant to enforce strong tenant isolation of volumes.

Signed-off-by: Ramana Raja <rraja@redhat.com>

ec_lost_unfound: set min_size to 2

We changed the default to k+1 instead of k.  Adjust test to compensate.

Fixes: http://tracker.ceph.com/issues/16416
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #1072 from trociny/wip-16539

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

packages.yaml: reflect python-ceph package split

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #1064 from jcsp/wip-vstart-fixes

tasks/cephfs: vstart_runner fixes

Reviewed-by: Ramana Raja <rraja@redhat.com>

vstart_runner: fix misleading log message

Message is logged as, filesystem is mounted,
despite the vstart_runner just trying to mount
at this stage.

Signed-off-by: Ramana Raja <rraja@redhat.com>

tasks/cephfs: rename 'long_running' decorator

Rename 'long_running' decorator as 'for_teuthology',
which is more intuitive.

Signed-off-by: Ramana Raja <rraja@redhat.com>

tasks/cephfs: use "blacklist clear"

... instead of iterating over all blacklist entries.  Fall
back to the old way if the new way doesn't work (i.e. on
old ceph)

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix race while mounting

This could fail if the mount hadn't finished
coming up.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix vstart_runner with cmake

This needed updating for the CLI living in bin/ now

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: disable setuid test in vstart environ

``long_running`` needs a better name, it's really just a
marker that says a test is for use in teuthology but not vstart.

Signed-off-by: John Spray <john.spray@redhat.com>

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #1008 from ceph/wip-systemd-tests

Wip systemd tests

Merge pull request #1069 from batrick/i16164

cephfs: test fragment size limit

Reviewed-by: John Spray <john.spray@redhat.com>

Run rados loadgen

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
Conflicts:
suites/smoke/systemd/tasks/systemd.yaml

Few comment fixes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

cephfs: test fragment size limit

The fragment configuration uses 10000 for the fragment max size. The reason for
this is that many tests add 1000 files to a single directory which will hit
this limit without fragmentation catching up.

The test_dirfrag_limit test confirms:

o That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations).
o That fragmentation (forced) will allow more entries to be created.
o That unlink fails when the stray directory fragment becomes too large and that unlinking may continue once those strays are purged.

Tests: https://github.com/ceph/ceph/pull/9789
Issue: http://tracker.ceph.com/issues/16164

Signed-off-by: Patrick Donnelly <batrick@batbytes.com>

Merge pull request #1068 from smithfarm/wip-drop-ceph-devel

packages.yaml: drop ceph-devel

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

packages.yaml: drop ceph-devel

There are no "-dev" packages listed in the deb section of this file, so it
serves no purpose to have ceph-devel here.

Also, once https://github.com/ceph/ceph/pull/9744 is merged the presence of
"ceph-devel" in this file will break teuthology for rpm targets.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1020 from ceph/wip-15563-master

Added file locking logic to avoid possible collisions …

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #1065 from ceph/wip-ec-min-size

rados/thrash/workloads/cache-agent-big: set min_size=2 on m=1 k=2 ec pool

rados/thrash/workloads/cache-agent-big: set min_size=2 on m=1 k=2 ec pool

Otherwise min_size will be 3 and we will wedge on a single down
osd.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #1063 from ceph/wip-no-fuse

Revert "tasks/scrub_test: use fuse path (do not assume filestore!)"

Merge pull request #1059 from ceph/wip-s3roundtrip-wusui

Allow force-branch for s3roundtrip tests.

Revert "tasks/scrub_test: use fuse path (do not assume filestore!)"

This reverts commit 9e2b455f45cca2f31b19c73d761661a9028c22ff.

Update cron_wrapper

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #970 from ceph/wip-fix-cd-override

Fix for 'branch' overrides to work

Merge pull request #1058 from ceph/wip-15016

tasks/ceph-deploy: Run on multiple Python versions (2, 3)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1002 from athanatos/wip-osd-recovery

osd_recovery: add delay to ensure maps propogate

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1062 from athanatos/wip-resolve-peering

resolve_stuck_peering: reset stats while looping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #1061 from athanatos/wip-short-log

short_pg_log: double params to avoid failures from dup detection failure

http://tracker.ceph.com/issues/16331

Merge pull request #1060 from athanatos/wip-flush-pg-stats

[ec_]lost_unfound: don't flush_pg_stats at the beginning

Reviewed-by: Sage Weil <sage@redhat.com>

11429: wait_for_clean between restarting the osds and starting the bench

ceph.restart now marks the osds down, so the objects are actually being
created while slowest of the osds boots. That causes a ton of 1 byte
objects to be created in a degraded state and causes the cleanup to take
a long time.  Also, reduce length of bench since it's only being used
to ensure the osds came up correctly.

Signed-off-by: Samuel Just <sjust@redhat.com>

tasks/ceph-deploy: Install appropriate Python version before running

Related to RM-15016

Signed-off-by: Oleh Prypin <oleh@pryp.in>

Allow force-branch for s3roundtrip tests.

Fixes tracker bug 15453
Signed-off-by: Warren Usui <wusui@redhat.com>

Merge pull request #1050 from ceph/wip-fix-scrub-test

test: Eliminate racey scrubbing in scrub_test.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1057 from ceph/wip-scrub

tasks/scrub_test: use fuse path (do not assume filestore!)

Reviewed-by: Samuel Just <sjust@redhat.com>

tasks/scrub_test: use fuse path (do not assume filestore!)

Signed-off-by: Sage Weil <sage@redhat.com>

rados/objectstore/objectstore.yaml: skip bluestore tests

Avoid breaking rados suite while we are developing rapidly.

Signed-off-by: Sage Weil <sage@redhat.com>