ceph: fix up log rotation stopper

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 451c5ca79de14a8462a76dc8287be1f4fd72fa1b)

Merge remote-tracking branch 'gh/next'

Merge pull request #455 from ceph/wip-11937

thrasher: Can't test ceph-objectstore-tool if nodes turned off (power…

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #452 from ceph/wip-logrotate

Logrotate

Reviewed-by: John Spray <john.spray@redhat.com>

ceph: update log rotation for review comments

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

thrasher: Can't test ceph-objectstore-tool if nodes turned off (powercycle)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #440 from ceph/wip-fs-clientnum

Reduce fs test node usage

Reviewed-by: John Spray <john.spray@redhat.com>

add log-rotate configs to the userspace suite

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph: support arbitrarily-named daemons in logrotate

And make it more configurable in terms of sizes.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph: enable mds log rotation

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

suites/multimds: use 2-node clusters for pure userspace subsuites

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

suites/fs: use a two-node layout for most userspace tests

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

suites/fs: switch standby-replay tests to use a smaller cluster

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #449 from ceph/wip-fsync

tasks/cephfs: add TestClientRecovery.test_fsync

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #451 from athanatos/wip-11677

Wip 11677

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #450 from ceph/wip-fix-blacklist

tasks/cephfs: fix typo in blacklist clearing

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tasks/cephfs: add TestClientRecovery.test_fsync

To test that metadata written recently is
preserved across a client+server crash when
barriered with a directory fsync.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix typo in blacklist clearing

Broken in aa0ffb3

Signed-off-by: John Spray <john.spray@redhat.com>

rados: add write_excl and append_excl

Signed-off-by: Samuel Just <sjust@redhat.com>

rados/thrash: add test for radosgw with snaps

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #443 from ceph/wip-11301

tasks/cephfs: reset osd blacklist between tests

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #447 from ceph/wip-11779

tasks/cephfs: fix timing in test_full

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #444 from ceph/wip-11756

tasks/cephfs: mount fusectl before listing fuse connections

Reviewed-by: John Spray <john.spray@redhat.com>

tasks/cephfs: fix timing in test_full

The interval between writes was too short because
it was not taking account of the way OSDMap full
flags are set on tick rather than immediately.

Fixes: #11779
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #438 from ceph/wip-11405

client-upgrade: test librbd backwards compatibility

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tasks/cephfs: mount fusectl before listing fuse connections

Fixes: #11756
Signed-off-by: Yan, Zheng <zyan@redhat.com>

tasks/cephfs: reset osd blacklist between tests

...to avoid OSDMap modifications happening in the
background due to blacklist expiry.

Fixes: #11301
Signed-off-by: John Spray <john.spray@redhat.com>

Added script to schedule 1/14th part of rados

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit a7064a7b3d8a865dead3800a4d97bd70da6dec33)

Added script to schedule 1/14th part of rados

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #442 from ceph/wip-client-cache-size

cephfs/test_client_limits.py: invalidate kernel dcache according to client cache size

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #425 from ceph/wip-flock-optional

cephfs/mount: check FUSE version before invoking flock

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #398 from ceph/wip-quota-full

Test for CephFS full handling with pool quotas (#11276)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #402 from ceph/wip-11343

Reviewed-by: Samuel Just <sjust@redhat.com>

cephfs/test_client_limits.py: invalidate kernel dcache according to client_cache_size

Signed-off-by: Yan, Zheng <zyan@redhat.com>

cephfs/test_client_recovery: check FUSE version before invoking flock

flock only works properly on FUSE versions >=2.9, which is newer
than eg Ubuntu Precise. So check the version on our client mounts and
only test flock if it's at least that new.

Fixes: #9995
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

suites: update log whitelist for TestQuotaFull

A quota ENOSPC looks different in the log than
as capacity ENOSPC

Signed-off-by: John Spray <john.spray@redhat.com>

suites: tweak config in mds-full task

The task runs a lot faster with a higher
OSD report frequency.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: cover pool quota in test_full

Run the same procedure as TestClusterFull, but
instead of limiting OSD memstore size, use pool
quota on the data pool.

Signed-off-by: John Spray <john.spray@redhat.com>

samba: disable MDS debugging to reduce log sizes

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Replaced work units with ceph-deploy/ceph-deploy_hello_world.sh re: #11547
Removed per Josh's suggestion fs as redundant and renamed rados to bacis

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit dcfd6a3b099fd0e14fbfd47a8678a411387e01da)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #436 from ceph/wip_11547_2

Replaced work units with ceph-deploy/ceph-deploy_hello_world.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Replaced work units with ceph-deploy/ceph-deploy_hello_world.sh re: #11547
Removed per Josh's suggestion fs as redundant and renamed rados to bacis

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #431 from ceph/wip-more-log-whitelist

suites/fs: add another log whitelist to journal-repair

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #426 from ceph/wip-11372

add new valgrind tests to rbd suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #433 from ceph/wip_11400

Initial check in for upgrades/hammer stable upgrade suite

Initial check in for upgrades/hammer stable upgrade suite
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

reg11184: Add regression test for 11184

Create divergent priors and a split and then move a pg using
ceph-objectstore-tool export/import
Add yaml file to run the reg11184 task

Fixes: #11343
Signed-off-by: David Zafman <dzafman@redhat.com>

divergent_priors2: Do divergent priors and ceph-objectstore-tool export/import

Based on tasks/divergent_priors.py but also do simple export/remove/import on
same osd.
Add yaml file to run the divergent_priors2 task

Signed-off-by: David Zafman <dzafman@redhat.com>

divergent_priors: Fix divergent_priors task

Flake8 fixes
Use new set_recovery_delay admin socket command
Fix bad value set for filestore_blackhole
Make sure log trims and only require 100 objects
Use kick_recovery_wq to properly set osd_recovery_delay_start to 0
Write and remove divergent and verify removal was undone
Fix to make compatible with wip-10809-11135-10290
Make sure to set_recovery_delay in a non-racey way (while osd running but down)
Leave divergent "in" so its PGs aren't treated as strays
Add yaml file to run the divergent_priors task

Signed-off-by: David Zafman <dzafman@redhat.com>

client-upgrade: added RBD notification tests

Verify correct behavior between two librbd versions when
proxying notify messages.

Fixes: #11405
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

client-upgrade: added librbd API test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

suites/fs: add another log whitelist to journal-repair

This is the table unreadable error for mds1_inotable that
we get when trying to start an MDS whose objects are gone.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge remote-tracking branch 'gh/next'

Conflicts:
suites/fs/recovery/tasks/client-limits.yaml
suites/fs/recovery/tasks/mds-full.yaml
tasks/ceph_deploy.py

smoke: white marked down on python test

Backport: hammer
Signed-off-by: Sage Weil <sage@redhat.com>

Added hammer-client-x to the mix
Renamed dirs to better reflect what is being tested
Removed old dirs
Added logging
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 3c8730d035a90f221c05526983c40f0a11da1ba6)

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #428 from athanatos/wip-11429

suites/rados: add test for 11429

Reviewed-by: Zack Cerza <zack@redhat.com>

knfs: remove the MDS debugging

It doesn't fit on the typica boxes with debugging, and we probably don't need
it for any NFS issues which do arise...

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

squash: use sleep instead of utility.sleep

squash: use != not is not in radosbench

squash: remove utility, moved to teuthology.git

squash: ceph_manager: add utility_task doc string

suites/rados: add test for 11429

This patch also adds some convenience facilities for making
some of the ceph_manager methods into tasks usable from a
yaml file.

Signed-off-by: Samuel Just <sjust@redhat.com>

Removed redundant tests issue #11520
Reduced nodes to be locked to two

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 8ab46b42c5ae387141917bb8eff35960e9c45699)

Merge pull request #427 from ceph/wip_11520

Removed redundant tests issue #11520
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Removed redundant tests issue #11520
Reduced nodes to be locked to two

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

All changes from https://github.com/ceph/ceph-qa-suite/commit/3ce379a689440ce084b2df79db59dd76e1b2db6c

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Removed unneeded workloads
Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 18a79fb8252d9ceee7162e8f774ab6bf806484c5)

Conflicts:

suites/ceph-deploy/fs/tasks/cfuse_workunit_suites_dbench.yaml
suites/ceph-deploy/fs/tasks/cfuse_workunit_suites_fsstress.yaml
suites/ceph-deploy/rados/tasks/rados_python.yaml
suites/ceph-deploy/rados/tasks/rados_workunit_loadgen_big.yaml
suites/ceph-deploy/rbd/tasks/rbd_api_tests_old_format.yaml
suites/ceph-deploy/rbd/tasks/rbd_cli_tests.yaml
suites/ceph-deploy/rbd/tasks/rbd_cls_test.yaml

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Removed unneeded workloads

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 18a79fb8252d9ceee7162e8f774ab6bf806484c5)

Fixing commits from https://github.com/ceph/ceph-qa-suite/pull/400/commits

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Reduced ammount of nodes from 4 to 3
Changed default 'replica' to 2
Reduced osd to 2 per node
Added logging

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit 3ce379a689440ce084b2df79db59dd76e1b2db6c)

Merge pull request #423 from ceph/wip-11495

Fix ceph_deploy task with multiple mons

Merge pull request #421 from ceph/wip-client-osd-caps

tasks/cephfs: add test_pool_perm

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

tasks/ceph_deploy: remove some dead code

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/ceph_deploy: fix for multiple mons

Now that service IDs are modified during run, we have
to avoid repeatedly evaluating first_mon for where
to run ceph_deploy, as the answer will change.

Fixes: #11495
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #422 from ceph/wip-log-whitelist

suites: log whitelist for journal repair test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suites/rbd: add new valgrind memtest tests

Exercise the C-API, Python-API, and FSX under valgrind.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tasks: add support for running fsx under valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

suites: log whitelist for journal repair test

"bad backtrace on dir ino" is emitted because
journaltool's recover-dentries doesn't write
backtraces.

Signed-off-by: John Spray <john.spray@redhat.com>

tasks/cephfs: add test_pool_perm

This tests that client checks pool permission before
copy data to its cache.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

rbd.xfstests: remove bad assert

Misguided input validation. A better error comes
from the run_xfstests() method later anyway.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Initial checkin for hammer-x suite
Used "ceph osd crush tunables hammer"

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>
(cherry picked from commit f825d32d9deab22f6113b96a6b36734fe1bdb289)

Merge pull request #405 from ceph/wip-10781

Initial checkin for hammer-x suite

Merge pull request #414 from ceph/wip_11413

Added hammer-client-x to the mix

xfstests: fix typo

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

xfstests: define defaults in root task

The early non-defaults caused failures due to xfstests_url: None not
being overridden by run_xfstests(). Move the defaults to xfstests() and
don pass xfstests_branch past that point.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

task/samba: ignore return code of fuser/losf

if mount point is not used by anyone, both fuser and losf return 1

Fixes: #10624
Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit c049387b7b0832e78a4a1370691798f726985799)

task/samba: use SIGTERM to stop samba server

man samba(8) contains sentences:

To shut down a user's smbd process it is recommended that SIGKILL (-9)
NOT be used, except as a last resort, as this may leave the shared
memory area in an inconsistent state. The safe way to terminate an smbd
is to send it a SIGTERM (-15) signal and wait for it to die on its own.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 86bd6bc3771694cd9ee5cd071255f3cf4c40c812)

Merge pull request #417 from ceph/wip-10657

cephfs/test_client_limits.py: test for client oldest tid warning

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #416 from ceph/wip-11316-next

Backport to next: fix ceph-deploy task with CephFS (#11316)

Merge pull request #415 from ceph/wip-10624

task/samba: ignore return code of fuser/losf

Reviewed-by: Sage Weil <sage@redhat.com>

cephfs/test_client_limits.py: test for client oldest tid warning

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tasks/ceph_deploy: configure CephFS

This test apparently had not been touched since
"fs new" was added.  In addition to calling
Filesystem.create:
 * modify the get_nodes_using_role
   function to modify ctx.cluster.remotes so that the
   service IDs match what ceph-deploy will set
 * log exceptions during ceph_deploy setup, as otherwise
   they can get lost if another exception occurs during
   teardown (so that it's all easier to debug).
 * default to passing --dev=master during install, so
   that we don't error out horribly when run without
   an explicit branch set (e.g. when run outside
   scheduled suite)

Fixes: #11316
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 7c2e6cdc6cd3d02969d3494d120b12daaa13efa8)

tasks/cephfs: cope with missing ctx.daemons attr

...so that we may be used with the ceph_deploy task
that doesn't set that up.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit f36011051cabb8e7b06194c778e640e98d328e8a)

tasks/cephfs: tweak use of mon for admin commands

... s/mon_remote/admin_remote/ and allow caller to pass
in which remote they want to use for that.  Enables use
with ceph_deploy task which does not give admin keys
to mons.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 8b61310ce3eedeb8ac60446933cd5db548de6434)

tasks/ceph: refactor legacy FS configuration check

Move up into Filesystem so that this can be used from
the ceph_deploy task as well.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit ea7c39222ae084c90b5a8510ce5af35a2f75a6e3)

task/samba: ignore return code of fuser/losf

if mount point is not used by anyone, both fuser and losf return 1

Fixes: #10624
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge branch 'greg-testing-snapshot-rename'

Merge pull request #404 from ceph/wip-3645-fixup

DNM: tasks: s/pending_create/pending_update/ in MDS

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

suites/fs: update log whitelist

These ones had been passing because of #11177,
now that that is fixed they require an update.

Fixes: #11300
Signed-off-by: John Spray <john.spray@redhat.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit fa2a86196c3ed39c1726be689e42ed5d4aae197a)

Merge remote-tracking branch 'origin/wip-3645-fixup' into greg-testing-snapshot-rename