qa/tasks: prolong revive_osd() timeout to 6 min

see also #17902

Fixes: http://tracker.ceph.com/issues/21474
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 749bbda07522114b99b17a4a01cfcf019520285d)

qa: do not wait for down/out osd for pg convergence

that osd is not invovlved in the PG state changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7f549af459b40c5bb51a24a3a443ed7a7116915e)

osd: add max-pg-per-osd limit

osd will refused to create new pgs, until its pg number is lower
than the max-pg-per-osd upper bound setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4c7df944c7f28232873ba681eedce72cdb062ea5)

qa/tasks: prolong revive_osd() timeout to 6 min

bluestore_fsck_on_mount and bluestore_fsck_on_mount_deep are enabled by
default. and bluestore is used as the default store backend. it takes
longer to perform the deep fsck with verbose log. so prolong the
revive_osd()'s timeout from 150 sec to 360 sec.

Fixes: http://tracker.ceph.com/issues/21474
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 42be200c564184feff1056b7771ce3c1b2cc5527)

qa: s/backfill/backfilling/

it's renamed "backfilling" in 4015343f .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit e21114274f6a2742748a5a9b965d415241c80f3c)

qa/ceph_manager: check pg state again before timedout

Pg state maybe all in active+clean when no recovering going on,
so check it again before timedout.

Fixes: http://tracker.ceph.com/issues/21294
Signed-off-by: huangjun <huangjun@xsky.com>

Fix node config for octo lab

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge branch 'multisite_tests' into 'rh-luminous'

Multisite tests

See merge request ceph/ceph!11

deleted test_multi.yaml

removed checkout of wip branch of ceph-qe-scripts

deleted unwanted files

cluters dir added

test version with rever yaml added

test suspend versioning

multipart upload download yaml added

multipart upload cancel yaml added

test multipart upload yaml added

 test delete key versions

test bucket delete yaml added

test basic versioning added

test acls reset added

test acls copy obj yaml added

test acls all user yaml added

test acls added

test Mbucket with Nobjects yaml added

test Mbuckets yaml added

fixed some bugs and verification to tests

removed print statement

Remove print statement, needs log object to print to file instead

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

update cluster to use 2 mds and 3 mons

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

update rest module loads via ceph-ansible vars

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

clone branch as is dont try to escape chars

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Debug: for test s3branch  wip-fix-416

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix rest test where the global vars is not defined properly

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix systemd tests to have more osd's

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add cephfs data/metdata vars and enable rest modules

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Run all tests by default, figure out how to skip landsat tests
in a better way

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

disable 2 size 2 min size to reduce test count

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

whitelist (MDS_CACHE_OVERSIZED)

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Disable for now, mpicc compilation issue on redhat

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use failure domain as host

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add more osd's for 6 node cluster

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

whitelist TOO_MANY_PGS

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix cluster for octo lab

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph-objectstore-tool: Make pg removal require --force

Add new export-remove to combine the 2 operations

Fixes: http://tracker.ceph.com/issues/21272
Signed-off-by: David Zafman <dzafman@redhat.com>

use the --distro option to run on various supported distro

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add back bluestore options

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/cephfs: Fix error in test_filtered_df

ceph df accounts for pool size, so there is no need to do it in the test.

Fixes: http://tracker.ceph.com/issues/21381
Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit b059cb6290d81df07bd8ec4e1ec9ef0be24892a2)

qa/cephfs: Refactor alternate pool test

Remove the alternate pool recovery test from test_data_scan. Newer
commits will place the test in its own file.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 47318f8ac4391148a1394846dd77c8452b7cf5ae)

qa/cephfs: test CephFS recovery pools

Test recovering metadata in to a separate RADOS pool with
cephfs_data_scan and friends.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 6af2ae80d327e564ad17f2caeded9c8142becf29)

qa/cephfs: support CephFS recovery pools

Add support for testing recovery of CephFS metadata into an alternate
RADOS pool, useful as a disaster recovery mechanism that avoids
modifying the metadata in-place.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 8f9a25202093339afb5308051d354d3ae79c6b2d)

qa/cephfs: Allow deferred fs creation

Permit Filesystem objects to be created and settings modified before
calling Filesystem.create().

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 5fafc03cb9e6d5f39ce494375ea28c89b896e2f5)

qa/ceph_test_case: support CephFS recovery pools

Add support for testing recovery of CephFS metadata into an alternate
RADOS pool, useful as a disaster recovery mechanism that avoids
modifying the metadata in-place.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit c85562c94a80b8a18975b8d0ee6a7fbd932cf024)

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 06c94de)

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add s3a test using 281

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix pg/pgp numbers and also set mon_max_pg_per_osd: 1024
so that tests dont complain when values exceed optimal values

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

yum needs remove not uninstall

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Test systemd with bluestore as well

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add ship_utilities after ansible task

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add ceph-mgr rest based test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use fio task for ansible encrypted test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Remove ubuntu references as distros will set the right version
we need for downstream

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Remove any reference to centos for downstream runs

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add test for ec with k=3 and m=2 and also up the pg's

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix ceph_ansible config for system and ganesha tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use better mirror cluster config

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add mgr role

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix encryption test failures

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa: add optional 'application' to pool creation helpers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: create_cache_pool no longer runs 'pool application enable'

Fixes: http://tracker.ceph.com/issues/21155
Signed-off-by: Casey Bodley <cbodley@redhat.com>

skip test that use public bucket landsat-pds from AWS

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

update 3.0 version

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/rgw: enable 'rgw' application on created pools

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: add optional 'application' to pool creation helpers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa: create_cache_pool no longer runs 'pool application enable'

Fixes: http://tracker.ceph.com/issues/21155
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Remove centos/ubuntu references

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Disable until the other ansible options works fine

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add basic cephmetrics deployment test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix cluster config for octo

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use filestore-xfs for basic tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix couple of unsupported configs and cluster for rbd

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add more options for ceph-ansible testing

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix few ceph-ansible tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Few additional whitelists and fs fix

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

calamari test no longer valid for rhcs 3.0

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add mgr role where required

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Increase sleep to 150 seconds to avoid race issue

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use filestore for testing rhcs 3.0

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Debug failed grep of purged_snapdirs

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Remove bluestore config option

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

rbd_fsx whitelist

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Print upto 50 instead of 1 ERR/WRN message

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Revert "print out all WRN/ERR messages for debug"

This reverts commit 2a4ea0f6826e157f23339891fa106853ad80c3a9.

Various whitelist fixes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

print out all WRN/ERR messages for debug

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Compress logs only once , pull_directory does gzip compression as well.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use ceph-ansible task for systemd

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix ceph_ansible overrides

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix fixed-2 cluster config for octo lab

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Remove 2 size 1 min size option

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>