qa/workunits/rbd: simplify split-brain test to avoid potential race

Fixes: http://tracker.ceph.com/issues/22485
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8c2ee6dd0f4c77b9fbbc24a673841b04d3e175d6)

Conflicts:
qa/workunits/rbd/rbd_mirror.sh: trivial resolution

rbd-mirror: sync image metadata when transfering remote image

Fixes: http://tracker.ceph.com/issues/21535
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9aa5dfb8ec46e8babd0f167dcc0e234a7c60a50c)

rbd-mirror: update asok hook name on image rename

Fixes: http://tracker.ceph.com/issues/20860
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 9ddb5da3377029d11cb1089102d7146e9ee2537a)

Merge branch 'wip-sagrawal-rbd' into 'rh-luminous'

workunit for installing operating system on a rbd image

Signed-off-by: Sidhant Agrawal <sagrawal@redhat.com>
See merge request !4

typo correction os_install.py
Signed-off-by: Sidhant Agrawal <sagrawal@redhat.com>

qa/suites/krbd/rbd/tasks:added rbd_os_install.yaml
Signed-off-by: Sidhant Agrawal <sagrawal@redhat.com>

/qa/workunits/rbd:update os_install.py to generate kickstart file automatically
Signed-off-by: Sidhant Agrawal <sagrawal@redhat.com>

qa/workunits/rbd:added script to install operating system on rbd image
Signed-off-by: Sidhant Agrawal <sagrawal@redhat.com>

qa/tests: use norequire_active_mds options only for test kernel mounts

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

sync rgw multi changes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

sync rbd.py changes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

sync rados.py changes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

sync thrash-osd-heath ignore's

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

sync ceph-manager and kernel mount options

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

sync fs script changes

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

mgr changes sync with upstream

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix cluster config

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add simple rbd performance tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use retry instead of bash while loop which can block indefinitly

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

gitlab changed from osas to gitlab.cee.redhat.com

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add support for 3.0 cdn builds

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge branch 'wip_cop_file_err' into 'rh-luminous'

io_info.yaml copy file error fixed

See merge request ceph/ceph!12

removed the rename of put file to master node and renamed the io_info yaml in master node

io_error for io_yaml fixed and test_acls_reset dictionary parse error fixed

io_info file renamed to target_node

reverted to copy_function and extra logs

put file and get file corrected to  put from target to master node

log info added before and after copy

removed wip branch checkout

wip branch added

removed copy_function to copy io_info.yaml file from master to target node

qa: use ceph-luminous branch for s3tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/suites/rados/thrash/d-*/*balancer*: debug osd in mgr

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash: combine balancer with require-luminous

We can only do the balancer stuff if we've done the mkfs-time
require luminous because otherwise the mgr commands aren't proxied via
the mon.

Only do balancer for thrash tests; not for basic, monthrash, verify.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: enable module by default

It will still be "off".

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 26710f0a9b176289237a52e5fa5894342ad163dc)
Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
Conflicts:
src/common/options.cc

qa: fix mgr _load_module helper

I inadvertently broke this with the latest change
to the module ls output.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 4fb3025682d812f86c50fa36532290fc0f8857ae)

qa: fix mgr caps

This was still using Kraken era settings

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 9988ebed9530718b523a23d729d499dedab5eb6d)

qa: expand mgr testing

Some extra coverage of the dashboard, including its standby
redirect mode and the publishing of URIs.

Also invoking the command_spam mode of the selftest module.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 05e648be6a51d3efa110ad9826bbdd0adcc4dd4d)

qa/mgr: fix influx/prometheus test names

This was a typo: they were swapped around.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d96a59e74b6984b77c9f3b15f702e3bf45053590)

qa: add mgr module selftest task

The module self test commands give us a chance to
catch any other ceph changes that change something
that a module was relying on reading.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 99352ceced9d0fe92ddad6b97b1393b41de75d50)

qa/suites/rados/thrash/d-balancer: enable balancer in various modes

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2c9c18d1ec3d33d385adf2a30dc1d3b8e987f9a5)

qa: stop testing deprecated "ceph osd create"

"ceph osd create" is not idempotent, and is considered deprecated.

Fixes: http://tracker.ceph.com/issues/21993
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 371d3f40911917704f4992c48236b4c60905bf4e)

Fix cluster config for octo node

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/tasks: prolong revive_osd() timeout to 6 min

see also #17902

Fixes: http://tracker.ceph.com/issues/21474
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 749bbda07522114b99b17a4a01cfcf019520285d)

qa: do not wait for down/out osd for pg convergence

that osd is not invovlved in the PG state changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7f549af459b40c5bb51a24a3a443ed7a7116915e)

osd: add max-pg-per-osd limit

osd will refused to create new pgs, until its pg number is lower
than the max-pg-per-osd upper bound setting.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4c7df944c7f28232873ba681eedce72cdb062ea5)

qa/tasks: prolong revive_osd() timeout to 6 min

bluestore_fsck_on_mount and bluestore_fsck_on_mount_deep are enabled by
default. and bluestore is used as the default store backend. it takes
longer to perform the deep fsck with verbose log. so prolong the
revive_osd()'s timeout from 150 sec to 360 sec.

Fixes: http://tracker.ceph.com/issues/21474
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 42be200c564184feff1056b7771ce3c1b2cc5527)

qa: s/backfill/backfilling/

it's renamed "backfilling" in 4015343f .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit e21114274f6a2742748a5a9b965d415241c80f3c)

qa/ceph_manager: check pg state again before timedout

Pg state maybe all in active+clean when no recovering going on,
so check it again before timedout.

Fixes: http://tracker.ceph.com/issues/21294
Signed-off-by: huangjun <huangjun@xsky.com>

Fix node config for octo lab

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge branch 'multisite_tests' into 'rh-luminous'

Multisite tests

See merge request ceph/ceph!11

deleted test_multi.yaml

removed checkout of wip branch of ceph-qe-scripts

deleted unwanted files

cluters dir added

test version with rever yaml added

test suspend versioning

multipart upload download yaml added

multipart upload cancel yaml added

test multipart upload yaml added

 test delete key versions

test bucket delete yaml added

test basic versioning added

test acls reset added

test acls copy obj yaml added

test acls all user yaml added

test acls added

test Mbucket with Nobjects yaml added

test Mbuckets yaml added

fixed some bugs and verification to tests

removed print statement

Remove print statement, needs log object to print to file instead

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

update cluster to use 2 mds and 3 mons

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

update rest module loads via ceph-ansible vars

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

clone branch as is dont try to escape chars

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Debug: for test s3branch  wip-fix-416

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix rest test where the global vars is not defined properly

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix systemd tests to have more osd's

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add cephfs data/metdata vars and enable rest modules

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Run all tests by default, figure out how to skip landsat tests
in a better way

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

disable 2 size 2 min size to reduce test count

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

whitelist (MDS_CACHE_OVERSIZED)

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Disable for now, mpicc compilation issue on redhat

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use failure domain as host

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add more osd's for 6 node cluster

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

whitelist TOO_MANY_PGS

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix cluster for octo lab

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

ceph-objectstore-tool: Make pg removal require --force

Add new export-remove to combine the 2 operations

Fixes: http://tracker.ceph.com/issues/21272
Signed-off-by: David Zafman <dzafman@redhat.com>

use the --distro option to run on various supported distro

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add back bluestore options

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

qa/cephfs: Fix error in test_filtered_df

ceph df accounts for pool size, so there is no need to do it in the test.

Fixes: http://tracker.ceph.com/issues/21381
Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit b059cb6290d81df07bd8ec4e1ec9ef0be24892a2)

qa/cephfs: Refactor alternate pool test

Remove the alternate pool recovery test from test_data_scan. Newer
commits will place the test in its own file.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 47318f8ac4391148a1394846dd77c8452b7cf5ae)

qa/cephfs: test CephFS recovery pools

Test recovering metadata in to a separate RADOS pool with
cephfs_data_scan and friends.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 6af2ae80d327e564ad17f2caeded9c8142becf29)

qa/cephfs: support CephFS recovery pools

Add support for testing recovery of CephFS metadata into an alternate
RADOS pool, useful as a disaster recovery mechanism that avoids
modifying the metadata in-place.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 8f9a25202093339afb5308051d354d3ae79c6b2d)

qa/cephfs: Allow deferred fs creation

Permit Filesystem objects to be created and settings modified before
calling Filesystem.create().

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit 5fafc03cb9e6d5f39ce494375ea28c89b896e2f5)

qa/ceph_test_case: support CephFS recovery pools

Add support for testing recovery of CephFS metadata into an alternate
RADOS pool, useful as a disaster recovery mechanism that avoids
modifying the metadata in-place.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
(cherry picked from commit c85562c94a80b8a18975b8d0ee6a7fbd932cf024)

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 06c94de)

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add s3a test using 281

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix pg/pgp numbers and also set mon_max_pg_per_osd: 1024
so that tests dont complain when values exceed optimal values

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

yum needs remove not uninstall

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>