ceph-volume tests.functional add new ceph-handlers role from ceph-ansible

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 956dc5e65c6e35e27ab6524d4fb9fdd05d616cf6)

Merge pull request #24280 from alfredodeza/luminous-rm27210

luminous ceph-volume custom cluster names fail on filestore trigger

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24211 from pdvian/wip-35854-luminous

luminous: doc: remove deprecated 'scrubq' from ceph(8)

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24190 from joke-lee/luminous

luminous:rgw: return x-amz-version-id: null when delete obj in versio…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24231 from pdvian/wip-35856-luminous

luminous: multisite: segfault on shutdown/realm reload

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24171 from ukernel/luminous-25046

luminous: mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests/functional update dmcrypt test playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 96e29adf72804cb4fca62c04416b8fe2d23587bb)

ceph-volume tests/functional/batch update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5f72d438da94323c0b04f261b1a60b13eaf03966)

ceph-volume tests/functional/lvm update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b0d2dc66968f81192f152fa166fbc2cb4dcb438)

ceph-volume tests/functional update playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ba10b6820fd7f352ed6ba78246a6976c54783cbd)

ceph-volume main move out the load_ceph_conf_path to the configuration module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 532908099eb63bd805db8aecae1ef12301076410)

ceph-volume tests.devices patch configuration.load to avoid raising non-existing ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe80e5399e9dd1f6b9f94404d732a76f961d9eb4)

ceph-volume tests reset the cluster name after every unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 877ee1a27e0651355812e27fbc6ac0a87ea56a68)

ceph-volume tests.functional add custom cluster name to centos7-filestore-create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e61848dc97b638c6b8e60f0896d325f436142864)

ceph-volume lvm.activate load the config always by reading the cluster name from the LV

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a88cbfc73a258b51b20e060e269234962444ce63)

ceph-volume configuration be able to set conf.ceph

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f8226f359d94a86a39492ed653d9a9e5d8dd8f9)

Merge pull request #24135 from linuxbox2/luminous-rgw-sync-loglevel

luminous: rgw: raise debug level on redundant data sync error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24065 from jdurgin/wip-luminous-cache-autotune

luminous: os/bluestore: cache autotuning and memory limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24115 from dzafman/wip-35991

luminous: tools: ceph-objectstore-tool: Allow target level as first positional …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: RGWAsyncPutSystemObjAttrs does not access coroutine memory

also exposes optional objv_tracker to RGWSimpleRadosWriteAttrsCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b01b3952db4c45feb98dbc7e224ca1982c224c72)

rgw: RGWAsyncPutSystemObj does not access coroutine memory

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 85396674811be8f7a25dc9d25fd2db68c93561c5)

rgw: RGWAsyncGetSystemObj does not access coroutine memory

now reads the buffer/attrs into local memory, and calling coroutines
copy out the result during request_complete()

Fixes: http://tracker.ceph.com/issues/35543
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 089db9f74266073eb1d45900d982f1831c72fe59)

Conflicts:
src/rgw/rgw_cr_rados.h : Resolved in request_complete

Merge pull request #24206 from alfredodeza/luminous-rm24794

luminous ceph-volume: earlier detection for --journal and --filestore flag requirements

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24209 from alfredodeza/luminous-rm36088

luminous ceph-volume  batch: allow journal+block.db sizing on the CLI

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mds: health warning for slow metadata IO

Fixes: http://tracker.ceph.com/issues/24879
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 0f735f40315448560fde049ed3ea019a7d30d868)

 Conflicts:
src/mds/Beacon.cc
src/mds/OpenFileTable.cc
        src/mds/MDSContext.h
        src/mds/MDSContext.cc

doc: remove deprecated 'scrubq' from ceph(8)

The option was removed in a30cbe9adea34782be8699c010d4281d245feb6b

Fixes: http://tracker.ceph.com/issues/35813
Signed-off-by: Ruben Kerkhof <ruben@rubenkerkhof.com>
(cherry picked from commit d05cd1a3e612f52eab9bde42c184022205f65384)

doc/man/ceph-volume update batch command with new sizing flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 042c2cbe475023cd3c34102b710b93c60d92077e)

ceph-volume tests.devices update fixture to include new size flags for strategies

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5df56a09c7b445c36d2b9d2874962c40b681a65c)

ceph-volume lvm.batch.filestore consume --journal-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2378c282c78f49f4d911316a7bc85e49a3f8e7f9)

ceph-volume lvm.batch.bluestore consume --block-db-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7e52bc559b21b18a06ff1dffd764dbe8aefe96a2)

ceph-volume lvm.batch add sizing flags for journal and block.db LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a16dbd7a8f597cde6debbdcd968cdaf44f6ce47)

Merge pull request #23773 from ifed01/wip-ifed-perf-reset-luminous

luminous: os/bluestore: bluestore_buffer_hit_bytes perf counter doesn't reset

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23817 from pdvian/wip-26992-luminous

luminous: discover_all_missing() not always called during activating

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23832 from pdvian/wip-27058-luminous

luminous: debian/rules: fix ceph-mgr .pyc files left behind

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23915 from liewegas/wip-pod-hostname-luminous

luminous: common: get real hostname from container/pod environment

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests.lvm verify that it is reported when --journal is not used with --filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ae657d2fe012b0e78959ead9d8d3baa1095a31ff)

ceph-volume lvm.prepare report back when --journal is not used when --filestore is set

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2fc1ee48ac10bd8008ff5d54382db1d83e57ec1b)

luminous:rgw: return x-amz-version-id: null when delete obj in versioning
suspended bucket

fix http://tracker.ceph.com/issues/35814

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 9c39147)

Merge pull request #24054 from smithfarm/wip-24983-luminous

luminous: 'radosgw-admin sync error trim' only trims partially

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24056 from smithfarm/wip-24985-luminous

luminous: multisite: object metadata operations are skipped by sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24069 from pdvian/wip-35709-luminous

luminous: rgw: fix deadlock on RGWIndexCompletionManager::stop

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23884 from smithfarm/wip-32084-luminous

luminous: mds: MDBalancer::try_rebalance() may stop prematurely

mds: fix premature stop of MDBalancer::try_rebalance()

When mds wants to export loads to multiple targets. The amount of single
export can easily be less than (0.2 * target_loads). For example:

6 active mds, mds.0's load is 11, other ranks' loads are 5. target_load
is 6. Export amount for each target is 1.

Fixes: http://tracker.ceph.com/issues/26973
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e68cbec6a647f0252d848620ce6e51b4f3eea1e3)

mds: don't queue straydir for split

straydir can not get splitted (for now at least)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 959e63c1f88b6ce6b2492cfb98e5875d928558e4)

Conflicts:
src/mds/MDBalancer.cc
- finesse split/merge conditional expression

Merge pull request #23789 from ukernel/luminous-mds-export-size

luminous: mds: optimize the way how max export size is enforced

Merge branch 'luminous' into luminous-mds-export-size

Merge pull request #23594 from dillaman/wip-23604-luminous

luminous: librbd: discard should wait for in-flight cache writeback to complete

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #24100 from pdvian/wip-35711-luminous

luminous: librbd: journaling unable request can not be sent to remote lock owner

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23793 from pdvian/wip-26983-luminous

luminous : client: fix choose_target_mds for requests that do name lookup

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23881 from smithfarm/wip-26977-luminous

luminous: cephfs-data-scan: print the max used ino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #23946 from ukernel/luminous-35721

luminous: mds: prevent MDSRank::evict_client from blocking finisher thread

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23990 from batrick/i35852

luminous: MDSMonitor: lookup of gid in prepare_beacon that has been removed will cause exception

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #24033 from smithfarm/wip-24934-luminous

luminous: cephfs-journal-tool: wrong layout info used

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24134 from batrick/qa-.qa-symlink

luminous: qa: add .qa helper link

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: raise debug level on redundant data sync error messages

each of these errors have already been logged at a lower level with a
more detailed error message. by logging them as ERRORs at level 0 here,
the messages could be easily confused as separate failures

Fixes: http://tracker.ceph.com/issues/36037
See-also: http://tracker.ceph.com/issues/35830

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a6e392f2b8cb885f6fede5dfa9b584659740e161)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23408 from ukernel/luminous-24820

luminous: mds: avoid using g_conf->get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23505 from ukernel/luminous-26834

luminous: mds: calculate load by checking self CPU usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23507 from ukernel/luminous-26858

luminous: mds: reset heartbeat map at potential time-consuming places

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23554 from batrick/i26906

luminous: MDSMonitor: consider raising priority of MMDSBeacons from MDS so they are processed before other client messages

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23556 from batrick/i26924

luminous: mds: mds got laggy because of MDSBeacon stuck in mqueue

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #23617 from pdvian/wip-26915-luminous

luminous: client: check for unmounted condition before printing debug output

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23677 from ukernel/luminous-26894

luminous: mds: crash when dumping ops in flight

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23678 from ukernel/luminous-24840

luminous: mds: explain delayed client_request due to subtree migration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23693 from tserlin/tserlin-require-fuse-luminous

luminous: debian/control: require fuse for ceph-fuse

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23726 from joscollin/wip-25205-luminous

luminous: CephVolumeClient: delay required after adding data pool to MDSMap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23758 from dillaman/wip-27987

luminous: librbd: fix refuse to release lock when cookie is the same at rewatch

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #23892 from pdvian/wip-27061-luminous

luminous: librbd: potential race on image create request complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24003 from pdvian/wip-35704-luminous

luminous: rbd: fix error import when the input is a pipe

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: discard should wait for in-flight cache writeback to complete

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(derived from commit 0e04de4bb9c08755bd0915776bd538ce8f7c2b5b)

qa: add .qa helper link

Barebones backport of #22719.

Command used:

    $ find qa/suites -type d -execdir ln -sfn ../.qa/ {}/.qa \;
    $ ln -sf ../ qa/suites/.qa

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23680 from k0ste/luminous_backports

luminous: osd: backport 'bench' and stdout changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23740 from k0ste/luminous_backports2

luminous: mgr/balancer: deepcopy best plan - otherwise we get latest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23791 from tchaikov/wip-23370-luminous

luminous: mgr/MgrClient: guard send_pgstats() with lock

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23862 from smithfarm/wip-25177-luminous

luminous: osd,mon: increase mon_max_pg_per_osd to 250

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23864 from smithfarm/wip-25203-luminous

luminous: rados python bindings use prval from stack

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23871 from smithfarm/wip-32106-luminous

luminous: object errors found in be_select_auth_object() aren't logged the same

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23901 from pdvian/wip-32078-luminous

luminous: test: Use pids instead of jobspecs which were wrong

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23943 from batrick/i35716

luminous: msg: "challenging authorizer" messages appear at debug_ms=0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23969 from zmedico/backport-lumious-17121-17239-17410-17820

luminous: PGPool::update optimizations

Reviewed-by: Robin H. Johnson <robbat2@gentoo.org>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23499 from pdvian/wip-26844-luminous

luminous: rgw_file: deep stat handling

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23522 from pdvian/wip-26846-luminous

luminous: rgw: Limit the number of lifecycle rules on one bucket.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #23545 from pdvian/wip-26848-luminous

luminous: rgw: The delete markers generated by object expiration should have owner

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23665 from pdvian/wip-26922-luminous

luminous: rgw: Fix log level of gc_iterate_entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23974 from pdvian/wip-35069-luminous

cls/rgw: ready rgw_usage_log_entry for extraction via ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24034 from smithfarm/wip-25087-luminous

luminous: change default rgw_thread_pool_size to 512

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24046 from pdvian/wip-35707-luminous

luminous: rgw: raise default rgw_curl_low_speed_time to 300 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: journaling unable request can not be sent to remote lock owner

Fixes: http://tracker.ceph.com/issues/26939
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 985535bae74106aec1938813bdb18661967f468f)

Merge pull request #24082 from alfredodeza/luminous-rm35970

luminous: ceph-volume fix zap not working with LVs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24080 from alfredodeza/luminous-rm35913

luminous: ceph-volume batch: allow --osds-per-device, default it to 1

Reviewed-by: Andrew Schoen aschoen@redhat.com

ceph-volume tests.functional comment why not use --destroy when zapping lvs in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 11858c74a1e704d63185f0ef4b6cd6aa02f17e52)

ceph-volume tests.functional use --destroy when zapping lvs in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 87af815691a7f43d14067c24e308d60f39d58051)

ceph-volume lvm.zap use remove_pv only for physical devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 036683fde799ab1cfad7d6192981515d6b3419cc)

ceph-volume tests.lvm.strategies initial filestore validation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d1d68eade56ba3f898cd3bfd29aa1ee3bf5460ec)

ceph-volume lvm.tests conftest fixtures for custom vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9f14db432317c15b29b187d1e1a4bd3a729a7043)

ceph-volume tests.devices.lvm add bluestore strategy cases

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2ced05a2bc6ba01cf8946f43d5839890a28cab20)

ceph-volume tests verify strategy validators for minimum size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9167f76c2a7107c82b781cbe5f51e24ee72e4db4)

ceph-volume lvm.batch.filestore allow osds-per-device

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 66947537b542da82185ea4d79bfaedaae3117918)

ceph-volume lvm.batch.validators account for osds per device, include collocation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 04a40eda0ac089c07654d9ab07958126f0fc8f84)

ceph-volume lvm.batch.bluestore allow osds-per-device with MixedType

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 028ef60df768c24a9573a044a451164dcb037677)

ceph-volume lvm.batch.bluestore allow osds-per-device with SingleType

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8c0e707110e284626fe4161097a3121170a3f09d)