ceph-volume lvm.activate pass no_systemd args when detecting objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 29de6820a9a7108c8de41036add9332f9b910359)

ceph-volume tests.lvm verify activate works with --auto-detect-objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 60dbeb6184797ccffeaeea2e85ef2d7fb4a1c54c)

Merge pull request #24338 from alfredodeza/luminous-rm36251

luminous ceph-volume add new ceph-handlers role from ceph-ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional add new ceph-handlers role from ceph-ansible

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 956dc5e65c6e35e27ab6524d4fb9fdd05d616cf6)

Merge pull request #24084 from batrick/i24862

luminous: ceph_volume_client: allow atomic update of RADOS objects

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24282 from pdvian/wip-35933-luminous

luminous:client:two ceph-fuse client, one can not list out files created by an…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24188 from pdvian/wip-35844-luminous

luminous: osdc/Objecter: fix split vs reconnect race

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24245 from dillaman/wip-36116-luminous

luminous: test/librbd: not valid to have different parents between image snapshots

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #24257 from pdvian/wip-35929-luminous

luminous: mon/OSDMonitor: invalidate max_failed_since on cancel_report

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24259 from tchaikov/luminous-35969

luminous: rpm: use updated gperftools

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24307 from alfredodeza/luminous-batch-report-fix

luminous ceph-volume: fix journal and filestore data size in `lvm batch --report`

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: only run ceph-common once during tests

Using import_role we can have just one play and avoid rerunning
ceph-common over and over again in daemon specific plays.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a81386123e66e2d61391371cec4485b3148ff5a7)

ceph-volume: fail fast during tests

This also rsyncs the ceph-volume code to the testing vms before
a ceph.conf is generated because ceph-volume is needed now for this
to figure out the number of osds when you're using 'lvm batch'.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit eed244473d11a153cd7a37e0e9bfaf804a20db89)

ceph-volume: when using batch --report return journal/data size as ints

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a5e8bc97e70195cee54c4b66d10684d6cd3f5e38)

Merge pull request #24280 from alfredodeza/luminous-rm27210

luminous ceph-volume custom cluster names fail on filestore trigger

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24211 from pdvian/wip-35854-luminous

luminous: doc: remove deprecated 'scrubq' from ceph(8)

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24190 from joke-lee/luminous

luminous:rgw: return x-amz-version-id: null when delete obj in versio…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24231 from pdvian/wip-35856-luminous

luminous: multisite: segfault on shutdown/realm reload

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24171 from ukernel/luminous-25046

luminous: mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client:two ceph-fuse client, one can not list out files created by another.

the problem happend after mds restored the stale client session,
right after one normal client created files under one specific
dir,while the restored client can not list out the just created
files.

Fixes: http://tracker.ceph.com/issues/27051
Signed-off-by: Peng Xie <peng.hse@xtaotech.com>
(cherry picked from commit fa1c2be4a93da46252a0b53776696baf80840700)

Conflicts:
src/client/Client.cc : Resolved for cap.issued

ceph-volume tests/functional update dmcrypt test playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 96e29adf72804cb4fca62c04416b8fe2d23587bb)

ceph-volume tests/functional/batch update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5f72d438da94323c0b04f261b1a60b13eaf03966)

ceph-volume tests/functional/lvm update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b0d2dc66968f81192f152fa166fbc2cb4dcb438)

ceph-volume tests/functional update playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ba10b6820fd7f352ed6ba78246a6976c54783cbd)

ceph-volume main move out the load_ceph_conf_path to the configuration module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 532908099eb63bd805db8aecae1ef12301076410)

ceph-volume tests.devices patch configuration.load to avoid raising non-existing ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe80e5399e9dd1f6b9f94404d732a76f961d9eb4)

ceph-volume tests reset the cluster name after every unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 877ee1a27e0651355812e27fbc6ac0a87ea56a68)

ceph-volume tests.functional add custom cluster name to centos7-filestore-create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e61848dc97b638c6b8e60f0896d325f436142864)

ceph-volume lvm.activate load the config always by reading the cluster name from the LV

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a88cbfc73a258b51b20e060e269234962444ce63)

ceph-volume configuration be able to set conf.ceph

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f8226f359d94a86a39492ed653d9a9e5d8dd8f9)

Merge pull request #24135 from linuxbox2/luminous-rgw-sync-loglevel

luminous: rgw: raise debug level on redundant data sync error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rpm: use updated gperftools

make sure we only build with the higher version of gperftools on
distros where both 2.4 and 2.6.1 are packaged. see
https://git.centos.org/summary/rpms!gperftools.git . at the time of
writing, gperftools 2.6.1 is packaged for CentOS/RHEL 7, if gperftools
(>= 2.4) is required by Ceph, and user already has this version
installed, when new Ceph packages are installed, the updated gperftools
2.6.1 version won't be installed as a dependency. when launching
Ceph compiled with tcmalloc enabled, we will have

symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm

so, by bumping up the required version of gperftools, the updated
gperftools will be installed.

see https://software.opensuse.org/package/gperftools, openSUSE/SLE offer
2.5. so they are safe at this moment.

Fixes: http://tracker.ceph.com/issues/35969
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit e69e50a859acebe0bf854c12a71bfe745e0c1cf6)

mon/OSDMonitor: invalidate max_failed_since on cancel_report

max_failed_since might reference the very failure-report which is to be
cancelled. We can simply invalidate it here to make **get_failed_since()**
recalculate if necessary.

Fixes: http://tracker.ceph.com/issues/35860
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 91a2d408a9349db08e916e2145040df8b1f17e14)

Merge pull request #24065 from jdurgin/wip-luminous-cache-autotune

luminous: os/bluestore: cache autotuning and memory limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24115 from dzafman/wip-35991

luminous: tools: ceph-objectstore-tool: Allow target level as first positional …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/librbd: not valid to have different parents between image snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3b8f4f136b52d591dc39e6c4f8bd59c1e63c158a)

Fixes: http://tracker.ceph.com/issues/36118
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: RGWAsyncPutSystemObjAttrs does not access coroutine memory

also exposes optional objv_tracker to RGWSimpleRadosWriteAttrsCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b01b3952db4c45feb98dbc7e224ca1982c224c72)

rgw: RGWAsyncPutSystemObj does not access coroutine memory

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 85396674811be8f7a25dc9d25fd2db68c93561c5)

rgw: RGWAsyncGetSystemObj does not access coroutine memory

now reads the buffer/attrs into local memory, and calling coroutines
copy out the result during request_complete()

Fixes: http://tracker.ceph.com/issues/35543
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 089db9f74266073eb1d45900d982f1831c72fe59)

Conflicts:
src/rgw/rgw_cr_rados.h : Resolved in request_complete

Merge pull request #24206 from alfredodeza/luminous-rm24794

luminous ceph-volume: earlier detection for --journal and --filestore flag requirements

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24209 from alfredodeza/luminous-rm36088

luminous ceph-volume  batch: allow journal+block.db sizing on the CLI

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mds: health warning for slow metadata IO

Fixes: http://tracker.ceph.com/issues/24879
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 0f735f40315448560fde049ed3ea019a7d30d868)

 Conflicts:
src/mds/Beacon.cc
src/mds/OpenFileTable.cc
        src/mds/MDSContext.h
        src/mds/MDSContext.cc

doc: remove deprecated 'scrubq' from ceph(8)

The option was removed in a30cbe9adea34782be8699c010d4281d245feb6b

Fixes: http://tracker.ceph.com/issues/35813
Signed-off-by: Ruben Kerkhof <ruben@rubenkerkhof.com>
(cherry picked from commit d05cd1a3e612f52eab9bde42c184022205f65384)

doc/man/ceph-volume update batch command with new sizing flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 042c2cbe475023cd3c34102b710b93c60d92077e)

ceph-volume tests.devices update fixture to include new size flags for strategies

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5df56a09c7b445c36d2b9d2874962c40b681a65c)

ceph-volume lvm.batch.filestore consume --journal-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2378c282c78f49f4d911316a7bc85e49a3f8e7f9)

ceph-volume lvm.batch.bluestore consume --block-db-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7e52bc559b21b18a06ff1dffd764dbe8aefe96a2)

ceph-volume lvm.batch add sizing flags for journal and block.db LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a16dbd7a8f597cde6debbdcd968cdaf44f6ce47)

Merge pull request #23773 from ifed01/wip-ifed-perf-reset-luminous

luminous: os/bluestore: bluestore_buffer_hit_bytes perf counter doesn't reset

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23817 from pdvian/wip-26992-luminous

luminous: discover_all_missing() not always called during activating

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23832 from pdvian/wip-27058-luminous

luminous: debian/rules: fix ceph-mgr .pyc files left behind

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23915 from liewegas/wip-pod-hostname-luminous

luminous: common: get real hostname from container/pod environment

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests.lvm verify that it is reported when --journal is not used with --filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ae657d2fe012b0e78959ead9d8d3baa1095a31ff)

ceph-volume lvm.prepare report back when --journal is not used when --filestore is set

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2fc1ee48ac10bd8008ff5d54382db1d83e57ec1b)

luminous:rgw: return x-amz-version-id: null when delete obj in versioning
suspended bucket

fix http://tracker.ceph.com/issues/35814

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 9c39147)

osdc/Objecter: fix split vs reconnect race

If we are reconnecting to an OSD, we may not have a connection established
so this feature check is completely unreliable.  If there happens to be a
split, that could mean we don't resend an op that should have been resent.

Instead, we can infer the OSD has the new behavior if require_osd_release
is luminous or later.  If not, we can look at the per-osd features in the
OSDMap.

Fixes: http://tracker.ceph.com/issues/22544
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 917fdbe5422d099902aaa08b3406efc0247ef155)

Merge pull request #24054 from smithfarm/wip-24983-luminous

luminous: 'radosgw-admin sync error trim' only trims partially

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24056 from smithfarm/wip-24985-luminous

luminous: multisite: object metadata operations are skipped by sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24069 from pdvian/wip-35709-luminous

luminous: rgw: fix deadlock on RGWIndexCompletionManager::stop

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23884 from smithfarm/wip-32084-luminous

luminous: mds: MDBalancer::try_rebalance() may stop prematurely

mds: fix premature stop of MDBalancer::try_rebalance()

When mds wants to export loads to multiple targets. The amount of single
export can easily be less than (0.2 * target_loads). For example:

6 active mds, mds.0's load is 11, other ranks' loads are 5. target_load
is 6. Export amount for each target is 1.

Fixes: http://tracker.ceph.com/issues/26973
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e68cbec6a647f0252d848620ce6e51b4f3eea1e3)

mds: don't queue straydir for split

straydir can not get splitted (for now at least)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 959e63c1f88b6ce6b2492cfb98e5875d928558e4)

Conflicts:
src/mds/MDBalancer.cc
- finesse split/merge conditional expression

Merge pull request #23789 from ukernel/luminous-mds-export-size

luminous: mds: optimize the way how max export size is enforced

Merge branch 'luminous' into luminous-mds-export-size

Merge pull request #23594 from dillaman/wip-23604-luminous

luminous: librbd: discard should wait for in-flight cache writeback to complete

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #24100 from pdvian/wip-35711-luminous

luminous: librbd: journaling unable request can not be sent to remote lock owner

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23793 from pdvian/wip-26983-luminous

luminous : client: fix choose_target_mds for requests that do name lookup

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23881 from smithfarm/wip-26977-luminous

luminous: cephfs-data-scan: print the max used ino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #23946 from ukernel/luminous-35721

luminous: mds: prevent MDSRank::evict_client from blocking finisher thread

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23990 from batrick/i35852

luminous: MDSMonitor: lookup of gid in prepare_beacon that has been removed will cause exception

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #24033 from smithfarm/wip-24934-luminous

luminous: cephfs-journal-tool: wrong layout info used

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24134 from batrick/qa-.qa-symlink

luminous: qa: add .qa helper link

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: raise debug level on redundant data sync error messages

each of these errors have already been logged at a lower level with a
more detailed error message. by logging them as ERRORs at level 0 here,
the messages could be easily confused as separate failures

Fixes: http://tracker.ceph.com/issues/36037
See-also: http://tracker.ceph.com/issues/35830

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a6e392f2b8cb885f6fede5dfa9b584659740e161)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23408 from ukernel/luminous-24820

luminous: mds: avoid using g_conf->get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23505 from ukernel/luminous-26834

luminous: mds: calculate load by checking self CPU usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23507 from ukernel/luminous-26858

luminous: mds: reset heartbeat map at potential time-consuming places

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23554 from batrick/i26906

luminous: MDSMonitor: consider raising priority of MMDSBeacons from MDS so they are processed before other client messages

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23556 from batrick/i26924

luminous: mds: mds got laggy because of MDSBeacon stuck in mqueue

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #23617 from pdvian/wip-26915-luminous

luminous: client: check for unmounted condition before printing debug output

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23677 from ukernel/luminous-26894

luminous: mds: crash when dumping ops in flight

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23678 from ukernel/luminous-24840

luminous: mds: explain delayed client_request due to subtree migration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23693 from tserlin/tserlin-require-fuse-luminous

luminous: debian/control: require fuse for ceph-fuse

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23726 from joscollin/wip-25205-luminous

luminous: CephVolumeClient: delay required after adding data pool to MDSMap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23758 from dillaman/wip-27987

luminous: librbd: fix refuse to release lock when cookie is the same at rewatch

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #23892 from pdvian/wip-27061-luminous

luminous: librbd: potential race on image create request complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24003 from pdvian/wip-35704-luminous

luminous: rbd: fix error import when the input is a pipe

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: discard should wait for in-flight cache writeback to complete

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(derived from commit 0e04de4bb9c08755bd0915776bd538ce8f7c2b5b)

qa: add .qa helper link

Barebones backport of #22719.

Command used:

    $ find qa/suites -type d -execdir ln -sfn ../.qa/ {}/.qa \;
    $ ln -sf ../ qa/suites/.qa

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23680 from k0ste/luminous_backports

luminous: osd: backport 'bench' and stdout changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23740 from k0ste/luminous_backports2

luminous: mgr/balancer: deepcopy best plan - otherwise we get latest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23791 from tchaikov/wip-23370-luminous

luminous: mgr/MgrClient: guard send_pgstats() with lock

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23862 from smithfarm/wip-25177-luminous

luminous: osd,mon: increase mon_max_pg_per_osd to 250

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23864 from smithfarm/wip-25203-luminous

luminous: rados python bindings use prval from stack

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23871 from smithfarm/wip-32106-luminous

luminous: object errors found in be_select_auth_object() aren't logged the same

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23901 from pdvian/wip-32078-luminous

luminous: test: Use pids instead of jobspecs which were wrong

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23943 from batrick/i35716

luminous: msg: "challenging authorizer" messages appear at debug_ms=0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23969 from zmedico/backport-lumious-17121-17239-17410-17820

luminous: PGPool::update optimizations

Reviewed-by: Robin H. Johnson <robbat2@gentoo.org>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23499 from pdvian/wip-26844-luminous

luminous: rgw_file: deep stat handling

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23522 from pdvian/wip-26846-luminous

luminous: rgw: Limit the number of lifecycle rules on one bucket.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #23545 from pdvian/wip-26848-luminous

luminous: rgw: The delete markers generated by object expiration should have owner

Reviewed-by: Orit Wasserman <owasserm@redhat.com>