Merge pull request #24188 from pdvian/wip-35844-luminous

luminous: osdc/Objecter: fix split vs reconnect race

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24245 from dillaman/wip-36116-luminous

luminous: test/librbd: not valid to have different parents between image snapshots

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #24257 from pdvian/wip-35929-luminous

luminous: mon/OSDMonitor: invalidate max_failed_since on cancel_report

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #24259 from tchaikov/luminous-35969

luminous: rpm: use updated gperftools

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24307 from alfredodeza/luminous-batch-report-fix

luminous ceph-volume: fix journal and filestore data size in `lvm batch --report`

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: only run ceph-common once during tests

Using import_role we can have just one play and avoid rerunning
ceph-common over and over again in daemon specific plays.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a81386123e66e2d61391371cec4485b3148ff5a7)

ceph-volume: fail fast during tests

This also rsyncs the ceph-volume code to the testing vms before
a ceph.conf is generated because ceph-volume is needed now for this
to figure out the number of osds when you're using 'lvm batch'.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit eed244473d11a153cd7a37e0e9bfaf804a20db89)

ceph-volume: when using batch --report return journal/data size as ints

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a5e8bc97e70195cee54c4b66d10684d6cd3f5e38)

Merge pull request #24280 from alfredodeza/luminous-rm27210

luminous ceph-volume custom cluster names fail on filestore trigger

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24211 from pdvian/wip-35854-luminous

luminous: doc: remove deprecated 'scrubq' from ceph(8)

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24190 from joke-lee/luminous

luminous:rgw: return x-amz-version-id: null when delete obj in versio…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24231 from pdvian/wip-35856-luminous

luminous: multisite: segfault on shutdown/realm reload

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24171 from ukernel/luminous-25046

luminous: mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests/functional update dmcrypt test playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 96e29adf72804cb4fca62c04416b8fe2d23587bb)

ceph-volume tests/functional/batch update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5f72d438da94323c0b04f261b1a60b13eaf03966)

ceph-volume tests/functional/lvm update tests to use 'test' cluster name

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7b0d2dc66968f81192f152fa166fbc2cb4dcb438)

ceph-volume tests/functional update playbooks to use --cluster

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ba10b6820fd7f352ed6ba78246a6976c54783cbd)

ceph-volume main move out the load_ceph_conf_path to the configuration module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 532908099eb63bd805db8aecae1ef12301076410)

ceph-volume tests.devices patch configuration.load to avoid raising non-existing ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe80e5399e9dd1f6b9f94404d732a76f961d9eb4)

ceph-volume tests reset the cluster name after every unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 877ee1a27e0651355812e27fbc6ac0a87ea56a68)

ceph-volume tests.functional add custom cluster name to centos7-filestore-create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e61848dc97b638c6b8e60f0896d325f436142864)

ceph-volume lvm.activate load the config always by reading the cluster name from the LV

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a88cbfc73a258b51b20e060e269234962444ce63)

ceph-volume configuration be able to set conf.ceph

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f8226f359d94a86a39492ed653d9a9e5d8dd8f9)

Merge pull request #24135 from linuxbox2/luminous-rgw-sync-loglevel

luminous: rgw: raise debug level on redundant data sync error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rpm: use updated gperftools

make sure we only build with the higher version of gperftools on
distros where both 2.4 and 2.6.1 are packaged. see
https://git.centos.org/summary/rpms!gperftools.git . at the time of
writing, gperftools 2.6.1 is packaged for CentOS/RHEL 7, if gperftools
(>= 2.4) is required by Ceph, and user already has this version
installed, when new Ceph packages are installed, the updated gperftools
2.6.1 version won't be installed as a dependency. when launching
Ceph compiled with tcmalloc enabled, we will have

symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm

so, by bumping up the required version of gperftools, the updated
gperftools will be installed.

see https://software.opensuse.org/package/gperftools, openSUSE/SLE offer
2.5. so they are safe at this moment.

Fixes: http://tracker.ceph.com/issues/35969
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit e69e50a859acebe0bf854c12a71bfe745e0c1cf6)

mon/OSDMonitor: invalidate max_failed_since on cancel_report

max_failed_since might reference the very failure-report which is to be
cancelled. We can simply invalidate it here to make **get_failed_since()**
recalculate if necessary.

Fixes: http://tracker.ceph.com/issues/35860
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 91a2d408a9349db08e916e2145040df8b1f17e14)

Merge pull request #24065 from jdurgin/wip-luminous-cache-autotune

luminous: os/bluestore: cache autotuning and memory limit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24115 from dzafman/wip-35991

luminous: tools: ceph-objectstore-tool: Allow target level as first positional …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/librbd: not valid to have different parents between image snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3b8f4f136b52d591dc39e6c4f8bd59c1e63c158a)

Fixes: http://tracker.ceph.com/issues/36118
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: RGWAsyncPutSystemObjAttrs does not access coroutine memory

also exposes optional objv_tracker to RGWSimpleRadosWriteAttrsCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b01b3952db4c45feb98dbc7e224ca1982c224c72)

rgw: RGWAsyncPutSystemObj does not access coroutine memory

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 85396674811be8f7a25dc9d25fd2db68c93561c5)

rgw: RGWAsyncGetSystemObj does not access coroutine memory

now reads the buffer/attrs into local memory, and calling coroutines
copy out the result during request_complete()

Fixes: http://tracker.ceph.com/issues/35543
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 089db9f74266073eb1d45900d982f1831c72fe59)

Conflicts:
src/rgw/rgw_cr_rados.h : Resolved in request_complete

Merge pull request #24206 from alfredodeza/luminous-rm24794

luminous ceph-volume: earlier detection for --journal and --filestore flag requirements

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24209 from alfredodeza/luminous-rm36088

luminous ceph-volume  batch: allow journal+block.db sizing on the CLI

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mds: health warning for slow metadata IO

Fixes: http://tracker.ceph.com/issues/24879
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 0f735f40315448560fde049ed3ea019a7d30d868)

 Conflicts:
src/mds/Beacon.cc
src/mds/OpenFileTable.cc
        src/mds/MDSContext.h
        src/mds/MDSContext.cc

doc: remove deprecated 'scrubq' from ceph(8)

The option was removed in a30cbe9adea34782be8699c010d4281d245feb6b

Fixes: http://tracker.ceph.com/issues/35813
Signed-off-by: Ruben Kerkhof <ruben@rubenkerkhof.com>
(cherry picked from commit d05cd1a3e612f52eab9bde42c184022205f65384)

doc/man/ceph-volume update batch command with new sizing flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 042c2cbe475023cd3c34102b710b93c60d92077e)

ceph-volume tests.devices update fixture to include new size flags for strategies

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5df56a09c7b445c36d2b9d2874962c40b681a65c)

ceph-volume lvm.batch.filestore consume --journal-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2378c282c78f49f4d911316a7bc85e49a3f8e7f9)

ceph-volume lvm.batch.bluestore consume --block-db-size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7e52bc559b21b18a06ff1dffd764dbe8aefe96a2)

ceph-volume lvm.batch add sizing flags for journal and block.db LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0a16dbd7a8f597cde6debbdcd968cdaf44f6ce47)

Merge pull request #23773 from ifed01/wip-ifed-perf-reset-luminous

luminous: os/bluestore: bluestore_buffer_hit_bytes perf counter doesn't reset

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23817 from pdvian/wip-26992-luminous

luminous: discover_all_missing() not always called during activating

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23832 from pdvian/wip-27058-luminous

luminous: debian/rules: fix ceph-mgr .pyc files left behind

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23915 from liewegas/wip-pod-hostname-luminous

luminous: common: get real hostname from container/pod environment

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume tests.lvm verify that it is reported when --journal is not used with --filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ae657d2fe012b0e78959ead9d8d3baa1095a31ff)

ceph-volume lvm.prepare report back when --journal is not used when --filestore is set

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2fc1ee48ac10bd8008ff5d54382db1d83e57ec1b)

luminous:rgw: return x-amz-version-id: null when delete obj in versioning
suspended bucket

fix http://tracker.ceph.com/issues/35814

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 9c39147)

osdc/Objecter: fix split vs reconnect race

If we are reconnecting to an OSD, we may not have a connection established
so this feature check is completely unreliable.  If there happens to be a
split, that could mean we don't resend an op that should have been resent.

Instead, we can infer the OSD has the new behavior if require_osd_release
is luminous or later.  If not, we can look at the per-osd features in the
OSDMap.

Fixes: http://tracker.ceph.com/issues/22544
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 917fdbe5422d099902aaa08b3406efc0247ef155)

Merge pull request #24054 from smithfarm/wip-24983-luminous

luminous: 'radosgw-admin sync error trim' only trims partially

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24056 from smithfarm/wip-24985-luminous

luminous: multisite: object metadata operations are skipped by sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24069 from pdvian/wip-35709-luminous

luminous: rgw: fix deadlock on RGWIndexCompletionManager::stop

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23884 from smithfarm/wip-32084-luminous

luminous: mds: MDBalancer::try_rebalance() may stop prematurely

mds: fix premature stop of MDBalancer::try_rebalance()

When mds wants to export loads to multiple targets. The amount of single
export can easily be less than (0.2 * target_loads). For example:

6 active mds, mds.0's load is 11, other ranks' loads are 5. target_load
is 6. Export amount for each target is 1.

Fixes: http://tracker.ceph.com/issues/26973
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e68cbec6a647f0252d848620ce6e51b4f3eea1e3)

mds: don't queue straydir for split

straydir can not get splitted (for now at least)

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 959e63c1f88b6ce6b2492cfb98e5875d928558e4)

Conflicts:
src/mds/MDBalancer.cc
- finesse split/merge conditional expression

Merge pull request #23789 from ukernel/luminous-mds-export-size

luminous: mds: optimize the way how max export size is enforced

Merge branch 'luminous' into luminous-mds-export-size

Merge pull request #23594 from dillaman/wip-23604-luminous

luminous: librbd: discard should wait for in-flight cache writeback to complete

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #24100 from pdvian/wip-35711-luminous

luminous: librbd: journaling unable request can not be sent to remote lock owner

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23793 from pdvian/wip-26983-luminous

luminous : client: fix choose_target_mds for requests that do name lookup

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23881 from smithfarm/wip-26977-luminous

luminous: cephfs-data-scan: print the max used ino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #23946 from ukernel/luminous-35721

luminous: mds: prevent MDSRank::evict_client from blocking finisher thread

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23990 from batrick/i35852

luminous: MDSMonitor: lookup of gid in prepare_beacon that has been removed will cause exception

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #24033 from smithfarm/wip-24934-luminous

luminous: cephfs-journal-tool: wrong layout info used

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #24134 from batrick/qa-.qa-symlink

luminous: qa: add .qa helper link

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: raise debug level on redundant data sync error messages

each of these errors have already been logged at a lower level with a
more detailed error message. by logging them as ERRORs at level 0 here,
the messages could be easily confused as separate failures

Fixes: http://tracker.ceph.com/issues/36037
See-also: http://tracker.ceph.com/issues/35830

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a6e392f2b8cb885f6fede5dfa9b584659740e161)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23408 from ukernel/luminous-24820

luminous: mds: avoid using g_conf->get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23505 from ukernel/luminous-26834

luminous: mds: calculate load by checking self CPU usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23507 from ukernel/luminous-26858

luminous: mds: reset heartbeat map at potential time-consuming places

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23554 from batrick/i26906

luminous: MDSMonitor: consider raising priority of MMDSBeacons from MDS so they are processed before other client messages

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23556 from batrick/i26924

luminous: mds: mds got laggy because of MDSBeacon stuck in mqueue

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #23617 from pdvian/wip-26915-luminous

luminous: client: check for unmounted condition before printing debug output

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23677 from ukernel/luminous-26894

luminous: mds: crash when dumping ops in flight

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23678 from ukernel/luminous-24840

luminous: mds: explain delayed client_request due to subtree migration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23693 from tserlin/tserlin-require-fuse-luminous

luminous: debian/control: require fuse for ceph-fuse

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23726 from joscollin/wip-25205-luminous

luminous: CephVolumeClient: delay required after adding data pool to MDSMap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23758 from dillaman/wip-27987

luminous: librbd: fix refuse to release lock when cookie is the same at rewatch

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #23892 from pdvian/wip-27061-luminous

luminous: librbd: potential race on image create request complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24003 from pdvian/wip-35704-luminous

luminous: rbd: fix error import when the input is a pipe

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: discard should wait for in-flight cache writeback to complete

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(derived from commit 0e04de4bb9c08755bd0915776bd538ce8f7c2b5b)

qa: add .qa helper link

Barebones backport of #22719.

Command used:

    $ find qa/suites -type d -execdir ln -sfn ../.qa/ {}/.qa \;
    $ ln -sf ../ qa/suites/.qa

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23680 from k0ste/luminous_backports

luminous: osd: backport 'bench' and stdout changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23740 from k0ste/luminous_backports2

luminous: mgr/balancer: deepcopy best plan - otherwise we get latest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23791 from tchaikov/wip-23370-luminous

luminous: mgr/MgrClient: guard send_pgstats() with lock

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23862 from smithfarm/wip-25177-luminous

luminous: osd,mon: increase mon_max_pg_per_osd to 250

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23864 from smithfarm/wip-25203-luminous

luminous: rados python bindings use prval from stack

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23871 from smithfarm/wip-32106-luminous

luminous: object errors found in be_select_auth_object() aren't logged the same

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23901 from pdvian/wip-32078-luminous

luminous: test: Use pids instead of jobspecs which were wrong

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23943 from batrick/i35716

luminous: msg: "challenging authorizer" messages appear at debug_ms=0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23969 from zmedico/backport-lumious-17121-17239-17410-17820

luminous: PGPool::update optimizations

Reviewed-by: Robin H. Johnson <robbat2@gentoo.org>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23499 from pdvian/wip-26844-luminous

luminous: rgw_file: deep stat handling

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23522 from pdvian/wip-26846-luminous

luminous: rgw: Limit the number of lifecycle rules on one bucket.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #23545 from pdvian/wip-26848-luminous

luminous: rgw: The delete markers generated by object expiration should have owner

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23665 from pdvian/wip-26922-luminous

luminous: rgw: Fix log level of gc_iterate_entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23974 from pdvian/wip-35069-luminous

cls/rgw: ready rgw_usage_log_entry for extraction via ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24034 from smithfarm/wip-25087-luminous

luminous: change default rgw_thread_pool_size to 512

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24046 from pdvian/wip-35707-luminous

luminous: rgw: raise default rgw_curl_low_speed_time to 300 seconds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: journaling unable request can not be sent to remote lock owner

Fixes: http://tracker.ceph.com/issues/26939
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 985535bae74106aec1938813bdb18661967f468f)

Merge pull request #24082 from alfredodeza/luminous-rm35970

luminous: ceph-volume fix zap not working with LVs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24080 from alfredodeza/luminous-rm35913

luminous: ceph-volume batch: allow --osds-per-device, default it to 1

Reviewed-by: Andrew Schoen aschoen@redhat.com