Merge PR #18366 into master

* refs/pull/18366/head:
mon: s/mds_blacklist_interval/mon_mds_blacklist_interval/
mon: tidy up mds_blacklist_interval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18384 into master

* refs/pull/18384/head:
mds: disable early reply for dir layout and quota related requests
mds: make mksnap/setlayout wait for unsafe requests on other mds
mds: don't rdlock locks in replica object while auth mds is recovering
mds: move ScatterLock::state_flags into SimpleLock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18393 into master

* refs/pull/18393/head:
ceph_context: re-expand admin_socket metavariables in child process

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18490 from dillaman/wip-21894

rbd-mirror: strip environment/CLI overrides for remote cluster

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18514 from ceph/wip-fix-cv-create-path

ceph-volume lvm.prepare update to use create_osd_path

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.prepare update to use create_osd_path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18500 from scienceluo/wip-luo-fix-doc-log-level-branch

doc: complete and update the subsystem logging level info table

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18467 from yuyuyu101/wip-21883

msg/async/AsyncConnection: state will be NONE if replacing by another one

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18469 from yuyuyu101/wip-remove-legacy

msg/async/AsyncConnection: remove legacy feature case handle

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18415 from majianpeng/msg-cleanup

msg/async: batch handle msg_iovlen

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18397 from tchaikov/wip-mon-opts

common/options: set max_background_jobs instead of max_background_compactions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18382 from ceph/wip-vstart-mgr-dashboard

cmake: add cython_rbd as a dependency to vstart target

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18480 from tchaikov/wip-doc-cleanup

doc: various cleanups

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18405 from dillaman/wip-21849

librbd: default to sparse-reads for any IO operation over 64K

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rpm,debian: use download.ceph.com/tarballs instead of ceph.com/download

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix warning reported by "build-doc linkcheck"

all the HTTP 301 (moved permanently) should be killed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: more options in new-style

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: set max_background_jobs instead of max_background_compactions

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18409 from liu-chunmei/fix_dpdk_assert

Fix two dpdk assert happened in dpdk library.

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: complete and update the subsystem logging level info table

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

Merge pull request #18460 from liewegas/wip-straw2-command

mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18428 from liewegas/wip-21878

os/bluestore/BlueFS: fix race with log flush during async log compaction

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #18470 from yuyuyu101/wip-tracker-lock

osd/OpRequest: reduce overhead when disabling tracking

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18472 from yuyuyu101/wip-kill-asynccompressor

compressor: kill AsyncCompressor which is broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #18255 into master

* refs/pull/18255/head:
os/bluestore: handle compressed extents in blob unsharing checks

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17722 from amitkumar50/cov-rgw-6

rgw: Initialization of epoch,len

Merge pull request #17305 from theanalyst/rgw-lc-tagging

RGW: support for tagging in lifecycle policies

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18291 from ZVampirEM77/wip-copy-errormsg

rgw: copy object add response error messages

Reviewed-by: Amit Kumar amitkuma@redhat.com
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18194 from ivancich/wip-mclock-repops-bitmap

osd: replace mclock subop opclass w/ rep_op opclass; combine duplicated code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18487 from liewegas/wip-20557

/etc/sysconfig/ceph: remove jemalloc option

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rbd-mirror: strip environment/CLI overrides for remote cluster

Fixes: http://tracker.ceph.com/issues/21894
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18448 from ceph/wip-bz1499840

ceph-volume lvm bluestore support

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18403 from dillaman/wip-librbd-flush

 librbd: sync flush should re-use existing async flush logic

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rbd: bench should flush prior to starting a test

For the new read-based bench tests, flushing prior to the start of the test
will result in the exclusive lock being acquired and the object map being
utilized.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

/etc/sysconfig/ceph: remove jemalloc option

This breaks when used with rocksdb, which is now the default.

See http://tracker.ceph.com/issues/20557

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18486 from liewegas/wip-20557

etc/default/ceph: remove jemalloc option

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

etc/default/ceph: remove jemalloc option

This breaks when used with rocksdb, which is now the default.

See http://tracker.ceph.com/issues/20557

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18408 from liewegas/wip-buffer-mempool-abi

buffer: avoid changing bufferlist ABI by removing new _mempool field

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18356 from wido/docs-zabbix-mgr

doc: Updated documentation for Zabbix Mgr module

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18474 from tchaikov/wip-debian-fix-upgrade

debian: fix package relationships after 40caf6a6

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.activate only prime-osd-dir when directory is empty

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd: replace mclock subop opclass w/ rep_op opclass; combine duplicated code

The OSD subops were replaced with replication ops (i.e., MSG_OSD_REPOP
and MSG_OSD_REPOPREPLY) in jewel. mClock incorrectly used them as an
op class rather than the newer replication ops. This fixes that and
places eight ops into the repop class -- MSG_OSD_REPOP,
MSG_OSD_REPOPREPLY, MSG_OSD_PG_UPDATE_LOG_MISSING,
MSG_OSD_PG_UPDATE_LOG_MISSING_REPLY, MSG_OSD_EC_WRITE,
MSG_OSD_EC_WRITE_REPLY, MSG_OSD_EC_READ, and MSG_OSD_EC_READ_REPLY.

This alo removes dupliated code related to op classes in
mClockOpClassQueue and mClockClientQueue and combines into
mClockOpClassSupport.

The three configuration options
osd_op_queue_mclock_osd_subop_{res,wgt,lim} have also been replaced
with osd_op_queue_mclock_osd_rep_op_{res,wgt,lim}.

This also removes OpQueueItem::OpQueueable::op_type_t::osd_subop since
it's not a category core ceph recognizes and is instead it's part of
an mclock view of ops.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

ceph-volume lvm.activate safeguard against auto detect objectstore flag not being set

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18423 from ashishkumsingh/wip-doc-fix-iscsi-chap-config

doc: Added CHAP configuration instructions for iSCSI

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2'

This is a common and recommended step that users should take once they
have clients that support straw2 (hammer!).  Let's make it easy.

This may result in a small amount of data movement, but usually it is
very little.

Signed-off-by: Sage Weil <sage@redhat.com>

debian: fix package relationships after d3ac8d18

d3ac8d18 moves ceph-client-debug from ceph-test to ceph-base without
updating the package relationships between the two involved packages.
which results in:

dpkg: error processing archive /var/cache/apt/archives/ceph-test_12.2.1-241-g43e027b-1trusty_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/ceph-client-debug', which is also in package ceph-base 10.2.10-14-gcbaddae-1trusty
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: fix package relationships after 40caf6a6

we have issues when running upgrade tests:

dpkg: error processing archive /var/cache/apt/archives/ceph-osd_13.0.0-2201-g6cc0b41-1trusty_amd64.deb (--unpack):
trying to overwrite '/usr/bin/ceph-osdomap-tool', which is also in package ceph-test 10.2.10-14-gcbaddae-1trusty

in 40caf6a6, we moves some tools from ceph-test out into ceph-osd,
ceph-mon and ceph-base respectively. but didn't update the relationships
between these packages accordingly. this causes the upgrade failure.

see https://www.debian.org/doc/debian-policy/#document-ch-relationships
for more details on "Breaks" and "Conflicts".

the reason why the package version to be replaced/conflicted is 12.2.2
is that: i assume that this change will be backported to luminous, and
the next release of it will be 12.2.2 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: batch handle msg_iovlen.

Also remove redundan code "msg.msg_iovlen = 0".

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

doc: update manual-deployment with latest code

rbd pool is not created by default any more.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18478 from trociny/wip-valgrid-trim-unittest

test/librbd: memory leak in recently added test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15811 from x-ion-de/wip-20309

doc: Update manual deployment

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librbd: memory leak in recently added test

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

compressor: kill AsyncCompressor which is broken

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/simple/Pipe: remove legacy feature case handle

Signed-off-by: Haomai Wang <haomai@xsky.com>

ceph_context: re-expand admin_socket metavariables in child process

Reset admin_socket raw value if it is defined in conffile.
Just in case it used metavarirables (e.g, $pid) which could be expanded
again to the correct value in child process.

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

osd/OpRequest: don't mark_event when disable tracker

Signed-off-by: Haomai Wang <haomai@xsky.com>

common/TrackedOp: make tracking_enabled atomic without lock protect

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: remove legacy feature case handle

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: state will be NONE if replacing by another one

Fixes: http://tracker.ceph.com/issues/21883
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge PR #18372 into master

* refs/pull/18372/head:
ptl-tool: Fix more readme examples

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: default to sparse-reads for any IO operation over 64K

Testing BlueStore against both HDDs and OSDs with fully allocated
and sparse-allocated objects shows a performance improvement with
sparse-read between 32K and 64K.

Fixes: http://tracker.ceph.com/issues/21849
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: templatize io::CopyupRequest and io::ObjectRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: sync flush should re-use existing async flush logic

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18368 from kungf/scrub_week_day_bound

osd: add scrub week day constraint

Reviewed-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: add bucket_set_alg()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18400 from dillaman/wip-21844

osdc/Objecter: skip sparse-read result decode if bufferlist is empty

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18418 from dillaman/wip-21860

common/common_init: disable ms subsystem log gathering for clients

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18427 from dillaman/wip-21845

osdc/Objecter: delay initialization of hobject_t in _send_op

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18450 from liewegas/wip-osd-removing

osd: add numpg_removing metric

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18452 from liewegas/wip-21880

ceph_test_objectstore: disable filestore_fiemap for tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18454 from liewegas/wip-21882

messages/MOSDMap: do compat reencode of crush map, too

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18457 from liewegas/wip-restful-health

qa/suites/rados/rest/mgr-restful: whitelist more health

qa/suites/rados/rest/mgr-restful: whitelist more health

The test is fiddling with OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDMap: do compat reencode of crush map, too

If we are reencoding an incremental, and it embeds a crush map, we need
to reencode that in a compatible way too.  This is especially true now
because we have the compat crush weight-sets.  Otherwise, a client may
learn the crush map through an incremental but not understand choose_args,
and not see/understand the alternate weight set.  It will send requests
to the wrong OSDs where they will just get dropped.

Fixes: http://tracker.ceph.com/issues/21882
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18406 from adamemerson/wip-insecticide

test,osd,kvstore_tool: silence warnings and prepare test buffer in the right way

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18379 from liewegas/wip-kill-replay-interval

osd/osd_types: pg_pool_t: remove crash_replay_interval member

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18407 from adamemerson/wip-upgrade-zstd

zstd: Upgrade to v1.3.2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18357 from asomers/have_libaio

Build: create a proper WITH_BLUESTORE option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18354 from kungf/no_scurb_deadline

osd: make scrub no deadline when max interval is zero

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18396 from xiexingguo/wip-scrub-mismatch

osd/osd_types.cc: copy extents map too while making clone

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: disable filestore_fiemap for tests

This option was enabled in 87f33376d977962ab7438c46873ea9b6292390d1 but
causes ObjectStore/StoreTest.Synthetic/1 (filestore) to fail.  Revert that
bit for now until we fix fiemap properly.

See http://tracker.ceph.com/issues/21880

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16958 from mikulely/fix-full-uri

rgw: fix opslog uri as per Amazon s3

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17546 from mikulely/rgw-cleanup-min

rgw: cleanup MIN macro with std::min

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17934 from joke-lee/rgw_list_objecst_wrong_result_with_marker_when_enable_versioning

rgw:fix list objects with marker wrong result  when bucket is enable versioning

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17673 from mogeb/blkin

blkin: link against lttng-ust-fork

ceph-volume util.system use strings for comparison in py3

Signed-off-by: Alfredo Deza <adeza@redhat.com>

f ceph-volume tests update changes from the help menu in lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18414 from majianpeng/pmem-work

libmpem: Revert "submodule: make libmpem as a submodule."

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: add numpg_removing metric

Signed-off-by: Sage Weil <sage@redhat.com>

doc: Added CHAP cofiguration instructions for iSCSI

Added details to specify the CHAP username and password while
discovering/login the iSCSI target.

Signed-off-by: Ashish Singh <assingh@redhat.com>

ceph-volume lvm.common --journal-size doesn't accept a volume group

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare filestore does not require a volume group

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common volume groups are no longer required

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume activate workflow mention tmpfs in osd dirs

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update activation to include bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update create to indicate bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update prepare with bluestore workflow

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume update filestore to indicate it is not the only/default objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume elaborate on env vars for interval and tries in activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common update cli flags help for bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>