vstart.sh: should quote the parameters to get them quoted

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18505 from badone/wip-unittest-crypto-getentropy-limit

unittest_crypto: Don't exceed limit for getentropy

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18181 from tchaikov/wip-vstart-quoted-print

vstart.sh: quote command only when necessary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #17266 into master

* refs/pull/17266/head:
qa: update test_ceph_argparse to test fs cmds
qa: use fs rm_data_pool
qa: fix mdsmap lookup
qa: remove usage of mds dump
PendingReleaseNotes: add obsoleted mds commands
qa: remove use of obsolete mds commands
ceph_volume_client: remove use of obsolete mds cmd
doc: update on obsolete mds commands
cephfs: obsolete deprecated mds commands

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #17909 from tchaikov/wip-coverity

test/objectstore/test_bluefs: cleanups

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: update test_ceph_argparse to test fs cmds

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18475 from yuyuyu101/wip-fix-alignment

os/ObjectStore: fix get_data_alignment return -1 causing problem in msg header

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #18297 from dzafman/wip-21803

osd/PG: fix objects degraded higher than 100%

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18479 from wzliao/wip-correct-scrub-cost

osd/OSD.cc: use option 'osd_scrub_cost' instead

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #18109 into master

* refs/pull/18109/head:
mds: clean up non-existent data pools in MDSMap
mds: reduce variable scope

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #18366 into master

* refs/pull/18366/head:
mon: s/mds_blacklist_interval/mon_mds_blacklist_interval/
mon: tidy up mds_blacklist_interval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18384 into master

* refs/pull/18384/head:
mds: disable early reply for dir layout and quota related requests
mds: make mksnap/setlayout wait for unsafe requests on other mds
mds: don't rdlock locks in replica object while auth mds is recovering
mds: move ScatterLock::state_flags into SimpleLock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18393 into master

* refs/pull/18393/head:
ceph_context: re-expand admin_socket metavariables in child process

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18490 from dillaman/wip-21894

rbd-mirror: strip environment/CLI overrides for remote cluster

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

qa: use fs rm_data_pool

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix mdsmap lookup

We were returning the file system structure and not the mdsmap.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove usage of mds dump

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18514 from ceph/wip-fix-cv-create-path

ceph-volume lvm.prepare update to use create_osd_path

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.prepare update to use create_osd_path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18500 from scienceluo/wip-luo-fix-doc-log-level-branch

doc: complete and update the subsystem logging level info table

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18467 from yuyuyu101/wip-21883

msg/async/AsyncConnection: state will be NONE if replacing by another one

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18469 from yuyuyu101/wip-remove-legacy

msg/async/AsyncConnection: remove legacy feature case handle

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18415 from majianpeng/msg-cleanup

msg/async: batch handle msg_iovlen

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18397 from tchaikov/wip-mon-opts

common/options: set max_background_jobs instead of max_background_compactions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18382 from ceph/wip-vstart-mgr-dashboard

cmake: add cython_rbd as a dependency to vstart target

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18480 from tchaikov/wip-doc-cleanup

doc: various cleanups

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18405 from dillaman/wip-21849

librbd: default to sparse-reads for any IO operation over 64K

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rpm,debian: use download.ceph.com/tarballs instead of ceph.com/download

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fix warning reported by "build-doc linkcheck"

all the HTTP 301 (moved permanently) should be killed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: more options in new-style

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: set max_background_jobs instead of max_background_compactions

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18409 from liu-chunmei/fix_dpdk_assert

Fix two dpdk assert happened in dpdk library.

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: complete and update the subsystem logging level info table

Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>

unittest_crypto: Don't exceed limit for getentropy

CryptoRandom::get_bytes calls getentropy directly if available and it
enforces a limit of 256 bytes.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #18460 from liewegas/wip-straw2-command

mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18428 from liewegas/wip-21878

os/bluestore/BlueFS: fix race with log flush during async log compaction

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #18470 from yuyuyu101/wip-tracker-lock

osd/OpRequest: reduce overhead when disabling tracking

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18472 from yuyuyu101/wip-kill-asynccompressor

compressor: kill AsyncCompressor which is broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Handle objects on non-actingbackfill nodes during recovery

With this fix we can see more misplaced and no degraded when
doing recovery to a new up/acting set with node(s) containing
objects that are not longer part of up/acting.

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Adjust misplaced calculation

When there are more backfill ndoes then misplaced nodes the
object calculation would cause misplace to drop too quickly.
Assuming that all backfill nodes are at approximate the same
place, we compute backfill per node and only adjust by
the number of misplaced nodes.

Fixes: http://tracker.ceph.com/issues/21898
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #18255 into master

* refs/pull/18255/head:
os/bluestore: handle compressed extents in blob unsharing checks

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17722 from amitkumar50/cov-rgw-6

rgw: Initialization of epoch,len

Merge pull request #17305 from theanalyst/rgw-lc-tagging

RGW: support for tagging in lifecycle policies

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #18291 from ZVampirEM77/wip-copy-errormsg

rgw: copy object add response error messages

Reviewed-by: Amit Kumar amitkuma@redhat.com
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18194 from ivancich/wip-mclock-repops-bitmap

osd: replace mclock subop opclass w/ rep_op opclass; combine duplicated code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18487 from liewegas/wip-20557

/etc/sysconfig/ceph: remove jemalloc option

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rbd-mirror: strip environment/CLI overrides for remote cluster

Fixes: http://tracker.ceph.com/issues/21894
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18448 from ceph/wip-bz1499840

ceph-volume lvm bluestore support

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18403 from dillaman/wip-librbd-flush

 librbd: sync flush should re-use existing async flush logic

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rbd: bench should flush prior to starting a test

For the new read-based bench tests, flushing prior to the start of the test
will result in the exclusive lock being acquired and the object map being
utilized.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

/etc/sysconfig/ceph: remove jemalloc option

This breaks when used with rocksdb, which is now the default.

See http://tracker.ceph.com/issues/20557

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: _update_calc_stats: backfill targets do not always affect degraded

Our progress on a backfill target should only count toward degraded if
the target is needed in order for us to reach the target pool size.  If
we have remapped other complete copies, we can backfill and not be degraded
at all.  And we may be some combination of the two if there are multiple
backfill targets; use the target(s) with the most objects.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18486 from liewegas/wip-20557

etc/default/ceph: remove jemalloc option

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

etc/default/ceph: remove jemalloc option

This breaks when used with rocksdb, which is now the default.

See http://tracker.ceph.com/issues/20557

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18408 from liewegas/wip-buffer-mempool-abi

buffer: avoid changing bufferlist ABI by removing new _mempool field

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18356 from wido/docs-zabbix-mgr

doc: Updated documentation for Zabbix Mgr module

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18474 from tchaikov/wip-debian-fix-upgrade

debian: fix package relationships after 40caf6a6

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.activate only prime-osd-dir when directory is empty

Signed-off-by: Alfredo Deza <adeza@redhat.com>

osd: replace mclock subop opclass w/ rep_op opclass; combine duplicated code

The OSD subops were replaced with replication ops (i.e., MSG_OSD_REPOP
and MSG_OSD_REPOPREPLY) in jewel. mClock incorrectly used them as an
op class rather than the newer replication ops. This fixes that and
places eight ops into the repop class -- MSG_OSD_REPOP,
MSG_OSD_REPOPREPLY, MSG_OSD_PG_UPDATE_LOG_MISSING,
MSG_OSD_PG_UPDATE_LOG_MISSING_REPLY, MSG_OSD_EC_WRITE,
MSG_OSD_EC_WRITE_REPLY, MSG_OSD_EC_READ, and MSG_OSD_EC_READ_REPLY.

This alo removes dupliated code related to op classes in
mClockOpClassQueue and mClockClientQueue and combines into
mClockOpClassSupport.

The three configuration options
osd_op_queue_mclock_osd_subop_{res,wgt,lim} have also been replaced
with osd_op_queue_mclock_osd_rep_op_{res,wgt,lim}.

This also removes OpQueueItem::OpQueueable::op_type_t::osd_subop since
it's not a category core ceph recognizes and is instead it's part of
an mclock view of ops.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

ceph-volume lvm.activate safeguard against auto detect objectstore flag not being set

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18423 from ashishkumsingh/wip-doc-fix-iscsi-chap-config

doc: Added CHAP configuration instructions for iSCSI

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2'

This is a common and recommended step that users should take once they
have clients that support straw2 (hammer!).  Let's make it easy.

This may result in a small amount of data movement, but usually it is
very little.

Signed-off-by: Sage Weil <sage@redhat.com>

debian: fix package relationships after d3ac8d18

d3ac8d18 moves ceph-client-debug from ceph-test to ceph-base without
updating the package relationships between the two involved packages.
which results in:

dpkg: error processing archive /var/cache/apt/archives/ceph-test_12.2.1-241-g43e027b-1trusty_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/ceph-client-debug', which is also in package ceph-base 10.2.10-14-gcbaddae-1trusty
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: fix package relationships after 40caf6a6

we have issues when running upgrade tests:

dpkg: error processing archive /var/cache/apt/archives/ceph-osd_13.0.0-2201-g6cc0b41-1trusty_amd64.deb (--unpack):
trying to overwrite '/usr/bin/ceph-osdomap-tool', which is also in package ceph-test 10.2.10-14-gcbaddae-1trusty

in 40caf6a6, we moves some tools from ceph-test out into ceph-osd,
ceph-mon and ceph-base respectively. but didn't update the relationships
between these packages accordingly. this causes the upgrade failure.

see https://www.debian.org/doc/debian-policy/#document-ch-relationships
for more details on "Breaks" and "Conflicts".

the reason why the package version to be replaced/conflicted is 12.2.2
is that: i assume that this change will be backported to luminous, and
the next release of it will be 12.2.2 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD.cc: use option 'osd_scrub_cost' instead

Signed-off-by: Liao Weizhong <liaoweizhong@unionpay.com>

msg/async: batch handle msg_iovlen.

Also remove redundan code "msg.msg_iovlen = 0".

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

doc: update manual-deployment with latest code

rbd pool is not created by default any more.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18478 from trociny/wip-valgrid-trim-unittest

test/librbd: memory leak in recently added test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15811 from x-ion-de/wip-20309

doc: Update manual deployment

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/librbd: memory leak in recently added test

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

compressor: kill AsyncCompressor which is broken

Signed-off-by: Haomai Wang <haomai@xsky.com>

os/ObjectStore: fix get_data_alignment return -1 causing problem in msg header

sizeof(data_off) in ceph_msg_header is uint16_t, so -1 will be 65535. When peer
side receive MOSDRepOp message, it will allocate extra 1 byte buffer to store
nonexisted buffer.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/simple/Pipe: remove legacy feature case handle

Signed-off-by: Haomai Wang <haomai@xsky.com>

ceph_context: re-expand admin_socket metavariables in child process

Reset admin_socket raw value if it is defined in conffile.
Just in case it used metavarirables (e.g, $pid) which could be expanded
again to the correct value in child process.

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

osd/OpRequest: don't mark_event when disable tracker

Signed-off-by: Haomai Wang <haomai@xsky.com>

common/TrackedOp: make tracking_enabled atomic without lock protect

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: remove legacy feature case handle

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: state will be NONE if replacing by another one

Fixes: http://tracker.ceph.com/issues/21883
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge PR #18372 into master

* refs/pull/18372/head:
ptl-tool: Fix more readme examples

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: default to sparse-reads for any IO operation over 64K

Testing BlueStore against both HDDs and OSDs with fully allocated
and sparse-allocated objects shows a performance improvement with
sparse-read between 32K and 64K.

Fixes: http://tracker.ceph.com/issues/21849
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: templatize io::CopyupRequest and io::ObjectRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: sync flush should re-use existing async flush logic

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18368 from kungf/scrub_week_day_bound

osd: add scrub week day constraint

Reviewed-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: add bucket_set_alg()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18400 from dillaman/wip-21844

osdc/Objecter: skip sparse-read result decode if bufferlist is empty

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18418 from dillaman/wip-21860

common/common_init: disable ms subsystem log gathering for clients

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18427 from dillaman/wip-21845

osdc/Objecter: delay initialization of hobject_t in _send_op

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18450 from liewegas/wip-osd-removing

osd: add numpg_removing metric

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18452 from liewegas/wip-21880

ceph_test_objectstore: disable filestore_fiemap for tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18454 from liewegas/wip-21882

messages/MOSDMap: do compat reencode of crush map, too

Reviewed-by: Loic Dachary <loic@dachary.org>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18457 from liewegas/wip-restful-health

qa/suites/rados/rest/mgr-restful: whitelist more health

qa/suites/rados/rest/mgr-restful: whitelist more health

The test is fiddling with OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDMap: do compat reencode of crush map, too

If we are reencoding an incremental, and it embeds a crush map, we need
to reencode that in a compatible way too.  This is especially true now
because we have the compat crush weight-sets.  Otherwise, a client may
learn the crush map through an incremental but not understand choose_args,
and not see/understand the alternate weight set.  It will send requests
to the wrong OSDs where they will just get dropped.

Fixes: http://tracker.ceph.com/issues/21882
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18406 from adamemerson/wip-insecticide

test,osd,kvstore_tool: silence warnings and prepare test buffer in the right way

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18379 from liewegas/wip-kill-replay-interval

osd/osd_types: pg_pool_t: remove crash_replay_interval member

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18407 from adamemerson/wip-upgrade-zstd

zstd: Upgrade to v1.3.2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18357 from asomers/have_libaio

Build: create a proper WITH_BLUESTORE option

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18354 from kungf/no_scurb_deadline

osd: make scrub no deadline when max interval is zero

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18396 from xiexingguo/wip-scrub-mismatch

osd/osd_types.cc: copy extents map too while making clone

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: disable filestore_fiemap for tests

This option was enabled in 87f33376d977962ab7438c46873ea9b6292390d1 but
causes ObjectStore/StoreTest.Synthetic/1 (filestore) to fail.  Revert that
bit for now until we fix fiemap properly.

See http://tracker.ceph.com/issues/21880

Signed-off-by: Sage Weil <sage@redhat.com>