Merge pull request #18845 from yaozongyou/wip-lifecycle-log-level

rgw: update life cycle related log level

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18873 from tchaikov/wip-bluestore-aio-log

os/bluestore: print aio in batch

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18888 from tchaikov/wip-osx

common,test,cmake: various changes to re-enable build on osx

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16369 from bspark8/wip_dmc2_client_tracker

dmclock: Delivery of the dmclock delta, rho and phase parameter + Enabling the client service tracker

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18798 from cbodley/wip-str-list-view

 common: add for_each_substr() for cheap string split

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18663 from dragonylffly/wip-rbd-nbd-fix

rbd-nbd: bug fix when running in container

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18713 from liewegas/wip-snapset-snaps

osd/PrimaryLogPG: do not pull-up snapc to snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18882 from ceph/wip-cv-simple-testing

ceph-volume add functional tests for simple, rearrange lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18617 from tchaikov/wip-mon-cleanup

mon: remove unused waiting_for_commit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18879 from liewegas/wip-zstd

compressor/zstd: improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18837 from tchaikov/wip-warn-on-old-libstdcxx

cmake: warn if libstdc++ older than 5.1.0 is used

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: warn if libstdc++ older than 5.1.0 is used

__GLIBCPP__ is used before 3.4.0, and gcc 4.8.0 and up is required to
build c++11 source, hence it's safe to check __GLIBCXX__. for the
versioning of libstdc++, see
https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html#abi.versioning,
for the magic number of 20150422ul see
https://gcc.gnu.org/develop.html#timeline.
for the reason why we want to have this warning, see
https://github.com/ceph/ceph/pull/18755#issuecomment-342736554 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18815 from Liuchang0812/wip-kv-store-crc

tool: misc cleanup of ceph-kvstore-tool

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18828 from majianpeng/fix-21932

os/bluestore: enlarege aligned_size avoid too many vector(> IOV_MAX)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18854 from ifed01/wip-ifed-release-alloc-cleanup3

os/bluestore: avoid excessive ops in _txc_release_alloc

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18822 from jcsp/wip-22082

mon: clean up cluster logging on mon events

Reviewed-by: Sage Weil <sage@redhat.com>

compressor/zstd: use compress_generic from compression, cleanup

This is based on Yann Collet's changes (but somewhat adapted).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18826 from liewegas/wip-22090

qa/suite/rados: fix balancer vs firefly tunables failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18805 from liewegas/wip-22039

os/bluestore: fix SharedBlob unregistration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18878 from liewegas/wip-perf-health

qa/suites/rados/perf: whitelist health warnings

Merge pull request #18884 from liewegas/wip-internal-safe-to-start-threads

common/config: make internal_safe_to_start_threads internal

Reviewed-by: Kefu Chai <kchai@redhat.com>

compressor/zstd: fix build options

Among other things, pass -O3, so that it goes faster.

Huge thank you to Yann Collet for identifying this as a build issue!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18825 from dillaman/wip-release-notes-rbd-caps

doc/release-notes: clarify that you need to keep your existing OSD caps

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rados: use size_t for `max_obj_len`

otherwise, on some platforms where size_t is not uint64_t, MacOS for
instance, no matched generate_random_number() can be selected.

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: print aio in batch

KernelDevice::aio_{submit,write,read}() are critical paths. calling
cct->_conf->subsys.should_gather() multi-times is not optimal. the
downside of this issue is that if the aio is printed, the size of
buffer in PrebufferedStreambuf could be large if the number of iov is
large, that could be a heavy load to the memory subsystem.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should add snappy header dir for libceph_snappy

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/unittest_compression: fail if factory is not ready

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/blkdev: should include "blkdev.h"

otherwise `string` and `set` are not declared.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/PluginRegistry: define PLUGIN_SUFFIX to support osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: make internal_safe_to_start_threads internal

There is no reason for this to be exposed like a normal config option
(even a legacy one) since it is an internal safety flag.

We do keep the clear_() helper there for the unit test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18404 from cbodley/wip-vstart-mrgw

rgw: mrgw.sh uses instance name 'client.rgw'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-volume tests.functional update lvm/tox.ini paths after moving it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional move top level tox.ini to lvm/tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional move xenial to lvm/xenial

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move xenial/bluestore to xenial/bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move xenial/create to xenial/filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional move centos7 to lvm/centos7

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move bluestore to bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move create to filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional simple xenial bluestore scenario

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional simple xenial filestore scenario

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional simple tox.ini update for xenial scnearios

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional simple tox.ini addition for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add initial files for simple bluestore activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add initial files for simple filestore activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

compressor/zstd: improvements

- static linking #define to get the advanced stuff
- hint src size to compressor
- no flush needed before end
- check for error

It's about twice as 25% faster now for 16KB chunks.

Note that a quick test to use resetCStream and reusing the same stream
did not appear to have any effect.

Signed-off-by: Sage Weil <sage@redhat.com>

common: get_str_vec and friends use for_each_substr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add for_each_substr() for cheap string split

using boost::string_view avoids copies (and potential allocation) of
the substrings

using a callback model avoids baking in the container type, along with
the copies and allocations required for insertion

this interface is ideal for cases where you just want to iterate over
the substrings, and don't actually need to store them in a container

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/common: extend str_list tests to include set

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add simplified interfaces to get_str_*

the simpler interfaces rely on return value optimization to avoid
copying the result. removing the container from the argument list makes
it easier to provide a default argument for 'delims'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18821 from prallabh/wip-22084

rgw: Fix swift object expiry not deleting objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18846 from yaozongyou/fix-lifecycle-bug

rgw: check going_down() when lifecycle processing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/rados/perf: whitelist health warnings

We may see these as cluster stabilizes.

Signed-off-by: Sage Weil <sage@redhat.com>

common/buffers: add function parameter to limit buffers size.

Fixes: http://tracker.ceph.com/issues/21932
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18877 from Jeegn-Chen/wip-issue-22103

rados: make ceph_perf_msgr_client work for multiple jobs

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa/suites/rados: stop testing firefly tunables

We can't mix the balancer compat-set testing with firefly tunables because
it requires that all buckets be straw2.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: fail a bit more gracefully if we can't create a compat weight-set

Signed-off-by: Sage Weil <sage@redhat.com>

rados: make ceph_perf_msgr_client work for multiple jobs

Move the invocation of msgr->start() before a client initiates a connection.
So the nonce is probably initialized and the server side is able to differenciate
the connections from different Messengers from the same client.

Fixes: http://tracker.ceph.com/issues/22103
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18876 from smithfarm/wip-rn-typos

doc: release notes: fix grammar/style nits

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc: release notes: fix grammar/style nits

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon: clean up cluster logging on mon events

These changes come from observing the output
when killing a mon and watching the survivors
form a new quorum.

Fixes: http://tracker.ceph.com/issues/22082
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18862 from Jeegn-Chen/wip-issue-22100

rados: Copy payload in ceph_perf_msgr_client

Reviewed-by: Haomai Wang <haomai@xsky.com>

rados: Copy payload in ceph_perf_msgr_client

Copy payload in ceph_perf_msgr_client so that payload can keep unchanged

Fixes: http://tracker.ceph.com/issues/22100
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18707 from liewegas/wip-filestore-rocksdb-compression

os/filestore: disable rocksdb compression

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

ceph-volume tests.functional create a specific test.yml for simple activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional create a separate tox.ini environ for
`simple`

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18823 from ceph/wip-bz1498200

ceph-volume: support GPT and other deployed OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

os/bluestore: fix SharedBlob unregistration

We use the SharedBlobSet remove() in three cases:

- from SharedBlob::put(), we try to remove ourselves from the set, but
  have to deal with a racing lookup, so the removal is conditional on
  nref still being 0.
- from split_cache(), we move the SharedBlob to another collection
- from make_blob_unshared(), we remove the entry when we clear the sbid.

The problem is that the condtiional remove() (for the first case) was being
used for all three cases, and in the second two cases nref is always != 0,
so it doesn't actually happen.  This can lead to a crash during cache
shutdown.

Fix by making two variants: remove() that is unconditional, and
try_remove() that is conditional.

Set the sb->coll pointer after because remove() asserts the parent matches
where we are unregistering.

Fixes: http://tracker.ceph.com/issues/22039
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18817 from ZVampirEM77/wip-em-zone-cleanup

rgw cleanup: some unnecessary function called and repeated assignment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18683 from tchaikov/wip-erasure-cleanup

erasure-code: refactor the interfaces to hide internals from public

Reviewed-by: Sage Weil <sage@redhat.com>

doc/ceph-volume include the top level systemd.rst in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate add --file in help docstring as an example

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume activate has a --file option for json files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate enable the osd when activating

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl always force symlink when masking ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate use the contents of the JSON file, fallback to CLI args

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate only check for missing id and uuid if file is not passed in

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate allow to pass a file to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util add tests for OSDPath validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking ceph-disk needs to be done by directly symlinking to dev/null

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking needs to use systemctl mask, not disable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan if mounted device cannot be detected, raise an error

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate remove str formatting from logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators always convert the path to an absolute path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: Fix swift object expiry not deleting objects

In cls_timeindex_list() though `to_index` has expired for a timespan, the marker is set for a subsequent index during the time boundary check.
This marker is further returned to RGWObjectExpirer::process_single_shard(), where this out_marker is trimmed from the respective shard,
resulting in a lost removal hint and a leaked object.

Fixes: http://tracker.ceph.com/issues/22084
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

os/bluestore: avoid excessive ops in _txc_release_alloc

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18781 from iliul/correct-error-messages

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18796 from ivancich/wip-fix-fmt-warnings

rgw: address warnings due to incorrect format code

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17659 from majianpeng/osd-wait-ratherthan-waitinterval

osd/OSD: Using Wait rather than WaitInterval to wait queue.is_empty().

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18759 from tchaikov/wip-mon-log-last

mon/LogMonitor: "log last" should return up to n entries

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18779 from tchaikov/wip-osd-remove-unused-objects_read_async

osd: remove unused ReplicatedBackend::objects_read_async()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Xinze Chi <xinze@xsky.com>

Merge pull request #18809 from tchaikov/wip-21904

mgr/zabbix: ignore osd with 0 kb capacity

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Signed-off-by: iliul <liul.stone@gmail.com>

rgw: check going_down() when lifecycle processing

Fixes: http://tracker.ceph.com/issues/22099
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-volume util.arg_validators require superuser privs to check paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan skip directories to scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.simple add activate CLI unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices add help menu unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume systemd describe the interaction with 'simple' activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume index initial description of simple

Signed-off-by: Alfredo Deza <adeza@redhat.com>