common: fix BoundedKeyCounter const_pointer_iterator

with libc++, clang fails to compile a call to:

  vector::assign(const_pointer_iterator, const_pointer_iterator)

because const_pointer_iterator does not satisfy the InputIterator
concept. added the necessary typedefs for value_type and reference
to reflect the pointer type

Fixes: http://tracker.ceph.com/issues/22139
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18883 from cbodley/wip-22083+

rgw: set num_shards on 'radosgw-admin data sync init'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: set num_shards on 'radosgw-admin data sync init'

Fixes: http://tracker.ceph.com/issues/22083
Signed-off-by: Casey Bodley <cbodley@redhat.com>

dencoder/rgw: expose rgw sync status types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18940 from trociny/wip-valgrind-DeadLockOwner

test/librbd: valgrind warning in TestMockManagedLockBreakRequest.DeadLockOwner

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17761 from cbodley/wip-rgw-bilog-trim

rgw multisite: automated trimming for bucket index logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #18811 from oritwas/wip-rgw-22046

tenant fixes for dynamic resharding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18920 from yaozongyou/wip-ceph-dencoder-add-types

ceph-dencoder: add support for cls_rgw_lc_obj_head

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18946 from joscollin/wip-add-cbodley

.githubmap: Add cbodley

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18900 from qrGitHub/master

rgw.iam: change '1' to '1ULL' in function print_actions

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

.githubmap: Add cbodley

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18921 from Leeshine/wip-data-sync-cmd

rgw: add metadata and data sync related cmd into radosgw-admin usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: add data sync related cmd into radosgw-admin help info

+ radosgw-admin data sync status
+ radosgw-admin data sync init
+ radosgw-admin data sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add metadata sync related cmd into radosgw-admin help info

+ radosgw-admin metadata sync status
+ radosgw-admin metadata sync init
+ radosgw-admin metadata sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

test/librbd: valgrind warning in TestMockManagedLockBreakRequest.DeadLockOwner

When running with 'debug_rbd >= 20':

  Conditional jump or move depends on uninitialised value(s)
    by 0x94277B: operator<< (ostream:167)
    by 0x94277B: librbd::managed_lock::BreakRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_watchers(int) (BreakRequest.cc:101)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18905 from ifed01/wip-bluestore-recalc-alloc

os/bluestore: remove ineffective BlueFS fnode extent calculation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18903 from jan--f/jan-mgr-prometheus-unkown-pgstate

pybind/mgr/prometheus: don't crash when encountering an unknown PG state

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18885 from dzafman/wip-22112

ceph-objectstore-tool: Fix set-size to clear data_digest if changing …

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18622 from majianpeng/bluestore-opseq

os/bluestore: OpSequencer: reduce kv_submitted_waiters if _is_all_kv_submitted() return true.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #17032 from yangdongsheng/rbd_qos

rbd: implement image qos in tokenbucket algorithm

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18483 from Songweibin/wip-list-children2

 rbd: fix rbd children listing when child is in trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18007 from yehudasa/wip-rgw-config-desc

options.cc: document rgw config options

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <alekshmanan@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18894 from liewegas/wip-fix-balancer

mgr/balancer: skip CRUSH_ITEM_NONE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18763 from cfanz/master

rgw: don't write bucket_header when it is not changed in rgw_bucket_prepare_op

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18852 from theanalyst/wip-22083

rgw: data sync: set num_shards when building full maps

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18866 from cbodley/wip-22101

rgw: RGWEnv::set() takes std::string

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #18875 from yaozongyou/rgw-cleanup

rgw: remove unused disable_signal_fd

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18895 from cooboos/wip-opt-maperase

rgw: optimize the rgw-attr del code logic

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18845 from yaozongyou/wip-lifecycle-log-level

rgw: update life cycle related log level

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-dencoder: adding support for cls_rgw_lc_obj_head

After adding cls_rgw_lc_obj_head, we can using the following command
to look into lc processing related information:
rados -p default.rgw.log getomapheader lc.0 ./header.bin --namespace=lc
ceph-dencoder type cls_rgw_lc_obj_head import ./header.bin decode dump_json

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18873 from tchaikov/wip-bluestore-aio-log

os/bluestore: print aio in batch

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18888 from tchaikov/wip-osx

common,test,cmake: various changes to re-enable build on osx

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16369 from bspark8/wip_dmc2_client_tracker

dmclock: Delivery of the dmclock delta, rho and phase parameter + Enabling the client service tracker

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18798 from cbodley/wip-str-list-view

 common: add for_each_substr() for cheap string split

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: add new API methods and expand the rbd CLI to list trashed images

implement librbd::RBD::list_children2 and rbd_list_children2
methods and expand the rbd CLI to list trashed images

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: limit IO per second by TokenBucketThrottle

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #18663 from dragonylffly/wip-rbd-nbd-fix

rbd-nbd: bug fix when running in container

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18713 from liewegas/wip-snapset-snaps

osd/PrimaryLogPG: do not pull-up snapc to snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18882 from ceph/wip-cv-simple-testing

ceph-volume add functional tests for simple, rearrange lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

pybind/mgr/prometheus: don't crash when encountering an unknown PG state

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #18617 from tchaikov/wip-mon-cleanup

mon: remove unused waiting_for_commit

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove ineffective BlueFS fnode extent calculation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/balancer: skip CRUSH_ITEM_NONE

Signed-off-by: Sage Weil <sage@redhat.com>

rgw.iam: change '1' to '1ULL' in function print_actions

* IAM has 54 operations corresponding to 54 bits. 1 is a signed integer
* which can cover 31 operations, so we need 1ULL here.

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

Merge pull request #18879 from liewegas/wip-zstd

compressor/zstd: improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18837 from tchaikov/wip-warn-on-old-libstdcxx

cmake: warn if libstdc++ older than 5.1.0 is used

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: warn if libstdc++ older than 5.1.0 is used

__GLIBCPP__ is used before 3.4.0, and gcc 4.8.0 and up is required to
build c++11 source, hence it's safe to check __GLIBCXX__. for the
versioning of libstdc++, see
https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html#abi.versioning,
for the magic number of 20150422ul see
https://gcc.gnu.org/develop.html#timeline.
for the reason why we want to have this warning, see
https://github.com/ceph/ceph/pull/18755#issuecomment-342736554 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18815 from Liuchang0812/wip-kv-store-crc

tool: misc cleanup of ceph-kvstore-tool

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18828 from majianpeng/fix-21932

os/bluestore: enlarege aligned_size avoid too many vector(> IOV_MAX)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18854 from ifed01/wip-ifed-release-alloc-cleanup3

os/bluestore: avoid excessive ops in _txc_release_alloc

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18822 from jcsp/wip-22082

mon: clean up cluster logging on mon events

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: optimize the rgw-attr del code logic

Signed-off-by: wangsongbo wangsongbo@cloudin.cn

compressor/zstd: use compress_generic from compression, cleanup

This is based on Yann Collet's changes (but somewhat adapted).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18826 from liewegas/wip-22090

qa/suite/rados: fix balancer vs firefly tunables failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18805 from liewegas/wip-22039

os/bluestore: fix SharedBlob unregistration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18878 from liewegas/wip-perf-health

qa/suites/rados/perf: whitelist health warnings

Merge pull request #18884 from liewegas/wip-internal-safe-to-start-threads

common/config: make internal_safe_to_start_threads internal

Reviewed-by: Kefu Chai <kchai@redhat.com>

compressor/zstd: fix build options

Among other things, pass -O3, so that it goes faster.

Huge thank you to Yann Collet for identifying this as a build issue!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18825 from dillaman/wip-release-notes-rbd-caps

doc/release-notes: clarify that you need to keep your existing OSD caps

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

rados: use size_t for `max_obj_len`

otherwise, on some platforms where size_t is not uint64_t, MacOS for
instance, no matched generate_random_number() can be selected.

Signed-off-by: Kefu Chai <kchai@redhat.com>

bluestore: print aio in batch

KernelDevice::aio_{submit,write,read}() are critical paths. calling
cct->_conf->subsys.should_gather() multi-times is not optimal. the
downside of this issue is that if the aio is printed, the size of
buffer in PrebufferedStreambuf could be large if the number of iov is
large, that could be a heavy load to the memory subsystem.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should add snappy header dir for libceph_snappy

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/unittest_compression: fail if factory is not ready

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/blkdev: should include "blkdev.h"

otherwise `string` and `set` are not declared.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/PluginRegistry: define PLUGIN_SUFFIX to support osx

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-objectstore-tool: Fix set-size to clear data_digest if changing disk size

Signed-off-by: David Zafman <dzafman@redhat.com>

common/config: make internal_safe_to_start_threads internal

There is no reason for this to be exposed like a normal config option
(even a legacy one) since it is an internal safety flag.

We do keep the clear_() helper there for the unit test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18404 from cbodley/wip-vstart-mrgw

rgw: mrgw.sh uses instance name 'client.rgw'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

ceph-volume tests.functional update lvm/tox.ini paths after moving it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional move top level tox.ini to lvm/tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional move xenial to lvm/xenial

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move xenial/bluestore to xenial/bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move xenial/create to xenial/filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional move centos7 to lvm/centos7

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move bluestore to bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional lvm move create to filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: more documentation for bilog trim config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix notify timeout for BucketTrimWatcher

from seconds to msec

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: curly brace style for bilog trim classes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketTrimWatcher checks handle in stop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: hold cr reference in MetadataListCR callback

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add comment for bucket in BucketCounter

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move shard marker helper into BucketIndexShardsManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add kwargs for debug output

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add test_bucket_index_log_trim()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add TrimComplete to watch/notify api

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWDataSyncSingleEntryCR calls BucketChangeObserver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start BucketTrimManager in RGWRados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BucketTrimInstanceCR

fetches bucket sync status from each peer, calculates the min markers
for each shard, and trims the bilog shards. calls the TrimObserver on
success

Fixes: http://tracker.ceph.com/issues/18229
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add HTTPManager to BucketTrimManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWGetBucketInstanceInfoCR takes rgw_bucket or metadata key

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWRadosBILogTrimCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWBucketInfo overload for BucketShard::init

for use by callers that have already read the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add /admin/log rest api for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add json decoders for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_bucket_sync_status() to bypass manager

RGWBucketSyncStatusManager::init() is doing a lot of extra work that's
not needed to serve the rest api (spawning an http manager thread,
fetching the bucket instance info, etc)

uses RGWShardCollectCR to limit the number of concurrent reads to 16

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add 'radosgw-admin bilog autotrim'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add configure_bucket_trim()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketTrimManager implements BucketTrimObserver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: collect cold buckets for trim

Signed-off-by: Casey Bodley <cbodley@redhat.com>