rgw: more documentation for bilog trim config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix notify timeout for BucketTrimWatcher

from seconds to msec

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: curly brace style for bilog trim classes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketTrimWatcher checks handle in stop

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: hold cr reference in MetadataListCR callback

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add comment for bucket in BucketCounter

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move shard marker helper into BucketIndexShardsManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add kwargs for debug output

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add test_bucket_index_log_trim()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add TrimComplete to watch/notify api

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWDataSyncSingleEntryCR calls BucketChangeObserver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: start BucketTrimManager in RGWRados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BucketTrimInstanceCR

fetches bucket sync status from each peer, calculates the min markers
for each shard, and trims the bilog shards. calls the TrimObserver on
success

Fixes: http://tracker.ceph.com/issues/18229
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add HTTPManager to BucketTrimManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWGetBucketInstanceInfoCR takes rgw_bucket or metadata key

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWRadosBILogTrimCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWBucketInfo overload for BucketShard::init

for use by callers that have already read the bucket instance info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add /admin/log rest api for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add json decoders for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_bucket_sync_status() to bypass manager

RGWBucketSyncStatusManager::init() is doing a lot of extra work that's
not needed to serve the rest api (spawning an http manager thread,
fetching the bucket instance info, etc)

uses RGWShardCollectCR to limit the number of concurrent reads to 16

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add 'radosgw-admin bilog autotrim'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add configure_bucket_trim()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketTrimManager implements BucketTrimObserver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: collect cold buckets for trim

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BucketTrimStatus

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add MetadataListCR to loop over bucket instances

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BucketTrimCR to spawn trim for active buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BucketTrimPollCR for interval and lease logic

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add TrimCounters api to BucketTrimWatcher

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BucketTrimWatcher to serve watch/notify apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: BucketTrimManager implements BucketChangeObserver

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: introduce BoundedKeyCounter and unit test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add skeleton for BucketTrimManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: introduce RGWRadosNotifyCR for aio_notify

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: MetadataManager interface takes const string refs

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18821 from prallabh/wip-22084

rgw: Fix swift object expiry not deleting objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18846 from yaozongyou/fix-lifecycle-bug

rgw: check going_down() when lifecycle processing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18877 from Jeegn-Chen/wip-issue-22103

rados: make ceph_perf_msgr_client work for multiple jobs

Reviewed-by: Haomai Wang <haomai@xsky.com>

rados: make ceph_perf_msgr_client work for multiple jobs

Move the invocation of msgr->start() before a client initiates a connection.
So the nonce is probably initialized and the server side is able to differenciate
the connections from different Messengers from the same client.

Fixes: http://tracker.ceph.com/issues/22103
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18876 from smithfarm/wip-rn-typos

doc: release notes: fix grammar/style nits

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc: release notes: fix grammar/style nits

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #18862 from Jeegn-Chen/wip-issue-22100

rados: Copy payload in ceph_perf_msgr_client

Reviewed-by: Haomai Wang <haomai@xsky.com>

rados: Copy payload in ceph_perf_msgr_client

Copy payload in ceph_perf_msgr_client so that payload can keep unchanged

Fixes: http://tracker.ceph.com/issues/22100
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18707 from liewegas/wip-filestore-rocksdb-compression

os/filestore: disable rocksdb compression

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #18823 from ceph/wip-bz1498200

ceph-volume: support GPT and other deployed OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18817 from ZVampirEM77/wip-em-zone-cleanup

rgw cleanup: some unnecessary function called and repeated assignment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18683 from tchaikov/wip-erasure-cleanup

erasure-code: refactor the interfaces to hide internals from public

Reviewed-by: Sage Weil <sage@redhat.com>

doc/ceph-volume include the top level systemd.rst in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate add --file in help docstring as an example

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume activate has a --file option for json files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate enable the osd when activating

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl always force symlink when masking ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate use the contents of the JSON file, fallback to CLI args

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate only check for missing id and uuid if file is not passed in

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate allow to pass a file to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util add tests for OSDPath validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking ceph-disk needs to be done by directly symlinking to dev/null

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking needs to use systemctl mask, not disable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan if mounted device cannot be detected, raise an error

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate remove str formatting from logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators always convert the path to an absolute path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: Fix swift object expiry not deleting objects

In cls_timeindex_list() though `to_index` has expired for a timespan, the marker is set for a subsequent index during the time boundary check.
This marker is further returned to RGWObjectExpirer::process_single_shard(), where this out_marker is trimmed from the respective shard,
resulting in a lost removal hint and a leaked object.

Fixes: http://tracker.ceph.com/issues/22084
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

Merge pull request #18781 from iliul/correct-error-messages

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18796 from ivancich/wip-fix-fmt-warnings

rgw: address warnings due to incorrect format code

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17659 from majianpeng/osd-wait-ratherthan-waitinterval

osd/OSD: Using Wait rather than WaitInterval to wait queue.is_empty().

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18759 from tchaikov/wip-mon-log-last

mon/LogMonitor: "log last" should return up to n entries

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18779 from tchaikov/wip-osd-remove-unused-objects_read_async

osd: remove unused ReplicatedBackend::objects_read_async()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Xinze Chi <xinze@xsky.com>

Merge pull request #18809 from tchaikov/wip-21904

mgr/zabbix: ignore osd with 0 kb capacity

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Signed-off-by: iliul <liul.stone@gmail.com>

rgw: check going_down() when lifecycle processing

Fixes: http://tracker.ceph.com/issues/22099
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-volume util.arg_validators require superuser privs to check paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan skip directories to scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.simple add activate CLI unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices add help menu unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume systemd describe the interaction with 'simple' activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume index initial description of simple

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm add back a system.rst with just the lvm portion of it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume make systemd.rst generic enough to be related to other sub-commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume move lvm/systemd.rst to top level, making it generic

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume simple.activate initial take on documenting activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm.activate update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm.systemd update anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume document the scan functionality

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18793 from idryomov/wip-qa-cram-path

qa/tasks/cram: include /usr/sbin in the PATH for all commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18556 from joke-lee/validate_cors_rule_header_in_cors_option_request

rgw: add cors header rule check in cors option request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18795 from idryomov/wip-krbd-enable-generic-050-448

qa/suites/krbd: enable generic/050 and generic/448

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18706 from cbodley/wip-rgw-policy-emplace

rgw: simplify use of map::emplace in iam

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #18770 from tchaikov/wip-doc-log-file

doc: fix desc of option "mon cluster log file"

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: cleanup repeated assignment to need_zone_update in zone modify command

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

doc: fix desc of option "mon cluster log file"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: ignore osd with 0 kb capacity

Fixes: http://tracker.ceph.com/issues/21904
Signed-off-by: Ilja Slepnev <islepnev@gmail.com>

Merge PR #18807 into master

* refs/pull/18807/head:
ptl-tool: put common use-cases up front in README
ptl-tool: only create tag if new branch
ptl-tool: hint how to skip contributor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: put common use-cases up front in README

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: only create tag if new branch

It doesn't make sense to create tags for branches that already exist.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: hint how to skip contributor

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: address warnings due to incorrect format code

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #18709 from ivancich/wip-bktidx-race-bugfix

rgw: fix BZ 1500904, stale bucket index entry remains after obj delete

qa/suites/krbd: enable generic/050 and generic/448

generic/448 has been failing on ext4 and is now fixed upstream.
The fix for generic/050 is in testing, scheduled for 4.15-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #18476 from majianpeng/objecter-misc-cleanup

osdc/Objecter: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>