rgw: set num_shards on 'radosgw-admin data sync init'

Fixes: http://tracker.ceph.com/issues/22083
Signed-off-by: Casey Bodley <cbodley@redhat.com>

dencoder/rgw: expose rgw sync status types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18617 from tchaikov/wip-mon-cleanup

mon: remove unused waiting_for_commit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18879 from liewegas/wip-zstd

compressor/zstd: improvements

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18837 from tchaikov/wip-warn-on-old-libstdcxx

cmake: warn if libstdc++ older than 5.1.0 is used

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: warn if libstdc++ older than 5.1.0 is used

__GLIBCPP__ is used before 3.4.0, and gcc 4.8.0 and up is required to
build c++11 source, hence it's safe to check __GLIBCXX__. for the
versioning of libstdc++, see
https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html#abi.versioning,
for the magic number of 20150422ul see
https://gcc.gnu.org/develop.html#timeline.
for the reason why we want to have this warning, see
https://github.com/ceph/ceph/pull/18755#issuecomment-342736554 .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18815 from Liuchang0812/wip-kv-store-crc

tool: misc cleanup of ceph-kvstore-tool

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18828 from majianpeng/fix-21932

os/bluestore: enlarege aligned_size avoid too many vector(> IOV_MAX)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18854 from ifed01/wip-ifed-release-alloc-cleanup3

os/bluestore: avoid excessive ops in _txc_release_alloc

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18822 from jcsp/wip-22082

mon: clean up cluster logging on mon events

Reviewed-by: Sage Weil <sage@redhat.com>

compressor/zstd: use compress_generic from compression, cleanup

This is based on Yann Collet's changes (but somewhat adapted).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18826 from liewegas/wip-22090

qa/suite/rados: fix balancer vs firefly tunables failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18805 from liewegas/wip-22039

os/bluestore: fix SharedBlob unregistration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18878 from liewegas/wip-perf-health

qa/suites/rados/perf: whitelist health warnings

Merge pull request #18884 from liewegas/wip-internal-safe-to-start-threads

common/config: make internal_safe_to_start_threads internal

Reviewed-by: Kefu Chai <kchai@redhat.com>

compressor/zstd: fix build options

Among other things, pass -O3, so that it goes faster.

Huge thank you to Yann Collet for identifying this as a build issue!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18825 from dillaman/wip-release-notes-rbd-caps

doc/release-notes: clarify that you need to keep your existing OSD caps

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

common/config: make internal_safe_to_start_threads internal

There is no reason for this to be exposed like a normal config option
(even a legacy one) since it is an internal safety flag.

We do keep the clear_() helper there for the unit test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18404 from cbodley/wip-vstart-mrgw

rgw: mrgw.sh uses instance name 'client.rgw'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

compressor/zstd: improvements

- static linking #define to get the advanced stuff
- hint src size to compressor
- no flush needed before end
- check for error

It's about twice as 25% faster now for 16KB chunks.

Note that a quick test to use resetCStream and reusing the same stream
did not appear to have any effect.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18821 from prallabh/wip-22084

rgw: Fix swift object expiry not deleting objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18846 from yaozongyou/fix-lifecycle-bug

rgw: check going_down() when lifecycle processing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/rados/perf: whitelist health warnings

We may see these as cluster stabilizes.

Signed-off-by: Sage Weil <sage@redhat.com>

common/buffers: add function parameter to limit buffers size.

Fixes: http://tracker.ceph.com/issues/21932
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #18877 from Jeegn-Chen/wip-issue-22103

rados: make ceph_perf_msgr_client work for multiple jobs

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa/suites/rados: stop testing firefly tunables

We can't mix the balancer compat-set testing with firefly tunables because
it requires that all buckets be straw2.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: fail a bit more gracefully if we can't create a compat weight-set

Signed-off-by: Sage Weil <sage@redhat.com>

rados: make ceph_perf_msgr_client work for multiple jobs

Move the invocation of msgr->start() before a client initiates a connection.
So the nonce is probably initialized and the server side is able to differenciate
the connections from different Messengers from the same client.

Fixes: http://tracker.ceph.com/issues/22103
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18876 from smithfarm/wip-rn-typos

doc: release notes: fix grammar/style nits

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc: release notes: fix grammar/style nits

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mon: clean up cluster logging on mon events

These changes come from observing the output
when killing a mon and watching the survivors
form a new quorum.

Fixes: http://tracker.ceph.com/issues/22082
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18862 from Jeegn-Chen/wip-issue-22100

rados: Copy payload in ceph_perf_msgr_client

Reviewed-by: Haomai Wang <haomai@xsky.com>

rados: Copy payload in ceph_perf_msgr_client

Copy payload in ceph_perf_msgr_client so that payload can keep unchanged

Fixes: http://tracker.ceph.com/issues/22100
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18707 from liewegas/wip-filestore-rocksdb-compression

os/filestore: disable rocksdb compression

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #18823 from ceph/wip-bz1498200

ceph-volume: support GPT and other deployed OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

os/bluestore: fix SharedBlob unregistration

We use the SharedBlobSet remove() in three cases:

- from SharedBlob::put(), we try to remove ourselves from the set, but
  have to deal with a racing lookup, so the removal is conditional on
  nref still being 0.
- from split_cache(), we move the SharedBlob to another collection
- from make_blob_unshared(), we remove the entry when we clear the sbid.

The problem is that the condtiional remove() (for the first case) was being
used for all three cases, and in the second two cases nref is always != 0,
so it doesn't actually happen.  This can lead to a crash during cache
shutdown.

Fix by making two variants: remove() that is unconditional, and
try_remove() that is conditional.

Set the sb->coll pointer after because remove() asserts the parent matches
where we are unregistering.

Fixes: http://tracker.ceph.com/issues/22039
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18817 from ZVampirEM77/wip-em-zone-cleanup

rgw cleanup: some unnecessary function called and repeated assignment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18683 from tchaikov/wip-erasure-cleanup

erasure-code: refactor the interfaces to hide internals from public

Reviewed-by: Sage Weil <sage@redhat.com>

doc/ceph-volume include the top level systemd.rst in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate add --file in help docstring as an example

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume activate has a --file option for json files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate enable the osd when activating

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl always force symlink when masking ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate use the contents of the JSON file, fallback to CLI args

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate only check for missing id and uuid if file is not passed in

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate allow to pass a file to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util add tests for OSDPath validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking ceph-disk needs to be done by directly symlinking to dev/null

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking needs to use systemctl mask, not disable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan if mounted device cannot be detected, raise an error

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate remove str formatting from logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators always convert the path to an absolute path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: Fix swift object expiry not deleting objects

In cls_timeindex_list() though `to_index` has expired for a timespan, the marker is set for a subsequent index during the time boundary check.
This marker is further returned to RGWObjectExpirer::process_single_shard(), where this out_marker is trimmed from the respective shard,
resulting in a lost removal hint and a leaked object.

Fixes: http://tracker.ceph.com/issues/22084
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

os/bluestore: avoid excessive ops in _txc_release_alloc

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18781 from iliul/correct-error-messages

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18796 from ivancich/wip-fix-fmt-warnings

rgw: address warnings due to incorrect format code

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17659 from majianpeng/osd-wait-ratherthan-waitinterval

osd/OSD: Using Wait rather than WaitInterval to wait queue.is_empty().

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18759 from tchaikov/wip-mon-log-last

mon/LogMonitor: "log last" should return up to n entries

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18779 from tchaikov/wip-osd-remove-unused-objects_read_async

osd: remove unused ReplicatedBackend::objects_read_async()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Xinze Chi <xinze@xsky.com>

Merge pull request #18809 from tchaikov/wip-21904

mgr/zabbix: ignore osd with 0 kb capacity

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Signed-off-by: iliul <liul.stone@gmail.com>

rgw: check going_down() when lifecycle processing

Fixes: http://tracker.ceph.com/issues/22099
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-volume util.arg_validators require superuser privs to check paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan skip directories to scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.simple add activate CLI unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices add help menu unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume systemd describe the interaction with 'simple' activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume index initial description of simple

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm add back a system.rst with just the lvm portion of it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume make systemd.rst generic enough to be related to other sub-commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume move lvm/systemd.rst to top level, making it generic

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume simple.activate initial take on documenting activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm.activate update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm.systemd update anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume document the scan functionality

Signed-off-by: Alfredo Deza <adeza@redhat.com>

tool: use coarse_mono_clock in ceph-kvstore-tool

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

tool: ceph-kvstore-tool's store-crc command did not save result to the file as expected

"store-crc <path>" command did not save result to <path>, it didn't use <path>
argument and alwayls passed a NULL to "st.traverse" function. now we will create file
<path>, pass a *ofstream to "st.traverse" function.

Fiexes: http://tracker.ceph.com/issues/22092

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

doc/release-notes: clarify that you need to keep your existing OSD caps

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: monmap log on active should be debug

This isn't even a change to the map, just something
that got echoed every time a new mon was the leader.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18793 from idryomov/wip-qa-cram-path

qa/tasks/cram: include /usr/sbin in the PATH for all commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18556 from joke-lee/validate_cors_rule_header_in_cors_option_request

rgw: add cors header rule check in cors option request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18795 from idryomov/wip-krbd-enable-generic-050-448

qa/suites/krbd: enable generic/050 and generic/448

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18706 from cbodley/wip-rgw-policy-emplace

rgw: simplify use of map::emplace in iam

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #18770 from tchaikov/wip-doc-log-file

doc: fix desc of option "mon cluster log file"

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: cleanup repeated assignment to need_zone_update in zone modify command

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

doc: fix desc of option "mon cluster log file"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: ignore osd with 0 kb capacity

Fixes: http://tracker.ceph.com/issues/21904
Signed-off-by: Ilja Slepnev <islepnev@gmail.com>

Merge PR #18807 into master

* refs/pull/18807/head:
ptl-tool: put common use-cases up front in README
ptl-tool: only create tag if new branch
ptl-tool: hint how to skip contributor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: put common use-cases up front in README

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: only create tag if new branch

It doesn't make sense to create tags for branches that already exist.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: hint how to skip contributor

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: address warnings due to incorrect format code

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #18709 from ivancich/wip-bktidx-race-bugfix

rgw: fix BZ 1500904, stale bucket index entry remains after obj delete

qa/suites/krbd: enable generic/050 and generic/448

generic/448 has been failing on ext4 and is now fixed upstream.
The fix for generic/050 is in testing, scheduled for 4.15-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #18476 from majianpeng/objecter-misc-cleanup

osdc/Objecter: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18768 from wumingqiao/wip-WaitActingChange

PG: fix name of WaitActingChange

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18789 from jcsp/wip-doc-config-set

doc: replace injectargs usage with "config set"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18769 from yaozongyou/wip-dashboard-format-label

mgr/dashboard: format tooltip's label as user friendly string

Reviewed-by: John Spray <john.spray@redhat.com>

doc: replace injectargs usage with "config set"

Cleaner and easier.  Also implicitly documents
the config set command, which hadn't been explicitly
called out in the docs before.

Signed-off-by: John Spray <john.spray@redhat.com>