ceph-dencoder: adding support for cls_rgw_lc_obj_head

After adding cls_rgw_lc_obj_head, we can using the following command
to look into lc processing related information:
rados -p default.rgw.log getomapheader lc.0 ./header.bin --namespace=lc
ceph-dencoder type cls_rgw_lc_obj_head import ./header.bin decode dump_json

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18826 from liewegas/wip-22090

qa/suite/rados: fix balancer vs firefly tunables failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18805 from liewegas/wip-22039

os/bluestore: fix SharedBlob unregistration

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18878 from liewegas/wip-perf-health

qa/suites/rados/perf: whitelist health warnings

Merge pull request #18884 from liewegas/wip-internal-safe-to-start-threads

common/config: make internal_safe_to_start_threads internal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18825 from dillaman/wip-release-notes-rbd-caps

doc/release-notes: clarify that you need to keep your existing OSD caps

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

common/config: make internal_safe_to_start_threads internal

There is no reason for this to be exposed like a normal config option
(even a legacy one) since it is an internal safety flag.

We do keep the clear_() helper there for the unit test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18404 from cbodley/wip-vstart-mrgw

rgw: mrgw.sh uses instance name 'client.rgw'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18821 from prallabh/wip-22084

rgw: Fix swift object expiry not deleting objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18846 from yaozongyou/fix-lifecycle-bug

rgw: check going_down() when lifecycle processing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/rados/perf: whitelist health warnings

We may see these as cluster stabilizes.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18877 from Jeegn-Chen/wip-issue-22103

rados: make ceph_perf_msgr_client work for multiple jobs

Reviewed-by: Haomai Wang <haomai@xsky.com>

qa/suites/rados: stop testing firefly tunables

We can't mix the balancer compat-set testing with firefly tunables because
it requires that all buckets be straw2.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: fail a bit more gracefully if we can't create a compat weight-set

Signed-off-by: Sage Weil <sage@redhat.com>

rados: make ceph_perf_msgr_client work for multiple jobs

Move the invocation of msgr->start() before a client initiates a connection.
So the nonce is probably initialized and the server side is able to differenciate
the connections from different Messengers from the same client.

Fixes: http://tracker.ceph.com/issues/22103
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18876 from smithfarm/wip-rn-typos

doc: release notes: fix grammar/style nits

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc: release notes: fix grammar/style nits

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #18862 from Jeegn-Chen/wip-issue-22100

rados: Copy payload in ceph_perf_msgr_client

Reviewed-by: Haomai Wang <haomai@xsky.com>

rados: Copy payload in ceph_perf_msgr_client

Copy payload in ceph_perf_msgr_client so that payload can keep unchanged

Fixes: http://tracker.ceph.com/issues/22100
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>

Merge pull request #18707 from liewegas/wip-filestore-rocksdb-compression

os/filestore: disable rocksdb compression

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #18823 from ceph/wip-bz1498200

ceph-volume: support GPT and other deployed OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

os/bluestore: fix SharedBlob unregistration

We use the SharedBlobSet remove() in three cases:

- from SharedBlob::put(), we try to remove ourselves from the set, but
  have to deal with a racing lookup, so the removal is conditional on
  nref still being 0.
- from split_cache(), we move the SharedBlob to another collection
- from make_blob_unshared(), we remove the entry when we clear the sbid.

The problem is that the condtiional remove() (for the first case) was being
used for all three cases, and in the second two cases nref is always != 0,
so it doesn't actually happen.  This can lead to a crash during cache
shutdown.

Fix by making two variants: remove() that is unconditional, and
try_remove() that is conditional.

Set the sb->coll pointer after because remove() asserts the parent matches
where we are unregistering.

Fixes: http://tracker.ceph.com/issues/22039
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18817 from ZVampirEM77/wip-em-zone-cleanup

rgw cleanup: some unnecessary function called and repeated assignment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18683 from tchaikov/wip-erasure-cleanup

erasure-code: refactor the interfaces to hide internals from public

Reviewed-by: Sage Weil <sage@redhat.com>

doc/ceph-volume include the top level systemd.rst in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate add --file in help docstring as an example

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume activate has a --file option for json files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate enable the osd when activating

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl always force symlink when masking ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate use the contents of the JSON file, fallback to CLI args

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate only check for missing id and uuid if file is not passed in

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate allow to pass a file to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util add tests for OSDPath validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking ceph-disk needs to be done by directly symlinking to dev/null

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl masking needs to use systemctl mask, not disable

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan if mounted device cannot be detected, raise an error

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate remove str formatting from logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators always convert the path to an absolute path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: Fix swift object expiry not deleting objects

In cls_timeindex_list() though `to_index` has expired for a timespan, the marker is set for a subsequent index during the time boundary check.
This marker is further returned to RGWObjectExpirer::process_single_shard(), where this out_marker is trimmed from the respective shard,
resulting in a lost removal hint and a leaked object.

Fixes: http://tracker.ceph.com/issues/22084
Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

Merge pull request #18781 from iliul/correct-error-messages

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18796 from ivancich/wip-fix-fmt-warnings

rgw: address warnings due to incorrect format code

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17659 from majianpeng/osd-wait-ratherthan-waitinterval

osd/OSD: Using Wait rather than WaitInterval to wait queue.is_empty().

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18759 from tchaikov/wip-mon-log-last

mon/LogMonitor: "log last" should return up to n entries

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18779 from tchaikov/wip-osd-remove-unused-objects_read_async

osd: remove unused ReplicatedBackend::objects_read_async()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Xinze Chi <xinze@xsky.com>

Merge pull request #18809 from tchaikov/wip-21904

mgr/zabbix: ignore osd with 0 kb capacity

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>

rgw/rgw_op.cc: Fix error message in rgw_user_get_all_buckets_stats

Signed-off-by: iliul <liul.stone@gmail.com>

rgw: check going_down() when lifecycle processing

Fixes: http://tracker.ceph.com/issues/22099
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-volume util.arg_validators require superuser privs to check paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.scan skip directories to scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.simple add activate CLI unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices add help menu unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume systemd describe the interaction with 'simple' activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume index initial description of simple

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm add back a system.rst with just the lvm portion of it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume make systemd.rst generic enough to be related to other sub-commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume move lvm/systemd.rst to top level, making it generic

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume simple.activate initial take on documenting activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm.activate update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm.systemd update anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume document the scan functionality

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/release-notes: clarify that you need to keep your existing OSD caps

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18793 from idryomov/wip-qa-cram-path

qa/tasks/cram: include /usr/sbin in the PATH for all commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18556 from joke-lee/validate_cors_rule_header_in_cors_option_request

rgw: add cors header rule check in cors option request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18795 from idryomov/wip-krbd-enable-generic-050-448

qa/suites/krbd: enable generic/050 and generic/448

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18706 from cbodley/wip-rgw-policy-emplace

rgw: simplify use of map::emplace in iam

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #18770 from tchaikov/wip-doc-log-file

doc: fix desc of option "mon cluster log file"

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: cleanup repeated assignment to need_zone_update in zone modify command

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

doc: fix desc of option "mon cluster log file"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: ignore osd with 0 kb capacity

Fixes: http://tracker.ceph.com/issues/21904
Signed-off-by: Ilja Slepnev <islepnev@gmail.com>

Merge PR #18807 into master

* refs/pull/18807/head:
ptl-tool: put common use-cases up front in README
ptl-tool: only create tag if new branch
ptl-tool: hint how to skip contributor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: put common use-cases up front in README

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: only create tag if new branch

It doesn't make sense to create tags for branches that already exist.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ptl-tool: hint how to skip contributor

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: address warnings due to incorrect format code

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #18709 from ivancich/wip-bktidx-race-bugfix

rgw: fix BZ 1500904, stale bucket index entry remains after obj delete

qa/suites/krbd: enable generic/050 and generic/448

generic/448 has been failing on ext4 and is now fixed upstream.
The fix for generic/050 is in testing, scheduled for 4.15-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #18476 from majianpeng/objecter-misc-cleanup

osdc/Objecter: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18768 from wumingqiao/wip-WaitActingChange

PG: fix name of WaitActingChange

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18789 from jcsp/wip-doc-config-set

doc: replace injectargs usage with "config set"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18769 from yaozongyou/wip-dashboard-format-label

mgr/dashboard: format tooltip's label as user friendly string

Reviewed-by: John Spray <john.spray@redhat.com>

doc: replace injectargs usage with "config set"

Cleaner and easier.  Also implicitly documents
the config set command, which hadn't been explicitly
called out in the docs before.

Signed-off-by: John Spray <john.spray@redhat.com>

osd: use is_erasure() when appropriate

sometimes, we are using pool.info.require_rollback() even when what we
mean is "pool.info.is_erasure()", so fix it to improve the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: remove unused ReplicatedBackend::objects_read_async()

async read is not used by replica pool at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #18689 into master

* refs/pull/18689/head:
cephfs: Do not check auth gid when not specified

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18386 into master

* refs/pull/18386/head:
ptl-tool: allow setting project/repo for GH req

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18658 from linuxbox2/wip-s3-post-20201

RGW: update S3 POST policy handling of Content-Type

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc/ceph-volume update the index to include the simple sub command and files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/tasks/cram: include /usr/sbin in the PATH for all commands

/usr/sbin is not in non-login PATH on centos.  We already do this for
workunits, see commit 0e53f5f38b0c ("workunit: include /usr/sbin in the
PATH for all commands").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-volume tests.simple add tests for trigger parsing from systemd

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.simple create etc_path scan tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.simple add checks for get_contents in scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util add binary verification tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a fixture for temporary files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd.systemctl helpers to mask ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume devices include simple at the module import level

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume main add simple to the sub-commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add an OSDPath argument validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.system add a helper to check if a file is a binary or not

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system add a context manager for temporary mounting/unmounting

Signed-off-by: Alfredo Deza <adeza@redhat.com>