ceph-object-corpus: pick up 16.2.0-90-g50f1821b4c

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40635 from pcuzner/prometheus_add_pool_metadata

mgr/prometheus:Improve the pool metadata

Merge pull request #39944 from yuvalif/wip-yuval-fix-49650

rgw/notifications: delete bucket notification object when empty

Merge pull request #40786 from tchaikov/wip-script-bit

build-integration-branch: retry when running into network failures

Reviewed-by: Sage Weil <sage@redhat.com>

build-integration-branch: retry when running into network failures

Signed-off-by: Kefu Chai <kchai@redhat.com>

build-integration-branch: define variables for pr_number and friends

so they can be reused later.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #40577 into master

* refs/pull/40577/head:
cephadm: normalize unqualified repo digests to docker.io
mgr/cephadm/upgrade: normalize unqualified target image

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge PR #40537 into master

* refs/pull/40537/head:
cephadm:persist the grafana.db file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #40653 into master

* refs/pull/40653/head:
mon: check mdsmap is resizeable before promoting standby-replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40642 into master

* refs/pull/40642/head:
client: don't allow access to MDS-private inodes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40481 into master

* refs/pull/40481/head:
qa: test standby-replay with fs:workloads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40431 into master

* refs/pull/40431/head:
qa/cephfs: remove create_keyring_file from cephfs_test_case.py
qa/cephfs: don't use sudo to write files in /tmp

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40389 into master

* refs/pull/40389/head:
mds: reject lookup ino requests for mds dirs
test: add test for invalid lookup of mdsdir

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #40723 from zdover23/wip-doc-second-attempt-mclock-rewrite-second-half-2021-Apr-09

doc/rados: rewrite mclock docs (2 of 2)

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40738 from jdurgin/wip-librados-docs

include/librados: fix doxygen syntax for docs build

Reviewed-by: Neha Ojha <nojha@redhat.com>

include/librados: fix doxygen syntax for docs build

The docs build is now warning about these like:

WARNING: Unparseable C cross-reference: '[in]'
Invalid C declaration: Expected identifier in nested name.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

cephadm: normalize unqualified repo digests to docker.io

A RepoDigests returned by docker|podman image inspect can either include
the docker.io/ prefix or not.  For reasons that aren't entirely clear,
this may vary between hosts in a cluster.  However, ceph/ceph@sha256:abc...
is the same thing as docker.io/ceph/ceph@sha256:abc..., and should be
treated as such.  Otherwise, upgrade can get into a loop where it pulls
the image on a new host, finds the other variant of the repodigests,
sees no overlap, updates target_digests, and restarts.  (It will then
find the first variant again on the first host and loop.)

Avoid this by normalizing any docker.io digests by always including the
docker.io/ prefix.

Note that it is technically possible that this assumption is wrong: it
may be that the image that already exists on the local host is from a
different registry in registries.conf's unqualified-search-registries.
However, we don't know which, since this is a search list.  In practice,
it should be exceeding rare that an image that *we* are installing using
a fully-qualified image name will end up having an unqualified repodigest
in the local registry.  Hopefully!

Fixes: https://tracker.ceph.com/issues/50114
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: normalize unqualified target image

If we get an unqualified target image, assume it's docker.io.  This
ensures that we're passing a fully-qualified target to docker|podman on
the various hosts and don't end up with something different based on the
per-host search path for unqualified image names.

Signed-off-by: Sage Weil <sage@newdream.net>

doc/rados: rewrite mclock docs (2 of 2)

This is my second attempt to rewrite the
second half of the mclock docs. The first attempt
is enshrined in https://github.com/ceph/ceph/pull/40571,
in which I got cute with git and got burned.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39216 from myoungwon/wip-manifest-dedup-test

osd, test: reworks for manifest dedup test cases

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40623 from ronen-fr/wip-ronenf-revert-40077

osd: Revert "osd: Try other PGs when reservation failures occur"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40606 from myoungwon/wip-49427-2

osd: recover unreadable snapshot before reading refcount info

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40510 from aclamk/wip-bluestore-sharding-rst

doc: Add BlueStore sharding documentation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: reject lookup ino requests for mds dirs

Fixes: https://tracker.ceph.com/issues/49922
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test standby-replay with fs:workloads

Fixes: https://tracker.ceph.com/issues/50045
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40486 into master

* refs/pull/40486/head:
mds: trim cache regularly for standby-replay
mds: remove extra heap release

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #40520 into master

* refs/pull/40520/head:
mds/scrub: background scrub error fixes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40633 into master

* refs/pull/40633/head:
mds: ensure export_pin rank < max_mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40638 into master

* refs/pull/40638/head:
mds: do not show the default auth if it's unambiguous
mds: switch to rank number instead

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon: check mdsmap is resizeable before promoting standby-replay

If any MDS is up:creating, some rank data structures may not exist yet.

Fixes: https://tracker.ceph.com/issues/50215
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40617 from tchaikov/wip-system-pmem

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40467 into master

* refs/pull/40467/head:
doc: detail `fs snapshot mirror daemon status` mgr command
doc: s/<fs>/<fs_name>/g

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #40654 from rzarzynski/wip-crimson-notify-lifetime

crimson/osd: fix the lifetime of Notify during timeouts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40641 from idryomov/wip-require-ceph-common-for-ioc

packaging: require ceph-common for immutable object cache daemon

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: improve debugs around Watch / Notify.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix the lifetime of Notify during timeouts.

This fixes a segfault during `LibRadosWatchNotify.WatchNotify2Timeout`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: detail `fs snapshot mirror daemon status` mgr command

@leseb noticed that this interface did not have any documentation. Thanks!

Fixes: http://tracker.ceph.com/issues/50229
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: s/<fs>/<fs_name>/g

Signed-off-by: Venky Shankar <vshankar@redhat.com>

run-make-check: enable WITH_RBD_RWL when WITH_PMEM is true

so we can at least build-test the rwl cache plugin in "make check" run.

this is an opt-in option which is only enabled if WITH_PMEM is "true",
we will set it in the corresponding ceph-build job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: do not assert() in the case of no obc

Upon rollback, we should handle ENOENT case,
so what we should do here is to return NOOP.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: remove unnecessary ref handling in _delete_oid

Let's consider the following case when handling a delete op.
1. Delete --> whiteouted
2. Make clone

In this case, current code clears chunk_map and calls dec_all_manifest_refcount()
in _delete_oid() even if the clone still has the references.

To fix this, This commit remove unnecessary ref handling in _delete_oid, and
makes finish_ctx() to handle ref handling, which can aware of whether the
clone is created or not.

Also, remove oi.size == 0 condition in finish_ctx() to handle ref. counting
upon a delete op with whitedouted clone.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc: Add BlueStore sharding documentation

Provide sharding explanation, configuration parameters explanation.
Provide resharding tool example into Pacific format.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Co-authored-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40596 from tchaikov/wip-cmake-clean

cmake: remove created directory when "clean" target is run

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/cephfs: remove create_keyring_file from cephfs_test_case.py

Since teuthology.orchestra.remote.mktemp() can write a temporary file
and not just create it, create_keyring_file() is now redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: don't use sudo to write files in /tmp

Files in /tmp cannot be written by any user( including the root user)
other than the file owner even if the permission mode on the file is
777.

Fixes: https://tracker.ceph.com/issues/49466
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #40342 from batrick/snap_schedule-qa-fixes

qa: snap_schedule test debugging

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40625 from TRYTOBE8TME/wip-rgw-notifications

rgw/test: Changing force-branch to master

Merge pull request #40622 from batrick/dev-make

doc: update some docs still referring to make

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40647 from rzarzynski/wip-crimson-notify-send_comp-broken-debug

crimson/osd: fix broken debug in Notify::send_completion().

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #40204 into master

* refs/pull/40204/head:
client: cleanup _preadv_pwritev_locked()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40427 into master

* refs/pull/40427/head:
mds: skip the buffer in UnknownPayload::decode()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40611 into master

* refs/pull/40611/head:
qa: retry cloning the ceph.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40636 from tchaikov/wip-50196

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds: ensure export_pin rank < max_mds

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

osd: recover unreadable snapshot before reading ref. count info

Manifest objects needs adjacent clones when incrementing/decrementing
refcount. This commit makes the current code to call get_manifest_ref_count
before reading ref. count info.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

mgr/prometheus:Improve the pool metadata

Adds percent_used and used_bytes to the per pool
metrics group, and add additional labels to the ceph_pool_metadata
metric
- compression_mode - so consumers can tell whether compression
  is active
- description - provide a text string showing the protection
  scheme of the pool (e.g. replica2, or ec:4+2
- type - text string showing replicated or erasure enabling
  easy filtering across the pools

These additional fields allow compression savings to be more
easily shown. The inclusion of percent_used ensures that any
prometheus based view of pool usage will ALWAYS match the CLI.

Fixes: https://tracker.ceph.com/issues/49049
Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #40632 from adamemerson/wip-warnless-4now

Warning Cleanup and Clang Compile Fix

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

rgw/test: Changing force-branch to master

This PR includes 2 things:
1. Changing force-branch to master and removing the git-remote. This change was forgetten for PR #39139.
2. Proper cleanup/removal after completion of commands more precisely removing the kafka logs directory.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

test/osd: Change const copy in loop to const reference

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix brace-enclosed-scalar warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix warnings from s3select

Since this is a submodule, just wrap the include in a pragma.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove unused variables

(Also mark the file as C++ rather than C)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Fix ambiguous call to addr.parse

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix signed comparison warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Disambiguate function call

Cleaning up a warning.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add missing extension to file in sources list

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38737 from IlsooByun/objmeta-key-underscore

rgw: keep underscore in metatdata key

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #39749 from linuxbox2/wip-olock-fixes

rgw: objectlock: improve client error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40263 from ivancich/wip-update-listing-comments

rgw: update bucket listing log output

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #40570 from zdover23/wip-doc-mclock-rewrite-first-half-2021-Apr-2

doc/rados: rewrite mclock-config-ref

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #40639 from yuvalif/wip-yuval-fix-50201

rgw/multisite: return correct error code when op fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: fix broken debug in Notify::send_completion().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

query the python version before trying to test diskprediction_local

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40581 from weixinwei/master

os/bluestore/BlueFS: do not _flush_range deleted files

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40601 from cbodley/wip-50147

qa/rgw: move ignore-pg-availability.yaml out of suites/rgw

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

client: don't allow access to MDS-private inodes

Fixes: https://tracker.ceph.com/issues/50112
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39907 from guits/fix_get_first_lv

ceph-volume: `get_first_lv()` refactor

packaging: require ceph-common for immutable object cache daemon

This daemon has a systemd service which starts it with --setuser ceph
--setgroup ceph.  "ceph" user and group are created by ceph-common and
won't be there unless ceph-common is installed.

Fixes: https://tracker.ceph.com/issues/50207
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40624 from rhcs-dashboard/read-only-config-access-disable

mgr/dashboard: Revoke read-only user's access to Manager modules

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/selftest: add a command for querying python version

so the test driver can skip certain tests based on the version of python
runtime on the test node

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/multisite: return correct error code when op fails

when trying to disable/enbale sync on non-master zone

Fixes: https://tracker.ceph.com/issues/50201
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

ceph-volume: `get_first_*()` refactor

As indicated by commit 17957d9beb42a04b8f180ccb7ba07d43179a41d3 those
fuctions were meant to avoid writing something like following:

```
lvs = get_lvs()
if len(lvs) >= 1:
lvs = lv[0]
```

Those functions should return `None` if 0 or more than 1 item is returned.
The current name of these functions are confusing and can lead to thinking that
we just want the first item returned, even though it returns more than 1
item, let's rename them to `get_single_pv()`, `get_single_vg()` and
`get_single_lv()`

Closes: https://tracker.ceph.com/issues/49643
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #40637 from ronen-fr/wip-ronenf-rgw-logbacking

rgw: fix lambda capture of a non-variable
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #40604 from liewegas/podman-registries-conf

qa/distros/podman: preserve registries.conf

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #40487 from tchaikov/wip-boost-1.75

install-deps.sh: install boost 1.75

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40583 from a16bitsysop/include

tools/cephfs_mirror/PeerReplayer.cc: add missing include

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40607 from tchaikov/wip-journald-if-msghdr

common: disable journald logging backend if struct msghdr is not found

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #40609 from tchaikov/wip-common-buffer-cleanup

common/buffer.cc: use shift_round_up() when appropriate

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mds: do not show the default auth if it's unambiguous

Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds: switch to rank number instead

Signed-off-by: Xiubo Li <xiubli@redhat.com>

rgw: fix lambda capture of a non-variable

One cannot just capture a structured binding "non-variable".

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Revert "osd: Try other PGs when reservation failures occur"

This reverts commit 08c3ede0844507074de2b43638423c094976b493.

Due to https://tracker.ceph.com/issues/49868
Should be reinstated once that bug is solved. See tracker comments for analysis
and suggested fixes.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Revert "test: Add test for scrub parallelism"

This reverts commit dd63577ab372d1d4b2da284c6e506603b293d85d.

As 08c3ede0844507074de2b43638423c094976b493 (the tested functionality) is reverted.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

os/bluestore/BlueFS: do not _flush_range deleted files

Fixes: https://tracker.ceph.com/issues/49861
Signed-off-by: weixinwei <weixw3@lenovo.com>

Merge PR #40603 into master

* refs/pull/40603/head:
qa/tasks/ceph.conf: shorten cephx TTL for testing

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #40190 from cbodley/wip-qa-rgw-sigv4-warnings

rgw: silence some unused variable warnings

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #40602 into master

* refs/pull/40602/head:
qa/tasks/cephadm: add apply() method/task

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #40597 into master

* refs/pull/40597/head:
cephadm: pass '-i' to docker|podman run for shell|enter

Reviewed-by: Adam King <adking@redhat.com>