tools: do not unload plugins during destruction.

FreeBSD ceph-dencoder crashes in the exit() calls, due to
invalid pointer references during the release process of
the loaded libraries.

Often this is signaled by libc reporting:
  __cxa_thread_call_dtors: dtr 0x47efc0 from unloaded dso, skipping

The cause for this is different behaviour between FreeBSD and Linux:
https://groups.google.com/g/bsdmailinglist/c/22ncTZAbDp4/m/Dii_pII5AwAJ
_The FreeBSD implementation here looks racy. If one thread dlcloses an
object while another thread is exiting, we can end up calling a
function at an invalid memory address. It also looks as if it may
be possible to unload one library, load another at the same address,
and end up executing entirely the wrong code, which would have some
serious security implications.

The GNU/Linux equivalent of this function locks the DSO in memory
until all references to it have gone away. A call to dlclose() on
GNU/Linux will not actually unload the library until all threads
with destructors in that library have been unloaded. I believe
that this reuses the same reference counting mechanism that
allows the same library to be dlopened and dlclosed multiple times.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #40577 into master

* refs/pull/40577/head:
cephadm: normalize unqualified repo digests to docker.io
mgr/cephadm/upgrade: normalize unqualified target image

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge PR #40537 into master

* refs/pull/40537/head:
cephadm:persist the grafana.db file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #40653 into master

* refs/pull/40653/head:
mon: check mdsmap is resizeable before promoting standby-replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40642 into master

* refs/pull/40642/head:
client: don't allow access to MDS-private inodes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40481 into master

* refs/pull/40481/head:
qa: test standby-replay with fs:workloads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40431 into master

* refs/pull/40431/head:
qa/cephfs: remove create_keyring_file from cephfs_test_case.py
qa/cephfs: don't use sudo to write files in /tmp

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40389 into master

* refs/pull/40389/head:
mds: reject lookup ino requests for mds dirs
test: add test for invalid lookup of mdsdir

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #40723 from zdover23/wip-doc-second-attempt-mclock-rewrite-second-half-2021-Apr-09

doc/rados: rewrite mclock docs (2 of 2)

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40738 from jdurgin/wip-librados-docs

include/librados: fix doxygen syntax for docs build

Reviewed-by: Neha Ojha <nojha@redhat.com>

include/librados: fix doxygen syntax for docs build

The docs build is now warning about these like:

WARNING: Unparseable C cross-reference: '[in]'
Invalid C declaration: Expected identifier in nested name.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

cephadm: normalize unqualified repo digests to docker.io

A RepoDigests returned by docker|podman image inspect can either include
the docker.io/ prefix or not.  For reasons that aren't entirely clear,
this may vary between hosts in a cluster.  However, ceph/ceph@sha256:abc...
is the same thing as docker.io/ceph/ceph@sha256:abc..., and should be
treated as such.  Otherwise, upgrade can get into a loop where it pulls
the image on a new host, finds the other variant of the repodigests,
sees no overlap, updates target_digests, and restarts.  (It will then
find the first variant again on the first host and loop.)

Avoid this by normalizing any docker.io digests by always including the
docker.io/ prefix.

Note that it is technically possible that this assumption is wrong: it
may be that the image that already exists on the local host is from a
different registry in registries.conf's unqualified-search-registries.
However, we don't know which, since this is a search list.  In practice,
it should be exceeding rare that an image that *we* are installing using
a fully-qualified image name will end up having an unqualified repodigest
in the local registry.  Hopefully!

Fixes: https://tracker.ceph.com/issues/50114
Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: normalize unqualified target image

If we get an unqualified target image, assume it's docker.io.  This
ensures that we're passing a fully-qualified target to docker|podman on
the various hosts and don't end up with something different based on the
per-host search path for unqualified image names.

Signed-off-by: Sage Weil <sage@newdream.net>

doc/rados: rewrite mclock docs (2 of 2)

This is my second attempt to rewrite the
second half of the mclock docs. The first attempt
is enshrined in https://github.com/ceph/ceph/pull/40571,
in which I got cute with git and got burned.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39216 from myoungwon/wip-manifest-dedup-test

osd, test: reworks for manifest dedup test cases

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40623 from ronen-fr/wip-ronenf-revert-40077

osd: Revert "osd: Try other PGs when reservation failures occur"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40606 from myoungwon/wip-49427-2

osd: recover unreadable snapshot before reading refcount info

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40510 from aclamk/wip-bluestore-sharding-rst

doc: Add BlueStore sharding documentation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: reject lookup ino requests for mds dirs

Fixes: https://tracker.ceph.com/issues/49922
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test standby-replay with fs:workloads

Fixes: https://tracker.ceph.com/issues/50045
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40486 into master

* refs/pull/40486/head:
mds: trim cache regularly for standby-replay
mds: remove extra heap release

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #40520 into master

* refs/pull/40520/head:
mds/scrub: background scrub error fixes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40633 into master

* refs/pull/40633/head:
mds: ensure export_pin rank < max_mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40638 into master

* refs/pull/40638/head:
mds: do not show the default auth if it's unambiguous
mds: switch to rank number instead

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon: check mdsmap is resizeable before promoting standby-replay

If any MDS is up:creating, some rank data structures may not exist yet.

Fixes: https://tracker.ceph.com/issues/50215
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40617 from tchaikov/wip-system-pmem

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40467 into master

* refs/pull/40467/head:
doc: detail `fs snapshot mirror daemon status` mgr command
doc: s/<fs>/<fs_name>/g

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #40654 from rzarzynski/wip-crimson-notify-lifetime

crimson/osd: fix the lifetime of Notify during timeouts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40641 from idryomov/wip-require-ceph-common-for-ioc

packaging: require ceph-common for immutable object cache daemon

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: improve debugs around Watch / Notify.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix the lifetime of Notify during timeouts.

This fixes a segfault during `LibRadosWatchNotify.WatchNotify2Timeout`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: detail `fs snapshot mirror daemon status` mgr command

@leseb noticed that this interface did not have any documentation. Thanks!

Fixes: http://tracker.ceph.com/issues/50229
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: s/<fs>/<fs_name>/g

Signed-off-by: Venky Shankar <vshankar@redhat.com>

run-make-check: enable WITH_RBD_RWL when WITH_PMEM is true

so we can at least build-test the rwl cache plugin in "make check" run.

this is an opt-in option which is only enabled if WITH_PMEM is "true",
we will set it in the corresponding ceph-build job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: do not assert() in the case of no obc

Upon rollback, we should handle ENOENT case,
so what we should do here is to return NOOP.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

osd: remove unnecessary ref handling in _delete_oid

Let's consider the following case when handling a delete op.
1. Delete --> whiteouted
2. Make clone

In this case, current code clears chunk_map and calls dec_all_manifest_refcount()
in _delete_oid() even if the clone still has the references.

To fix this, This commit remove unnecessary ref handling in _delete_oid, and
makes finish_ctx() to handle ref handling, which can aware of whether the
clone is created or not.

Also, remove oi.size == 0 condition in finish_ctx() to handle ref. counting
upon a delete op with whitedouted clone.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

doc: Add BlueStore sharding documentation

Provide sharding explanation, configuration parameters explanation.
Provide resharding tool example into Pacific format.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Co-authored-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40596 from tchaikov/wip-cmake-clean

cmake: remove created directory when "clean" target is run

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa/cephfs: remove create_keyring_file from cephfs_test_case.py

Since teuthology.orchestra.remote.mktemp() can write a temporary file
and not just create it, create_keyring_file() is now redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: don't use sudo to write files in /tmp

Files in /tmp cannot be written by any user( including the root user)
other than the file owner even if the permission mode on the file is
777.

Fixes: https://tracker.ceph.com/issues/49466
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #40342 from batrick/snap_schedule-qa-fixes

qa: snap_schedule test debugging

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40625 from TRYTOBE8TME/wip-rgw-notifications

rgw/test: Changing force-branch to master

Merge pull request #40622 from batrick/dev-make

doc: update some docs still referring to make

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40647 from rzarzynski/wip-crimson-notify-send_comp-broken-debug

crimson/osd: fix broken debug in Notify::send_completion().

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #40204 into master

* refs/pull/40204/head:
client: cleanup _preadv_pwritev_locked()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40427 into master

* refs/pull/40427/head:
mds: skip the buffer in UnknownPayload::decode()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40611 into master

* refs/pull/40611/head:
qa: retry cloning the ceph.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40636 from tchaikov/wip-50196

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds: ensure export_pin rank < max_mds

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

osd: recover unreadable snapshot before reading ref. count info

Manifest objects needs adjacent clones when incrementing/decrementing
refcount. This commit makes the current code to call get_manifest_ref_count
before reading ref. count info.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #40632 from adamemerson/wip-warnless-4now

Warning Cleanup and Clang Compile Fix

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

rgw/test: Changing force-branch to master

This PR includes 2 things:
1. Changing force-branch to master and removing the git-remote. This change was forgetten for PR #39139.
2. Proper cleanup/removal after completion of commands more precisely removing the kafka logs directory.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

test/osd: Change const copy in loop to const reference

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix brace-enclosed-scalar warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix warnings from s3select

Since this is a submodule, just wrap the include in a pragma.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove unused variables

(Also mark the file as C++ rather than C)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Fix ambiguous call to addr.parse

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix signed comparison warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Disambiguate function call

Cleaning up a warning.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add missing extension to file in sources list

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38737 from IlsooByun/objmeta-key-underscore

rgw: keep underscore in metatdata key

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #39749 from linuxbox2/wip-olock-fixes

rgw: objectlock: improve client error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40263 from ivancich/wip-update-listing-comments

rgw: update bucket listing log output

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #40570 from zdover23/wip-doc-mclock-rewrite-first-half-2021-Apr-2

doc/rados: rewrite mclock-config-ref

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #40639 from yuvalif/wip-yuval-fix-50201

rgw/multisite: return correct error code when op fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: fix broken debug in Notify::send_completion().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

query the python version before trying to test diskprediction_local

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40581 from weixinwei/master

os/bluestore/BlueFS: do not _flush_range deleted files

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40601 from cbodley/wip-50147

qa/rgw: move ignore-pg-availability.yaml out of suites/rgw

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

client: don't allow access to MDS-private inodes

Fixes: https://tracker.ceph.com/issues/50112
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39907 from guits/fix_get_first_lv

ceph-volume: `get_first_lv()` refactor

packaging: require ceph-common for immutable object cache daemon

This daemon has a systemd service which starts it with --setuser ceph
--setgroup ceph.  "ceph" user and group are created by ceph-common and
won't be there unless ceph-common is installed.

Fixes: https://tracker.ceph.com/issues/50207
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40624 from rhcs-dashboard/read-only-config-access-disable

mgr/dashboard: Revoke read-only user's access to Manager modules

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/selftest: add a command for querying python version

so the test driver can skip certain tests based on the version of python
runtime on the test node

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/multisite: return correct error code when op fails

when trying to disable/enbale sync on non-master zone

Fixes: https://tracker.ceph.com/issues/50201
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

ceph-volume: `get_first_*()` refactor

As indicated by commit 17957d9beb42a04b8f180ccb7ba07d43179a41d3 those
fuctions were meant to avoid writing something like following:

```
lvs = get_lvs()
if len(lvs) >= 1:
lvs = lv[0]
```

Those functions should return `None` if 0 or more than 1 item is returned.
The current name of these functions are confusing and can lead to thinking that
we just want the first item returned, even though it returns more than 1
item, let's rename them to `get_single_pv()`, `get_single_vg()` and
`get_single_lv()`

Closes: https://tracker.ceph.com/issues/49643
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #40637 from ronen-fr/wip-ronenf-rgw-logbacking

rgw: fix lambda capture of a non-variable
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #40604 from liewegas/podman-registries-conf

qa/distros/podman: preserve registries.conf

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #40487 from tchaikov/wip-boost-1.75

install-deps.sh: install boost 1.75

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40583 from a16bitsysop/include

tools/cephfs_mirror/PeerReplayer.cc: add missing include

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40607 from tchaikov/wip-journald-if-msghdr

common: disable journald logging backend if struct msghdr is not found

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #40609 from tchaikov/wip-common-buffer-cleanup

common/buffer.cc: use shift_round_up() when appropriate

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mds: do not show the default auth if it's unambiguous

Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds: switch to rank number instead

Signed-off-by: Xiubo Li <xiubli@redhat.com>

rgw: fix lambda capture of a non-variable

One cannot just capture a structured binding "non-variable".

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Revert "osd: Try other PGs when reservation failures occur"

This reverts commit 08c3ede0844507074de2b43638423c094976b493.

Due to https://tracker.ceph.com/issues/49868
Should be reinstated once that bug is solved. See tracker comments for analysis
and suggested fixes.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Revert "test: Add test for scrub parallelism"

This reverts commit dd63577ab372d1d4b2da284c6e506603b293d85d.

As 08c3ede0844507074de2b43638423c094976b493 (the tested functionality) is reverted.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

os/bluestore/BlueFS: do not _flush_range deleted files

Fixes: https://tracker.ceph.com/issues/49861
Signed-off-by: weixinwei <weixw3@lenovo.com>

Merge PR #40603 into master

* refs/pull/40603/head:
qa/tasks/ceph.conf: shorten cephx TTL for testing

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #40190 from cbodley/wip-qa-rgw-sigv4-warnings

rgw: silence some unused variable warnings

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #40602 into master

* refs/pull/40602/head:
qa/tasks/cephadm: add apply() method/task

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #40597 into master

* refs/pull/40597/head:
cephadm: pass '-i' to docker|podman run for shell|enter

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40553 from dang/wip-dang-zipper-list

RGW Zipper - Make sure bucket list progresses

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #39939 into master

* refs/pull/39939/head:
cephfs: ceph-dokan - properly log the mounted root
cephfs: Update ceph-dokan "--removable" flag
cephfs: document using multiple fs on Windows
cephfs: provide additional volume details on Windows
cephfs: add ceph-dokan unmap command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40418 into master

* refs/pull/40418/head:
test: unmount when finished ino_release_cb
test: wait a time for inode release
qa: move ino_release_cb to libcephfs sub-suite
qa: simplify recall triggers for bug
qa: fix name for qa task referencing tracker issue

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40460 into master

* refs/pull/40460/head:
client: only check pool permissions for regular files

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40465 into master

* refs/pull/40465/head:
test: bump up retries for `test_mirroring_init_failure_with_recovery` test
test: fix typo
test: disable mgr/mirroring for `test_mirroring_init_failure_with_recovery` test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40468 into master

* refs/pull/40468/head:
mds/metrics: add one whitespace between metric type the metainfo

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>