doc/rados: rewrite mclock docs (2 of 2)

This is my second attempt to rewrite the
second half of the mclock docs. The first attempt
is enshrined in https://github.com/ceph/ceph/pull/40571,
in which I got cute with git and got burned.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40510 from aclamk/wip-bluestore-sharding-rst

doc: Add BlueStore sharding documentation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #40486 into master

* refs/pull/40486/head:
mds: trim cache regularly for standby-replay
mds: remove extra heap release

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #40520 into master

* refs/pull/40520/head:
mds/scrub: background scrub error fixes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40633 into master

* refs/pull/40633/head:
mds: ensure export_pin rank < max_mds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40638 into master

* refs/pull/40638/head:
mds: do not show the default auth if it's unambiguous
mds: switch to rank number instead

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40617 from tchaikov/wip-system-pmem

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40467 into master

* refs/pull/40467/head:
doc: detail `fs snapshot mirror daemon status` mgr command
doc: s/<fs>/<fs_name>/g

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #40654 from rzarzynski/wip-crimson-notify-lifetime

crimson/osd: fix the lifetime of Notify during timeouts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40641 from idryomov/wip-require-ceph-common-for-ioc

packaging: require ceph-common for immutable object cache daemon

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: improve debugs around Watch / Notify.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix the lifetime of Notify during timeouts.

This fixes a segfault during `LibRadosWatchNotify.WatchNotify2Timeout`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: detail `fs snapshot mirror daemon status` mgr command

@leseb noticed that this interface did not have any documentation. Thanks!

Fixes: http://tracker.ceph.com/issues/50229
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: s/<fs>/<fs_name>/g

Signed-off-by: Venky Shankar <vshankar@redhat.com>

run-make-check: enable WITH_RBD_RWL when WITH_PMEM is true

so we can at least build-test the rwl cache plugin in "make check" run.

this is an opt-in option which is only enabled if WITH_PMEM is "true",
we will set it in the corresponding ceph-build job.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install libpmem libraries if WITH_PMEM is set

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Add BlueStore sharding documentation

Provide sharding explanation, configuration parameters explanation.
Provide resharding tool example into Pacific format.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Co-authored-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40596 from tchaikov/wip-cmake-clean

cmake: remove created directory when "clean" target is run

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #40342 from batrick/snap_schedule-qa-fixes

qa: snap_schedule test debugging

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40625 from TRYTOBE8TME/wip-rgw-notifications

rgw/test: Changing force-branch to master

Merge pull request #40622 from batrick/dev-make

doc: update some docs still referring to make

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40647 from rzarzynski/wip-crimson-notify-send_comp-broken-debug

crimson/osd: fix broken debug in Notify::send_completion().

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #40204 into master

* refs/pull/40204/head:
client: cleanup _preadv_pwritev_locked()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40427 into master

* refs/pull/40427/head:
mds: skip the buffer in UnknownPayload::decode()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40611 into master

* refs/pull/40611/head:
qa: retry cloning the ceph.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40636 from tchaikov/wip-50196

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mds: ensure export_pin rank < max_mds

Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge pull request #40632 from adamemerson/wip-warnless-4now

Warning Cleanup and Clang Compile Fix

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

rgw/test: Changing force-branch to master

This PR includes 2 things:
1. Changing force-branch to master and removing the git-remote. This change was forgetten for PR #39139.
2. Proper cleanup/removal after completion of commands more precisely removing the kafka logs directory.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

test/osd: Change const copy in loop to const reference

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix brace-enclosed-scalar warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix warnings from s3select

Since this is a submodule, just wrap the include in a pragma.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove unused variables

(Also mark the file as C++ rather than C)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Fix ambiguous call to addr.parse

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix signed comparison warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/librados: Disambiguate function call

Cleaning up a warning.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Add missing extension to file in sources list

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38737 from IlsooByun/objmeta-key-underscore

rgw: keep underscore in metatdata key

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #39749 from linuxbox2/wip-olock-fixes

rgw: objectlock: improve client error messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40263 from ivancich/wip-update-listing-comments

rgw: update bucket listing log output

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #40570 from zdover23/wip-doc-mclock-rewrite-first-half-2021-Apr-2

doc/rados: rewrite mclock-config-ref

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #40639 from yuvalif/wip-yuval-fix-50201

rgw/multisite: return correct error code when op fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: fix broken debug in Notify::send_completion().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

query the python version before trying to test diskprediction_local

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40581 from weixinwei/master

os/bluestore/BlueFS: do not _flush_range deleted files

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40601 from cbodley/wip-50147

qa/rgw: move ignore-pg-availability.yaml out of suites/rgw

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #39907 from guits/fix_get_first_lv

ceph-volume: `get_first_lv()` refactor

packaging: require ceph-common for immutable object cache daemon

This daemon has a systemd service which starts it with --setuser ceph
--setgroup ceph.  "ceph" user and group are created by ceph-common and
won't be there unless ceph-common is installed.

Fixes: https://tracker.ceph.com/issues/50207
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40624 from rhcs-dashboard/read-only-config-access-disable

mgr/dashboard: Revoke read-only user's access to Manager modules

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/selftest: add a command for querying python version

so the test driver can skip certain tests based on the version of python
runtime on the test node

Fixes: https://tracker.ceph.com/issues/50196
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/multisite: return correct error code when op fails

when trying to disable/enbale sync on non-master zone

Fixes: https://tracker.ceph.com/issues/50201
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

ceph-volume: `get_first_*()` refactor

As indicated by commit 17957d9beb42a04b8f180ccb7ba07d43179a41d3 those
fuctions were meant to avoid writing something like following:

```
lvs = get_lvs()
if len(lvs) >= 1:
lvs = lv[0]
```

Those functions should return `None` if 0 or more than 1 item is returned.
The current name of these functions are confusing and can lead to thinking that
we just want the first item returned, even though it returns more than 1
item, let's rename them to `get_single_pv()`, `get_single_vg()` and
`get_single_lv()`

Closes: https://tracker.ceph.com/issues/49643
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #40637 from ronen-fr/wip-ronenf-rgw-logbacking

rgw: fix lambda capture of a non-variable
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #40604 from liewegas/podman-registries-conf

qa/distros/podman: preserve registries.conf

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #40487 from tchaikov/wip-boost-1.75

install-deps.sh: install boost 1.75

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40583 from a16bitsysop/include

tools/cephfs_mirror/PeerReplayer.cc: add missing include

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40607 from tchaikov/wip-journald-if-msghdr

common: disable journald logging backend if struct msghdr is not found

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #40609 from tchaikov/wip-common-buffer-cleanup

common/buffer.cc: use shift_round_up() when appropriate

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mds: do not show the default auth if it's unambiguous

Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds: switch to rank number instead

Signed-off-by: Xiubo Li <xiubli@redhat.com>

rgw: fix lambda capture of a non-variable

One cannot just capture a structured binding "non-variable".

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

os/bluestore/BlueFS: do not _flush_range deleted files

Fixes: https://tracker.ceph.com/issues/49861
Signed-off-by: weixinwei <weixw3@lenovo.com>

Merge PR #40603 into master

* refs/pull/40603/head:
qa/tasks/ceph.conf: shorten cephx TTL for testing

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #40190 from cbodley/wip-qa-rgw-sigv4-warnings

rgw: silence some unused variable warnings

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #40602 into master

* refs/pull/40602/head:
qa/tasks/cephadm: add apply() method/task

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #40597 into master

* refs/pull/40597/head:
cephadm: pass '-i' to docker|podman run for shell|enter

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40553 from dang/wip-dang-zipper-list

RGW Zipper - Make sure bucket list progresses

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #39939 into master

* refs/pull/39939/head:
cephfs: ceph-dokan - properly log the mounted root
cephfs: Update ceph-dokan "--removable" flag
cephfs: document using multiple fs on Windows
cephfs: provide additional volume details on Windows
cephfs: add ceph-dokan unmap command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40418 into master

* refs/pull/40418/head:
test: unmount when finished ino_release_cb
test: wait a time for inode release
qa: move ino_release_cb to libcephfs sub-suite
qa: simplify recall triggers for bug
qa: fix name for qa task referencing tracker issue

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40460 into master

* refs/pull/40460/head:
client: only check pool permissions for regular files

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40465 into master

* refs/pull/40465/head:
test: bump up retries for `test_mirroring_init_failure_with_recovery` test
test: fix typo
test: disable mgr/mirroring for `test_mirroring_init_failure_with_recovery` test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40468 into master

* refs/pull/40468/head:
mds/metrics: add one whitespace between metric type the metainfo

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #40469 into master

* refs/pull/40469/head:
qa: check mounts attribute in ctx

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40501 into master

* refs/pull/40501/head:
client: fix the opened inodes counter increasing

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: rewrite mclock-config-ref

This PR rewrites the material provided in
PR#40531, improving its elegance and
readability.

This PR rewrites the first 48 percent of the
material provided in PR#40531.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40113 from yuvalif/wip-yuval-fix-49799

rgw/notification: set correct type to "post" and "copy" notifications

mgr/dashboard: Revoke read-only user's access to Manager modules

This will disable read only user to read/open Manager Modules page in
Ceph Dashboard where some of the security related informations are
shown.

Fixes: https://tracker.ceph.com/issues/50174
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge PR #40620 into master

* refs/pull/40620/head:
cephfs-top: fix typo in help

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40613 into master

* refs/pull/40613/head:
doc/cephfs/nfs: add user id, fs name and key to FSAL block

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: update some docs still referring to make

We use ninja now.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40616 from rhcs-dashboard/fix-50044-master

mgr/dashboard: debug nodeenv hangs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: trim cache regularly for standby-replay

This change is slightly awkward because standby-replay MDS do not do all
the kinds of upkeep a normal active MDS does. In particular, it is not
going to recall client state from clients.

This diff also merges the extra recall_client_state in
MDCache::check_memory_usage into its only caller (the upkeep thread)
where it was also doing a recall. That's just a matter of merging the
recall flags. This has the added benefit of making
MDCache::check_memory_usage callable for all MDS daemons regardless of
state.

Fixes: https://tracker.ceph.com/issues/50048
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove extra heap release

We now regularly do this unconditionally in the MDS, see the upkeep
thread.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40615 from tchaikov/wip-doc-header-css

doc/_themes: fix the styling of section header

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/_themes: fix the styling of section header and list items

in the latest document generated from RtD, the section headers are now
in <section> tags instead of <div class="section">, so update the css
accordingly.

also tweak the style of the list items in unordered list to be the same
as it was.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40587 from tchaikov/wip-addr-parse-cleanup

src: use entity_addr_t::parse(string_view) when possible

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: debug nodeenv hangs

Increase verbosity in nodeenv command for debugging purposes.

Fixes: https://tracker.ceph.com/issues/50044
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

doc/cephfs/nfs: add user id, fs name and key to FSAL block

Fixes: https://tracker.ceph.com/issues/50161
Signed-off-by: Varsha Rao <varao@redhat.com>

common/buffer.cc: use shift_round_up() when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: retry cloning the ceph.git

The cloning may take a very long time, 2.5 hours for example, usally
it should be done in server minutes. If it really very slow, there
maybe something wrong with the current socket connections, retry it
maybe fixed.

This fixing will timeout the clone command after 30m, and retry it
by doubling the timeout value.

Fixes: https://tracker.ceph.com/issues/50021
Signed-off-by: Xiubo Li <xiubli@redhat.com>

rgw/notification: set correct type to "post" and "copy" notifications

Fixes: https://tracker.ceph.com/issues/49799
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

common: disable journald logging backend if struct msghdr is not found

* cmake/modules/CephChecks.cmake: detect the existence of struct msghdr,
  define HAVE_MSGHDR if it is found
* src/common/CMakeLists.txt: do not compile journald.cc if HAVE_MSGHDR
  is FALSE. as in that case, we cannot use sendmsg() to write to
  journald unix domain socket
* src/test/CMakeLists.txt, src/test/common/CMakeLists.txt: disable test
  exercising journald logging backend if HAVE_MSGHDR is not defined
* src/common/Journald.h: define a dummy JournaldLogger and a dummy
  JournaldClusterLogger when HAVE_MSGHDR is not defined, in order to
  minimize the change in Log.h and Log.cc, otherwise the source code of
  Log.h and Log.cc would be segmented into smaller chunks by
  `ifdef HAVE_MSGHDR` macros.
* src/include/config-h.in.cmake: define a new macro named
  HAVE_MSGHDR.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephfs-top: fix typo in help

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #40582 from a16bitsysop/32bit

src/common/buffer.cc: change cast to static_cast

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40578 from tchaikov/wip-cmake-pmem

cmake: require libpmem 1.7 and cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #40572 from athanatos/sjust/wip-48613

osd/PeeringState: fix acting_set_writeable min_size check

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40475 from tchaikov/wip-qa-focal

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #40591 from tchaikov/wip-dencoder

tools/ceph-dencoder: link against libtcmalloc

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Sage Weil <sage@redhat.com>

osd/PeeringState: fix acting_set_writeable min_size check

acting.size() >= pool.info.min_size is meant to check min_size against
acting set participants, but acting is a vector with placeholders.
actingset is the representation with placeholders removed.

The upshot of this bug is that the activation process will basically
ignore min_size for an ec pool allowing writes in cases where it
shouldn't.  PastIntervals::check_new_interval, however, performs
the check correctly, and will therefore discount intervals in which
we really did serve writes as not writeable.  This can trigger many
different problem conditions including but not limited to:
  - Unfound objects due to accepting a last_update with insufficient
    osds
  - Lost writes
  - Crashes due to peering rules being violated

This bug was originally introduced with recovery below min_size in
e5a96fd, and then preserved through refactors in 749a13d and 95bec9.

7cb818a exposed it with with expansion of recovery below min_size
to include ec pools (acting.size() is sufficient for replicated
pools).

Fixes: https://tracker.ceph.com/issues/48613
Fixes: https://tracker.ceph.com/issues/48417
Signed-off-by: Samuel Just <sjust@redhat.com>