README.md: fix couple minor syntax issues

Signed-off-by: Srishti Guleria <guleriasrishti01@gmail.com>

Merge PR #41860 into master

* refs/pull/41860/head:
qa: log messages when falling back to force/lazy umount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #41941 from tchaikov/wip-crimson-errorator-loop

crimson/common: extract parallel_for_each_state out

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41949 from tchaikov/wip-crimson-prometheus

crimson/osd: expose metrics using http server

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd: expose metrics using http server

so, we can query the metrics using HTTP API, like

http://localhost:9180/metrics?name=io*

or

http://192.168.2.8:9180/metrics?name=io_queue_delay

or

http://localhost:9180/metrics

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41934 from cyx1231st/wip-seastore-onode-logs

crimson/onode-staged-tree: improve logs to understand inconsistent load from seastore

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: print NodeExtent with the header

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: validate node header when load

Add logs to detect corruptions when load nodes. assert() is not
informative enough to understand the context.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: delete copy constructor of DummyNodeExtent

Dummy backend is used for unit tests without transactions, so there
should be no copy-on-write behavior.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add trace logs when start to load nodes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41861 from AmnonHanuhov/wip-Refactor_crimson_internals

crimson/net: Complete the refactor to std::unique_ptr inside Messenger

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41921 from gregsfortytwo/wip-mon-stretch-crush-rule

mon: Sanely set the default CRUSH rule when creating pools in stretch…

Reviewed-by: Samuel Just <sjust@redhat.com>

tools/crimson: Use crimson::make_message() in perf_crimson_msgr

Instead of ceph::make_message() because conn::send() in crimson expects
a std::unique_ptr and not boost::intrusive_ptr

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41845 from agayev/zoned-revise-per-zone-naming-scheme

os/bluestore: Revise the naming scheme for per-zone cleaning informat…

Reviewed-by: Igor Fedotov <ifedotov@suse,com>

test/crimson: Use crimson::make_message() in test_alien_echo

Instead of ceph::make_message() because conn::send() in crimson expects
a std::unique_ptr and not boost::intrusive_ptr

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41830 from tchaikov/wip-ceph-argparse-cleanup

pybind/ceph_argparse: cleanups preparing for type annotations

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

crimson/net: Use MessageURef in messenger internals

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

crimson/osd: Get rid of send_to_osd() overloading

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

osd: Overload send_osd_message() in PeeringState

To allow passing MessageURef from crimson-osd and MessageRef from
ceph-osd

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

crimson/osd: Move message to send_to_osd() in ShardServices

To avoid refcounting the underlying RefCountedObject

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41923 from liewegas/fix-51234

ceph_test_librados_service: wait longer for servicemap to update

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41914 from lxbsz/wip-51092

os/memstore: make the used_bytes to atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41896 from ifed01/wip-ifed-verbose-kernel-read

blk/KernelDevice: be more verbose on read errors.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: specialize errorator<> for future<>

otherwise it always needs a return value.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: extract parallel_for_each_state out

if `parallel_for_each_state` is defined as a nested class in errorator,
clang fails to compile it:

../src/crimson/common/errorator.h:716:47: error: no class named 'parallel_for_each_state' in 'errorator<AllowedErrors...>'
    friend class errorator<AllowedErrors...>::parallel_for_each_state;
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^

and the forward declaration does not help. so we have to extract it
out of the errorator. to speed up the compilation, it is moved into
errorator-loop.h. its name mirrors `include/seastar/core/loop.h`.

we could extract the `errorator<>::parallel_for_each()` out as well,
as its return type can be deduced from the type of Iterator and Func.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41920 from ljflores/patch-1

doc: fixed a small typo in Perf Counters documentation

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41925 from tchaikov/wip-fmtlib

fmt: pickup fix of link failure with clang

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #41900 into master

* refs/pull/41900/head:
qa: use centos latest for fs:upgrade

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #41899 into master

* refs/pull/41899/head:
mon/MDSMonitor: check fscid exists for legacy case

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #41898 into master

* refs/pull/41898/head:
mon/MDSMonitor: fix whitespace in debug message

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #41892 into master

* refs/pull/41892/head:
client: remove unused include from barrier.cc

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41833 into master

* refs/pull/41833/head:
cephfs-mirror: silence warnings when connecting via mon host

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41723 into master

* refs/pull/41723/head:
mds: to print the unknow type value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #40997 into master

* refs/pull/40997/head:
test: add test to verify adding an active peer back to source
pybind/mirroring: disallow adding a active peer back to source
pybind/cephfs: interface to fetch file system id

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36823 into master

* refs/pull/36823/head:
qa : add a test for the cmd, dump cache
mds : add timeout to the command, dump cache, to prevent it from running too long and affecting the service

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fixed a small typo in Perf Counters documentation

There is a small typo in the Perf Counters documentation. Gauge was spelled incorrectly.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #40506 from p-se/pse-update-grafana-deprecated-variables

mgr/dashboard: deprecated variable usage in Grafana dashboards

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: p-se <NOT@FOUND>

Merge pull request #41808 from rhcs-dashboard/51164-show-only-days-in-bucket-details

mgr/dashboard: bucket details: show lock retention period only in days

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41758 from rhcs-dashboard/support-multiple-crush-trees

mgr/dashboard: crushmap tree doesn't display crush type other than root

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #35903 from agayev/fix-deployment-guide

doc: Add a missing instruction to manual deployment guide.

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Add a missing instruction to manual deployment guide.

Following the instructions as is results in the following error at step 15:

$ sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
global_init: error reading config file.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

ceph.spec.in: bump up the required version of fmt-devel to 6.2.1

6.2.1 is the version packaged by EPEL8, in other words, this is the
version we've been testing. so to be more consistent with the
known-to-be-good version, let's bump up the required version.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/control: add libfmt-dev for "make check"

so, on debian derivatives, we can use the libfmt-dev package for
building Ceph. this change is created in hope to reduce the compile
time.

>= 6.1.2 is specified, as it is the version packaged by ubuntu focal,
which is used for running "make check" and intergration tests.

find_package(fmt 6.0.0 QUIET)

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: pickup fix of link failure with clang

fmtlib v7.1.3 contains the fix of https://github.com/fmtlib/fmt/issues/1753

so let's bump up the submodule to the latest master HEAD of fmtlib
for more fixes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41911 from tchaikov/wip-crimson-nbd-cleanup

crimson/tools/store_nbd: better cleanup

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: replace wait_pending with seastar::gate

the inc_pending + promise<> solution is pratically identical to
seastar::gate, so let's use the prepackaged solution instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: drop unnecessary seastar::now()

the body of handle_exception() is a synchronous operation, there is no
need to return seastar::now() here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: better cleanup

* remove unix domain socket file when cleanup
  so we don't need to remove it manually after each run.
* shutdown input and output streams when cleanup
  so reactor does not watch them anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: s/socket/server_socket/

to prepare for the next commit, which will keep track of the
connected_socket as another member variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41916 from AmnonHanuhov/wip-Refactor_test_messenger

test/crimson: Use crimson's make_message in test_messenger

ceph_test_librados_service: wait longer for servicemap to update

mon thrashing may make this take a long time

Fixes: https://tracker.ceph.com/issues/51234
Signed-off-by: Sage Weil <sage@newdream.net>

mon: Sanely set the default CRUSH rule when creating pools in stretch mode

If we get a pool create request while in stretch mode that does not explicitly
specify a crush rule, look at the stretch-mode pools and their rules, and
select the most common one.

Also update set_up_stretch_mode.sh to add a few more rules that let me test
this locally.

Fixes: https://tracker.ceph.com/issues/51270
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test/crimson: Use crimson's make_message in test_messenger

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41905 from ivancich/wip-improve-cls-rgw-tracing

rgw: clean-up logging of function entering to make thorough and consistent

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

os/bluestore: Revise the naming scheme for per-zone cleaning information.

Use a single letter (G) for the namespace, and use zone_num+oid as the key.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

rgw: clean-up logging of function entering to make thorough and consistent

This provides more thorough and consistent function tracing in CLS/RGW
when logging is set to 10 or higher.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #41903 from liewegas/update-rook-client

rook-client-python: update to update-june-21

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41835 from TRYTOBE8TME/wip-rgw-keycloak-failure-fix

qa/tasks: Keycloak failure fix

Reviewed-by: Ali Maredia <amaredia@redhat.com>

mgr/rook: comment out osd creation functions

This commit comments out the OSD creation functions in rook_cluster.py
and module.py, since the submodule update has broken them.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: fix some mypy typing errors in rook_cluster.py

This commit fixes some errors caught by mypy in rook_cluster.py. Most of the
errors were caused by the update of the rook-client-python submodule in a previous
commit.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: pass zone attribute to CephObjectStore CR when creating rgw

This commit passes the zone attribute to the CephObjectStore CR when
creating a RGW instance using the Rook Orchestrator backend:
`ceph orch apply rgw <rgw-name> --realm=<realm-name> --zone=<zone-name>`

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

os/memstore: make the used_bytes to atomic

Fixes: https://tracker.ceph.com/issues/51092
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #41856 from rhcs-dashboard/maintenance-bug-fix

mgr/dashboard: Fix 500 error while exiting out of maintenance

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #41694 from jmolmo/kcli_cephadm_doc

doc: Add kcli utilization for development environments

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41848 from xxhdx1985126/wip-errorator-parallel_for_each

crimson: errorator parallel_for_each

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41894 from tchaikov/wip-crimson-sigint

crimson/{osd,store_nbd}: handle SIGINT

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41895 from ceph/wip-cacephcom

MIRRORS: Add ca.ceph.com

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix 500 error while exiting out of maintenance

When you add a host in maintenance mode and then exit the maintenance
mode, a 500 server error will popup which will interrupt the whole
exit maintenance process and leave the host in an unknown/offline state.
It happened when I was setting the status of the host through the
HostSpec(). With this change, I am using the enter_maintenance api of
the orch to enable the maintenance.

Fixes: https://tracker.ceph.com/issues/51218
Signed-off-by: Nizamudeen A <nia@redhat.com>

rook-client-python: update to update-june-21

Signed-off-by: Sage Weil <sage@newdream.net>

qa: use centos latest for fs:upgrade

Fixes: https://tracker.ceph.com/issues/51250
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MDSMonitor: check fscid exists for legacy case

If a client does not have permission to see the legacy fs, the monitor
will throw an exception when looking up the mdsmap later in the code.
We need to check existence for both code paths.

Fixes: https://tracker.ceph.com/issues/51077
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MDSMonitor: fix whitespace in debug message

So it doesn't look like:

    2021-06-16T14:37:52.953+0000 7fec41d7c700 10 mon.a@0(leader).mds e10 check_sub: is_mds=0, fscid= 1

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

blk/KernelDevice: be more verbose on read errors.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #41890 into master

* refs/pull/41890/head:
doc/cephadm: removing "Octopus" from procedure

Reviewed-by: Sage Weil <sage@redhat.com>

MIRRORS: Add ca.ceph.com

Signed-off-by: David Galloway <dgallowa@redhat.com>

crimson/osd: use stop_signal from seastar

and disable app_cfg.auto_handle_sigint_sigterm, otherwise app template
handles SIGINT and SIGTERM by itself, and calls app.stop(). but we don't
use this mechinary at all. we use seastar::defer() instead of
seastar::at_exit() for doing graceful shutdown and cleanup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: update example usage in comment

--total-device-size is not supported any more.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: add graceful shutdown support

we could have more sophisticate mechinary for interrupting fio job,
but so far it is able to stop itself if it idle by handling ctrl-C.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/seastore/nbd: destruct the store before create

Otherwise the store will register the conflicting metrics and result in
double_registration.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephadm: removing "Octopus" from procedure

This PR removes "Octopus" from the curl-based installation
procedure.

After we moved on to Pacific, referring to Octopus looks wrong.
It looks wrong because it now is wrong.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41882 from tchaikov/wip-crimson-int-safty

crimson/osd: guard non-pg-op handling with with_sequencer()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41885 from tchaikov/wip-crimson-os-cleanups

crimson/os: cleanups and reformat

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/osd: reindent

for less indent, hence better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: wrap line at 80

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: guard non-pg-op handling with with_sequencer()

because we should only ensure the ordering of the requests touching
the objects, the other requests like pgls should not be ordered along
with them. so as the second step, guard the non-pg-op handling with
with_sequencer().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use reference for loop variable

for better performance, also silences the warning like:

../src/crimson/os/seastore/random_block_manager/nvme_manager.cc:444:23: warning: loop variable ‘b’ creates a copy from type ‘const crimson::os::seastore::rbm_alloc_delta_t’ [-Wrange-loop-construct]
  444 |       for (const auto b : alloc_blocks) {
      |                       ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41859 from sebastian-philipp/mypy-constrains.txt

global,tox.ini: add mypy-constrains.txt

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

crimson/os: return seastar::now() in "finally()" block

so finally() is able to identify the return is a future, and discard it
manually.
otherwise the return value will be discarded even the future is marked
[[nodiscard]], hence the C++ compiler warns.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: remove unnecessary now()

the previous continuation in the chain already returns a future, no need
to hook up another now().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: remove unnecessary parentheses

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: fix the indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

global,tox.ini: add mypy-constrains.txt

let's avoid getting new versions of those packages by accident.
Unfortunately this means we have to manually update those
packages regurarly.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

crimson/osd: extract with_sequencer() method out

because we should only ensure the ordering of the requests touching
the objects, the other requests like pgls should not be ordered along
with them, so as the first step, let's extract with_sequencer() method
out, so we can differentiate these two cases with minimal repeatings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41881 from tchaikov/wip-crimson-alien

crimson: adapt to the new alien API

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: adapt to the new alien API

since seastar now deprecates the old alien::submit_to() API, and
replaced it with a variant which requires a new parameter of "alien"
instance, let's pick it up, to amortize the cost to catch up with
the seastar master HEAD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: add test case for errorator parallel_for_each

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson: add parallel_for_each to errorator

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #41825 from tchaikov/wip-avl-cleanup

os/bluestore/AvlAllocator: specialize _block_picker() and cleanups

Reviewed-by: Igor Fedotov <ifedotov@suse,com>

Merge pull request #41346 from adk3798/enough-mds-upgrade

mgr/cephadm: skip ok-to-stop for mds in upgrade if not enough mds daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>