common/pick_addr: use grading machinery to refactor pick_address()

as picking iface on the same NUMA node is not a hard requirement, the
grading machinery is a nice fit for this purpose.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_ipaddr: do not write to stdout

no news is good a news.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/pick_address: prefer non-loopback addresses

instead of filtering out loopback ifaces, check for loopback addresses,
and prefer non-loopback addresses over loopback addresses.

before this change, iface named "lo" is filtered out by default,
and "lo" is allowed if `ms_bind_exclude_lo_iface` is false.

after this change, iface with address out of 127/8 is prefered.
the iface marked down is not considered.

the option of "ms_bind_exclude_lo_iface" is removed. the tests are
updated accordingly.

Fixes: https://tracker.ceph.com/issues/50456
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40664 from xxhdx1985126/wip-seastar-missing-apis

crimson/os/seastore: add set/get attrs and write/read meta methods for seastore

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40991 from rzarzynski/wip-crimson-backtrace-si_addr

crimson/common: print the address that caused the fault on SIGSEGV.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: limit the max size of objects' ns and oid

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: add set/get attrs test

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: add write/read_meta tests

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add write/read_meta methods for seastore

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: print the address that caused the fault on SIGSEGV.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #40526 into master

* refs/pull/40526/head:
spec: add nfs to spec file
mgr/nfs: Don't enable nfs module by default
mgr/nfs: check for invalid chars in cluster id
mgr/nfs: Use CLICommand wrapper
mgr/nfs: reorg nfs files
mgr/nfs: Check if transport or protocol are list instance
mgr/nfs: reorg cluster class and common helper methods
mgr/nfs: move common export helper methods to ExportMgr class
mgr/nfs: move validate methods into new ValidateExport class
mgr/nfs: add custom exception module
mgr/nfs: create new module for export utils
mgr/nfs: rename fs dir to export
mgr/volumes/nfs: Move nfs code out of volumes plugin

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/common: fix the messed labels for backtrace printing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: rename print_with_backtrace in fatal_signal.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #40940 from rhcs-dashboard/50440-fix-nfs-duplicated-rows

mgr/dashboard: fix duplicated rows when creating NFS export.

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

crimson/common: drop a non-existing method from the FatalSignal's declaration.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #40979 from leseb/raw-list-fix-dup

ceph-volume: fix raw listing when finding OSDs from different clusters

Merge PR #40913 into master

* refs/pull/40913/head:
qa/tasks/cephadm: fix ctx archive check for teuthology

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40912 from liewegas/fix-bug-49237

msgr/async: fix unsafe access in unregister_conn()

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40916 from trociny/wip-50395

os/FileStore: don't propagate split/merge error to "create"/"remove"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40900 from myoungwon/wip-fix-50352

test: extend retry timeout from 150s to 300s

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40944 from Daniel-Pivonka/cephadmhostlabelsorder

python-common: use OrderedDict instead of Set to remove duplicates from host labels list

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

mgr/dashboard: update documentation about creating NFS export.

- You learn first that orchestrator-managed clusters are detected automatically (therefore the documentation that follows is exclusively for user-defined clusters).
- Include nfs-ganesha in the security scope list.

Fixes: https://tracker.ceph.com/issues/50440
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: fix duplicated rows when creating NFS export.

- Show an error message if the same pool & namespace is used by more than 1 cluster.
- Fix error handling when no rgw daemons found.

Fixes: https://tracker.ceph.com/issues/50440
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

ceph-volume: fix raw listing when finding OSDs from different clusters

When listing OSDs on host with 2 OSDs with the same ID, the output gets
overwritten with the last listed device. So a single OSD will show up.
See the ceph-volume.log which correctly parsed both disks:

```
[2021-04-22 09:44:21,391][ceph_volume.devices.raw.list][DEBUG ] Examining /dev/sda1
[2021-04-22 09:44:21,391][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda1
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout {
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "/dev/sda1": {
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "osd_uuid": "423bf64d-f241-4f4b-a589-25a66fc836d1",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "size": 6442450944,
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "btime": "2021-04-22T09:32:55.894961+0000",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "description": "main",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_blocks": "1572864",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_blocks_per_key": "128",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_bytes_per_block": "4096",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_size": "6442450944",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bluefs": "1",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "ceph_fsid": "d3cd4b72-5342-4fd3-96ec-a6e581261eab",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "kv_backend": "rocksdb",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "magic": "ceph osd volume v026",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "mkfs_done": "yes",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "osd_key": "AQDGQoFg+XHqJBAAw9ZQmtrnotHCLI0Nc2to6A==",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "ready": "ready",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "whoami": "0"
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout }
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout }
[2021-04-22 09:44:21,419][ceph_volume.devices.raw.list][DEBUG ] Examining /dev/sda2
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda2
[2021-04-22 09:44:21,445][ceph_volume.process][INFO  ] stdout {
[2021-04-22 09:44:21,445][ceph_volume.process][INFO  ] stdout "/dev/sda2": {
[2021-04-22 09:44:21,445][ceph_volume.process][INFO  ] stdout "osd_uuid": "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "size": 6442450944,
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "btime": "2021-04-22T09:32:21.814768+0000",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "description": "main",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_blocks": "1572864",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_blocks_per_key": "128",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_bytes_per_block": "4096",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_size": "6442450944",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bluefs": "1",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "ceph_fsid": "69c40cb1-22af-42e4-9d59-4a4468a2f58f",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "kv_backend": "rocksdb",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "magic": "ceph osd volume v026",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "mkfs_done": "yes",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "osd_key": "AQCkQoFgre9SKBAANgHH6scIb+IiyKxh6MhY0A==",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "ready": "ready",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "require_osd_release": "16",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "whoami": "0"
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout }
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout }
```

However, a single OSD gets listed by `ceph-volume raw list`:

```
[root@2b5a3b8bf31c /]# ceph-volume raw list
{
    "0": {
        "ceph_fsid": "69c40cb1-22af-42e4-9d59-4a4468a2f58f",
        "device": "/dev/sda2",
        "osd_id": 0,
        "osd_uuid": "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe",
        "type": "bluestore"
    }
}
```

We now use the osd_uuid so the output will never conflict:

```
[root@2b5a3b8bf31c /]# ceph-volume raw list
{
    "423bf64d-f241-4f4b-a589-25a66fc836d1": {
        "ceph_fsid": "d3cd4b72-5342-4fd3-96ec-a6e581261eab",
        "dev": "/dev/sda1",
        "osd_id": 0,
        "osd_uuid": "423bf64d-f241-4f4b-a589-25a66fc836d1",
        "type": "bluestore"
    },
    "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe": {
        "ceph_fsid": "69c40cb1-22af-42e4-9d59-4a4468a2f58f",
        "dev": "/dev/sda2",
        "osd_id": 0,
        "osd_uuid": "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe",
        "type": "bluestore"
    }
}
```

Fixes: https://tracker.ceph.com/issues/50478
Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #40954 from rhcs-dashboard/password-in-dev-tools

mgr/dashboard: Remove username and password from request body

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #40909 from tchaikov/wip-osd-cast

osd/ReplicatedBackend: cast variable to unsigned before comparing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore: add get/set attr methods

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #40942 from athanatos/sjust/wip-retired-placeholder

crimson/os/seastore: track retired extents until in-progress transactions complete

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40956 from rzarzynski/wip-crimson-depends-on-binutils

ceph.spec.in: crimson depends on binutils b/c of addr2line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40970 from neha-ojha/wip-delete-some

osd: remove references to _delete_some()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40971 from tchaikov/wip-common-option-osd_erasure_code_plugins

common/options: set default variable of osd_erasure_code_plugins

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40964 from tchaikov/wip-disable-rook-mgr-module

vstart.sh: disable "rook" mgr module by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #40580 from anthonyeleven/anthonyeleven/clean-up-cephfs-standby

doc/cephfs: refine cephfs/standby

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore/cached_extent: fix maybe_generate_relative

The only case we really care about here is that record relative addresses
for initial_pending extents need to be turned into block relative extents.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: retain placeholders for retired, uncached extents

We need to track extents retired without first being in cache.  Create
RetiredExtentPlaceholder extent type for conflict detection on those
cases.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: speed up mount with debugging enabled

Logging allocation information really slows down initial mount.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: check cached extents for is_retired

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: ensure retired extents remain until transactions complete

Adds a structure to ensure that retired extents remain in the index until
any transactions which might reference them complete.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore_types: fix NULL_SEG_OFF declaration

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #40974 from tchaikov/wip-doc-more-conf-vals

doc/rados/configuration/mon-config-ref: use confval directive

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40725 from sseshasa/wip-mclock-internal-doc

doc/dev: Add internal documentation for mclock scheduler

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

spec: add nfs to spec file

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: Don't enable nfs module by default

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: check for invalid chars in cluster id

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: Use CLICommand wrapper

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: reorg nfs files

This patch does the following:
- removed export dir and moved its files to nfs dir
- renamed nfs.py to export.py

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: Check if transport or protocol are list instance

This is to fix "int object is not iterable” error.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: reorg cluster class and common helper methods

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: move common export helper methods to ExportMgr class

This reorganization is to support RGW exports in future PR.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: move validate methods into new ValidateExport class

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: add custom exception module

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: create new module for export utils

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: rename fs dir to export

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Move nfs code out of volumes plugin

This PR is to support integration of dashboard with volumes/nfs plugin.

Fixes: https://tracker.ceph.com/issues/50080
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/_ext/ceph_confval: print out option name if it does not render

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/mon-config-ref: link by ref instead of relative hyperlink

the latter is more fragile.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/mon-config-ref: use confval directive

for defining options

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration: add missing options

and link to them when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: add description to osd_op_num_shards*

so they can be rendered

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40968 from rzarzynski/wip-crimson-only-fix-fwd-in-interruptor

crimson/common: fix forwarding in non_futurized_call_with_interruption().

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40904 from inspur-wyq/patch-3

doc/radosgw/multisite.rst: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40932 from lyarwood/pybind_rados_connect_timeout

pybind/rados: Document that timeout arg is ignored by connect

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #40938 into master

* refs/pull/40938/head:
mgr/cephadm: less noise about osd specs
mgr/cephadm: kick serve loop when adding/removing labels
mgr/cephadm: do not place osds on _no_schedule hosts
doc/cephadm: document _no_schedule label
mgr/cephadm: fix 'orch ls' count to reflect schedulable hosts
mgr/cephadm: do not schedule on _no_schedule hosts

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

common/options: set default variable of osd_erasure_code_plugins

5ddda38da4f introduced a regression which failed to substitude
osd_erasure_code_plugins's default value with the the corresponding
CMake variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: less noise about osd specs

We don't want this message every time around the serve() loop

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: kick serve loop when adding/removing labels

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not place osds on _no_schedule hosts

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm: document _no_schedule label

Signed-off-by: Sage Weil <sage@newdream.net>

osd: remove references to _delete_some()

It was replaced by do_delete_work() in b927419dd40987728684cc773e85130d29d06c39

Signed-off-by: Neha Ojha <nojha@redhat.com>

python-common: use OrderedDict instead of Set to remove duplicates from host labels list

this preserves the order of the list instead of randomizing the order

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #40969 from zdover23/wip-doc-cephadm-osd-filters-colons-2021-apr-21

doc/cephadm: osd.rst -- removing colons

Reviewed-by: Sage Weil <sweil@redhat.com>

doc/cephadm: osd.rst -- removing colons

This PR removes colons that weren't necessary.
This is a companion PR to PR#40996.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40705 from orozery/libcryptsetup-return-checks

librbd/crypto: fix bad return checks from libcryptsetup

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #40966 into master

* refs/pull/40966/head:
doc/cephadm: osd: rewrite "additional opts"

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/common: fix forwarding in non_futurized_call_with_interruption().

This commit fixes a memory corruption problem found on the intersection
of `seastar::thread` and `interruptor` by a teuthology job [1].
It turns out to be caused by a use-after-free which happens due to
an issue with type deduction in `non_futurized_call_with_interruption`.
However, it might be beneficial to audit other members of the family too.

If `Result` is deduced to be a non-reference type (i.e. the `Func`
returns e.g. `ceph::bufferlist`), `std::forward<Result>` forwards
it as a r-value reference (`Result&&`). `decltype(auto)` makes this
reference the return type of the entire function which isn't what
we want.

Debugging
=========

Instrumentation
---------------
```diff
--- a/src/crimson/osd/ops_executer.cc
+++ b/src/crimson/osd/ops_executer.cc
@@ -28,6 +28,21 @@ namespace {

 namespace crimson::osd {

+struct scope_dbgr {
+  scope_dbgr() {
+    logger().debug("{}:{} on {}", __func__, __LINE__, (void*)this);
+  }
+  scope_dbgr(const scope_dbgr& src) {
+    logger().debug("{}:{} copy on {}; src {}", __func__, __LINE__, (void*)this, (void*)&src);
+  }
+  scope_dbgr(scope_dbgr&& src) noexcept {
+    logger().debug("{}:{} move on {}; src {}", __func__, __LINE__, (void*)this, (void*)&src);
+  }
+  ~scope_dbgr() {
+    logger().debug("{}:{} on {}", __func__, __LINE__, (void*)this);
+  }
+};
+
 OpsExecuter::call_ierrorator::future<> OpsExecuter::do_op_call(OSDOp& osd_op)
 {
   std::string cname, mname;
@@ -84,14 +99,16 @@ OpsExecuter::call_ierrorator::future<> OpsExecuter::do_op_call(OSDOp& osd_op)
       ceph::bufferlist outdata;
       auto cls_context = reinterpret_cast<cls_method_context_t>(this);
       const auto ret = method->exec(cls_context, indata, outdata);
-      return std::make_pair(ret, std::move(outdata));
+      logger().debug("do_ops_call: outdata just after exec: {}", outdata);
+      return std::make_tuple(ret, std::move(outdata), scope_dbgr{});
     }
   ).then_interruptible(
     [this, prev_rd, prev_wr, &osd_op, flags]
     (auto outcome) -> call_errorator::future<> {
-      auto& [ret, outdata] = outcome;
+      auto& [ret, outdata, dbgr] = outcome;
       osd_op.rval = ret;
-
+      logger().debug("do_ops_call: outdata just after then_interruptible: {}",
+                     outdata);
       logger().debug("do_op_call: method returned ret={}, outdata.length()={}"
                      " while num_read={}, num_write={}",
                      ret, outdata.length(), num_read, num_write);
```

After the fix
-------------
```
DEBUG 2021-04-21 15:32:30,012 [shard 0] bluestore - get_attr
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - do_ops_call: outdata just after exec: buffer::list(len=18,
        buffer::ptr(0~18 0x6000000ef000 in raw 0x6000000ef000 len 4000 nref 1)
)
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:33 on 0x60000085feb3
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000085ff00; src 0x60000085feb3
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085feb3
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000085fed0; src 0x60000085ff00
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085ff00
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000085ff40; src 0x60000085fed0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fed0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000015a740; src 0x60000085ff40
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085ff40
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018df0; src 0x60000015a740
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000015a740
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x6000005196d8; src 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018df0; src 0x6000005196d8
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x6000005196d8
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018db0; src 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x600000807b58; src 0x7ffc4d018db0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018db0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018d30; src 0x600000807b58
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - do_ops_call: outdata just after then_interruptible: buffer::list(len=18,
        buffer::ptr(0~18 0x6000000ef000 in raw 0x6000000ef000 len 4000 nref 1)
)
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018d30
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x600000807b58
```

Before
------
```
DEBUG 2021-04-21 16:06:24,354 [shard 0] bluestore - get_attr
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_ops_call: outdata just after exec: buffer::list(len=18,
        buffer::ptr(0~18 0x6000000f7000 in raw 0x6000000f7000 len 4000 nref 1)
)
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:33 on 0x60000085fe84
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x60000085fed0; src 0x60000085fe84
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fe84
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fed0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x60000085fea0; src 0x60000085fed0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x60000085ff40; src 0x60000085fea0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fea0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x6000008360b0; src 0x60000085ff40
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085ff40
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3ee00; src 0x6000008360b0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x6000008360b0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x600000801a38; src 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3ee00; src 0x600000801a38
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x600000801a38
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3edc0; src 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x600000807bf8; src 0x7ffe48b3edc0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3edc0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3ed40; src 0x600000807bf8
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_ops_call: outdata just after then_interruptible: buffer::list(len=18,
        buffer:ptr(0~18 no raw)
)
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3ed40
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x600000807bf8
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_osd_ops: osd_op(client.4131.0:4 1.e 1:70fc7f5f:test-rados-api-o06-82166-2::foo:head {call lock.get_info in=31b out=18b}
 snapc 0={} ondisk+read+known_if_redirected e6) v8 - object 1:70fc7f5f:test-rados-api-o06-82166-2::foo:head all operations successful
Reactor stalled for 521 ms on shard 0.
Backtrace:
  0x00000000017101b8
  0x00000000016d7960
  0x00000000016d7edc
  0x00000000016d810f
  /lib64/libpthread.so.0+0x0000000000012b2f
  0x00000000016b563e
  0x00000000016b801c
  0x00000000016b8504
  0x0000000000d68120
```

[1]: http://pulpito.front.sepia.ceph.com/rzarzynski-2021-04-19_12:11:51-rados-master-distro-basic-smithi/6058778/

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: osd: rewrite "additional opts"

This PR rewrites the "Additional Options"
subsection of the OSD chapter of the cephadm
guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40914 from zdover23/wip-doc-cephadm-osd-advanced-up-to-filters-2021-apr-18

doc/cephadm: rewrite "advanced osd s. specs"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

vstart.sh: disable "rook" mgr module by default

as we don't have "jsonpatch" installed for running "make check", "rook"
module always fails to load, and the error message when mgr is
unable to load is misleading and distracting. let's just disabled it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/PyModuleRegistry: use vector<> when appropriate

no need to use a set<string> for storing module name, they are
inherently unique.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev: Add internal documentation for mclock scheduler

The document details the refinements to mclock scheduler and presents the
results of a comparison study performed between the mclock scheduler and
the WPQ scheduler.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

librbd/crypto: fix bad return checks from libcryptsetup

This commit fixes the return checks for libcryptsetup functions
that may return non-zero success codes.

Fixes: https://tracker.ceph.com/issues/50461
Signed-off-by: Or Ozeri <oro@il.ibm.com>

ceph.spec.in: crimson depends on binutils b/c of addr2line.

This small utility is needed due to the backtrace processing.
See: d3dcf510c35228a525311732fa5f7385c4a736ba.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #40935 from TRYTOBE8TME/wip-rgw-bootstrap-fix

test/rgw: Removing unrecognized option from bootstrap file

mgr/dashboard: Remove username and password from request body

Fixes: https://tracker.ceph.com/issues/50451
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #40953 from tchaikov/wip-doc-confval

common/options,doc/rados/configuration: extract crimson options, use confval directive

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/conf.py: use glob.glob() to find .yaml.in files

improve the maintainability of .yaml.in files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use file(GLOB ..) to find .yaml.in files

instead of hardcode the option file names, use file(GLOB ..) to find
and collect them instead, for better maintainability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/optinos: extract crimson options out

prepare for the change to split the options by services

in future, osd will have its own osd.yaml.in, while crimson will consume
both osd.yaml.in and crimson.yaml.in.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/mon-config-ref: use confval directive

for defining options

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/general-config-ref: use confval directive

for defining options

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/osd-config-ref: use confval directive

for defining options

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40915 from rzarzynski/wip-crimson-backtrace-3

crimson: make backtraces more human-readable with boost::stacktrace and addr2line

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40939 from tchaikov/wip-doc-confval

doc/rados/configuration/osd-config-ref:  use confval directive

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

src/test: Removing unrecognized option from bootstrap file

Removing '--distribute' option from a command in bootstrap file which seemed to be the reason of failure in:
http://qa-proxy.ceph.com/teuthology/dang-2021-04-19_12:28:43-rgw-wip-dang-zipper-sysobj-distro-basic-smithi/6059513/teuthology.log

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge PR #39742 into master

* refs/pull/39742/head:
client: make Inode to inherit from RefCountedObject
client: minor cleanup to Inode class to simplify the code
client: minor cleanup to Inode related code

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40191 from dang/wip-dang-zipper-sysobj

RGW Zipper - Remove rgw_pool from API

crimson/os/seastore/cache: factor out remove_from_dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

mgr/cephadm: fix 'orch ls' count to reflect schedulable hosts

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not schedule on _no_schedule hosts

Reuse and rename existing helper to return schedulable hosts.

Signed-off-by: Sage Weil <sage@newdream.net>

doc/rados/configuration/osd-config-ref:  use confval directive

for defining options

Signed-off-by: Kefu Chai <kchai@redhat.com>