ceph-volume: fix "device" output

During
https://github.com/ceph/ceph/commit/ec0f5f3b22d24754c16131a1996e42b787e4255f
I changed "device" by "dev", let's revert that!

Fixes: https://tracker.ceph.com/issues/50478
Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #38368 from rishabh-d-dave/cv-list-by-osd-id

ceph-volume: allow listing devices by OSD ID

Merge pull request #40992 from zdover23/wip-doc-cephadm-osd-advanced-adding-a-word-2021-apr-22

doc/cephadm: adding "device" to a sentence

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #40724 from rhcs-dashboard/upgrade-caniuse-lite

mgr/dashboard: upgrade Browserslist: caniuse-lite

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41029 from zdover23/wip-doc-cephadm-nfs-service-corrections-2021-Apr-26

doc/cephadm: rewrite "nfs.rst"

Reviewed-by: Sage Weil <sweil@redhat.com>

Merge pull request #40594 from mkogan1/wip-rgw-multipart-complete

rgw: return OK on consecutive complete-multipart reqs

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm: rewrite "nfs.rst"

This PR makes a few (very small) stylistic
corrections, mostly typographical, to the text.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39929 from ceph/dependabot/npm_and_yarn/src/pybind/mgr/dashboard/frontend/elliptic-6.5.4

build(deps): bump elliptic from 6.5.3 to 6.5.4 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41015 from inspur-wyq/patch-doc

doc/radosgw/role.rst: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39605 from inspur-wyq/patch-1

doc/radosgw/cloud-sync-module: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/radosgw/role.rst: fix typos

Signed-off-by: wangyunqing <wangyunqing@inspur.com>

Merge pull request #41011 from tchaikov/wip-cephadm-test

mgr/cephadm: s/_hosts_with_daemon_inventory/_schedulable_hosts/

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

mgr/cephadm: s/_hosts_with_daemon_inventory/_schedulable_hosts/

`_hosts_with_daemon_inventory` was renamed to `_schedulable_hosts`
in f260c48f723641a5f82c64c0e31451199aa191f8 which was merged before
eebb842d0487660c93baf9eafda28a2f87e482f3 got merged, hence the
regression.

this change just updates the test with the new method name.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34109 from CongMinYin/optimizing-bluestore-lock

os/bluestore: Optimizing the lock of bluestore writing process

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40863 from adk3798/maint-fix

mgr/cephadm: don't remove daemons on hosts in maintenance or offline mode

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #38641 from agayev/zoned-cleaner

os/bluestore: Some more plumbing for zone cleaning (WIP)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #40978 from idryomov/wip-reset-authenticate-err

mon/MonClient: reset authenticate_err in _reopen_session()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41006 from yuriw/wip-yuriw-simlinks-master

qa/tests: changed simlink to upgrade/parallel only

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40994 from neha-ojha/wip-50388

qa/suites/upgrade/pacific-x/rgw-multisite: fix install version

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tests: changed simlink to upgrade/parallel only

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #40986 from rhcs-dashboard/50484-update-run-api-tests

mgr/dashboard: set required env. variables in run-backend-api-tests.sh

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #40664 from xxhdx1985126/wip-seastar-missing-apis

crimson/os/seastore: add set/get attrs and write/read meta methods for seastore

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40991 from rzarzynski/wip-crimson-backtrace-si_addr

crimson/common: print the address that caused the fault on SIGSEGV.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: limit the max size of objects' ns and oid

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: add set/get attrs test

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: add write/read_meta tests

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add write/read_meta methods for seastore

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

qa/suites/upgrade/pacific-x/rgw-multisite: fix install version

Before:

2021-04-22T06:20:58.260 INFO:teuthology.task.install:Ceph rpm upgrade from
16.2.1-103.gb54cefd0.el8 to 16.2.1-103.gb54cefd0

After:

2021-04-22T21:08:16.565 INFO:teuthology.task.install:Ceph rpm upgrade from
16.2.1-105.g540c2494.el8 to 17.0.0-3515.ga5b16446

Fixes: https://tracker.ceph.com/issues/50388
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/cephadm: adding "device" to a sentence

This tiny PR fixes an issue that I thought
I had squashed into PR#40914: a missing
word, "device", which was caught by Josh
Durgin during review.

Alas. I had not then caught it.

Here then it is, corrected at last.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: return OK on consecutive complete-multipart reqs

Fixes: https://tracker.ceph.com/issues/50141
Signed-off-by: Mark Kogan <mkogan@redhat.com>
fixup! rgw: return OK on consecutive complete-multipart reqs

crimson/common: print the address that caused the fault on SIGSEGV.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #40526 into master

* refs/pull/40526/head:
spec: add nfs to spec file
mgr/nfs: Don't enable nfs module by default
mgr/nfs: check for invalid chars in cluster id
mgr/nfs: Use CLICommand wrapper
mgr/nfs: reorg nfs files
mgr/nfs: Check if transport or protocol are list instance
mgr/nfs: reorg cluster class and common helper methods
mgr/nfs: move common export helper methods to ExportMgr class
mgr/nfs: move validate methods into new ValidateExport class
mgr/nfs: add custom exception module
mgr/nfs: create new module for export utils
mgr/nfs: rename fs dir to export
mgr/volumes/nfs: Move nfs code out of volumes plugin

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mon/MonClient: reset authenticate_err in _reopen_session()

Otherwise, if "mon host" list has at least one unqualified IP address
without a port and both msgr1 and msgr2 are turned on, there is a race
affecting MonClient::authenticate().

For backwards compatibility reasons such an address is expanded into
two entries, each being treated as a separate monitor.  For example,
"mon host = 1.2.3.4" generates the following initial monmap:

  0: v1:1.2.3.4:6789/0
  1: v2:1.2.3.4:3300/0

See MonMap::_add_ambiguous_addr() for details.

Then, the following can happen:

1. we connect to both endpoints and attempt to authenticate
2. authenticate() sets authenticate_err to 1 and sleeps on auth_cond
3. msgr1 authenticates first (i.e. it gets the final MAuth message
   before msgr2 gets the monmap)
4. active_con is set to msgr1 connection, msgr2 connection is closed
   as redundant
5. _finish_auth() sets authenticate_err to 0 and signals auth_cond,
   but before either the monmap is received or authenticate() wakes
   up, msgr1 connection is closed due to a network hiccup
6. ms_handle_reset() calls _reopen_session() which clears active_con
   and again connects to both endpoints and attempts to authenticate
7. authenticate() wakes up, sees that there is no active_con and goes
   back to sleep, but this time with authenticate_err == 0
8. msgr2 authenticates first but doesn't call _finish_auth() because
   it is called only if authenticate_err == 1
9. active_con is set to msgr2 connection, msgr1 connection is closed
   as redundant
10. authenticate() hangs on auth_cond until timeout defaulting to 5
    minutes

The discrepancy between msgr1 and msgr2 plays a key role.  For msgr1,
authentication is considered to be complete as soon as the final MAuth
message is received -- the monmap is not waited for.  For msgr2,
authentication is considered to be complete only after the monmap is
received.

Avoid the race by setting authenticate_err to 1 in _reopen_session(),
so that _finish_auth() is called on/after every authentication attempt
instead of just the first one.

Fixes: https://tracker.ceph.com/issues/50477
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson/common: fix the messed labels for backtrace printing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: rename print_with_backtrace in fatal_signal.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #40940 from rhcs-dashboard/50440-fix-nfs-duplicated-rows

mgr/dashboard: fix duplicated rows when creating NFS export.

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

crimson/common: drop a non-existing method from the FatalSignal's declaration.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: set required env. variables in run-backend-api-tests.sh

Fixes: https://tracker.ceph.com/issues/50484
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #40979 from leseb/raw-list-fix-dup

ceph-volume: fix raw listing when finding OSDs from different clusters

Merge PR #40913 into master

* refs/pull/40913/head:
qa/tasks/cephadm: fix ctx archive check for teuthology

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40912 from liewegas/fix-bug-49237

msgr/async: fix unsafe access in unregister_conn()

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40916 from trociny/wip-50395

os/FileStore: don't propagate split/merge error to "create"/"remove"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40900 from myoungwon/wip-fix-50352

test: extend retry timeout from 150s to 300s

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #40944 from Daniel-Pivonka/cephadmhostlabelsorder

python-common: use OrderedDict instead of Set to remove duplicates from host labels list

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

mgr/dashboard: update documentation about creating NFS export.

- You learn first that orchestrator-managed clusters are detected automatically (therefore the documentation that follows is exclusively for user-defined clusters).
- Include nfs-ganesha in the security scope list.

Fixes: https://tracker.ceph.com/issues/50440
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: fix duplicated rows when creating NFS export.

- Show an error message if the same pool & namespace is used by more than 1 cluster.
- Fix error handling when no rgw daemons found.

Fixes: https://tracker.ceph.com/issues/50440
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

ceph-volume: fix raw listing when finding OSDs from different clusters

When listing OSDs on host with 2 OSDs with the same ID, the output gets
overwritten with the last listed device. So a single OSD will show up.
See the ceph-volume.log which correctly parsed both disks:

```
[2021-04-22 09:44:21,391][ceph_volume.devices.raw.list][DEBUG ] Examining /dev/sda1
[2021-04-22 09:44:21,391][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda1
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout {
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "/dev/sda1": {
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "osd_uuid": "423bf64d-f241-4f4b-a589-25a66fc836d1",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "size": 6442450944,
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "btime": "2021-04-22T09:32:55.894961+0000",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "description": "main",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_blocks": "1572864",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_blocks_per_key": "128",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_bytes_per_block": "4096",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bfm_size": "6442450944",
[2021-04-22 09:44:21,418][ceph_volume.process][INFO  ] stdout "bluefs": "1",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "ceph_fsid": "d3cd4b72-5342-4fd3-96ec-a6e581261eab",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "kv_backend": "rocksdb",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "magic": "ceph osd volume v026",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "mkfs_done": "yes",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "osd_key": "AQDGQoFg+XHqJBAAw9ZQmtrnotHCLI0Nc2to6A==",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "ready": "ready",
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout "whoami": "0"
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout }
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] stdout }
[2021-04-22 09:44:21,419][ceph_volume.devices.raw.list][DEBUG ] Examining /dev/sda2
[2021-04-22 09:44:21,419][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-bluestore-tool show-label --dev /dev/sda2
[2021-04-22 09:44:21,445][ceph_volume.process][INFO  ] stdout {
[2021-04-22 09:44:21,445][ceph_volume.process][INFO  ] stdout "/dev/sda2": {
[2021-04-22 09:44:21,445][ceph_volume.process][INFO  ] stdout "osd_uuid": "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "size": 6442450944,
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "btime": "2021-04-22T09:32:21.814768+0000",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "description": "main",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_blocks": "1572864",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_blocks_per_key": "128",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_bytes_per_block": "4096",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bfm_size": "6442450944",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "bluefs": "1",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "ceph_fsid": "69c40cb1-22af-42e4-9d59-4a4468a2f58f",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "kv_backend": "rocksdb",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "magic": "ceph osd volume v026",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "mkfs_done": "yes",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "osd_key": "AQCkQoFgre9SKBAANgHH6scIb+IiyKxh6MhY0A==",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "ready": "ready",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "require_osd_release": "16",
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout "whoami": "0"
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout }
[2021-04-22 09:44:21,446][ceph_volume.process][INFO  ] stdout }
```

However, a single OSD gets listed by `ceph-volume raw list`:

```
[root@2b5a3b8bf31c /]# ceph-volume raw list
{
    "0": {
        "ceph_fsid": "69c40cb1-22af-42e4-9d59-4a4468a2f58f",
        "device": "/dev/sda2",
        "osd_id": 0,
        "osd_uuid": "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe",
        "type": "bluestore"
    }
}
```

We now use the osd_uuid so the output will never conflict:

```
[root@2b5a3b8bf31c /]# ceph-volume raw list
{
    "423bf64d-f241-4f4b-a589-25a66fc836d1": {
        "ceph_fsid": "d3cd4b72-5342-4fd3-96ec-a6e581261eab",
        "dev": "/dev/sda1",
        "osd_id": 0,
        "osd_uuid": "423bf64d-f241-4f4b-a589-25a66fc836d1",
        "type": "bluestore"
    },
    "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe": {
        "ceph_fsid": "69c40cb1-22af-42e4-9d59-4a4468a2f58f",
        "dev": "/dev/sda2",
        "osd_id": 0,
        "osd_uuid": "c7c66bbd-7b38-4dcd-ad6d-3769c516f2fe",
        "type": "bluestore"
    }
}
```

Fixes: https://tracker.ceph.com/issues/50478
Signed-off-by: Sébastien Han <seb@redhat.com>

mon/MonClient: remove reopen_session() callback mechanism

It's been unused for over 5 years, since commit 17d24292b812 ("osd:
remove old stats backoff mechanism").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #40954 from rhcs-dashboard/password-in-dev-tools

mgr/dashboard: Remove username and password from request body

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #40909 from tchaikov/wip-osd-cast

osd/ReplicatedBackend: cast variable to unsigned before comparing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore: add get/set attr methods

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #40942 from athanatos/sjust/wip-retired-placeholder

crimson/os/seastore: track retired extents until in-progress transactions complete

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40956 from rzarzynski/wip-crimson-depends-on-binutils

ceph.spec.in: crimson depends on binutils b/c of addr2line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40970 from neha-ojha/wip-delete-some

osd: remove references to _delete_some()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40971 from tchaikov/wip-common-option-osd_erasure_code_plugins

common/options: set default variable of osd_erasure_code_plugins

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40964 from tchaikov/wip-disable-rook-mgr-module

vstart.sh: disable "rook" mgr module by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #40580 from anthonyeleven/anthonyeleven/clean-up-cephfs-standby

doc/cephfs: refine cephfs/standby

Reviewed-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore/cached_extent: fix maybe_generate_relative

The only case we really care about here is that record relative addresses
for initial_pending extents need to be turned into block relative extents.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: retain placeholders for retired, uncached extents

We need to track extents retired without first being in cache.  Create
RetiredExtentPlaceholder extent type for conflict detection on those
cases.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: speed up mount with debugging enabled

Logging allocation information really slows down initial mount.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: check cached extents for is_retired

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: ensure retired extents remain until transactions complete

Adds a structure to ensure that retired extents remain in the index until
any transactions which might reference them complete.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore_types: fix NULL_SEG_OFF declaration

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #40974 from tchaikov/wip-doc-more-conf-vals

doc/rados/configuration/mon-config-ref: use confval directive

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40725 from sseshasa/wip-mclock-internal-doc

doc/dev: Add internal documentation for mclock scheduler

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

spec: add nfs to spec file

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: Don't enable nfs module by default

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: check for invalid chars in cluster id

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: Use CLICommand wrapper

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: reorg nfs files

This patch does the following:
- removed export dir and moved its files to nfs dir
- renamed nfs.py to export.py

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: Check if transport or protocol are list instance

This is to fix "int object is not iterable” error.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: reorg cluster class and common helper methods

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: move common export helper methods to ExportMgr class

This reorganization is to support RGW exports in future PR.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: move validate methods into new ValidateExport class

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: add custom exception module

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: create new module for export utils

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: rename fs dir to export

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Move nfs code out of volumes plugin

This PR is to support integration of dashboard with volumes/nfs plugin.

Fixes: https://tracker.ceph.com/issues/50080
Signed-off-by: Varsha Rao <varao@redhat.com>

doc/_ext/ceph_confval: print out option name if it does not render

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/mon-config-ref: link by ref instead of relative hyperlink

the latter is more fragile.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/mon-config-ref: use confval directive

for defining options

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration: add missing options

and link to them when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: add description to osd_op_num_shards*

so they can be rendered

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40968 from rzarzynski/wip-crimson-only-fix-fwd-in-interruptor

crimson/common: fix forwarding in non_futurized_call_with_interruption().

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40904 from inspur-wyq/patch-3

doc/radosgw/multisite.rst: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40932 from lyarwood/pybind_rados_connect_timeout

pybind/rados: Document that timeout arg is ignored by connect

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #40938 into master

* refs/pull/40938/head:
mgr/cephadm: less noise about osd specs
mgr/cephadm: kick serve loop when adding/removing labels
mgr/cephadm: do not place osds on _no_schedule hosts
doc/cephadm: document _no_schedule label
mgr/cephadm: fix 'orch ls' count to reflect schedulable hosts
mgr/cephadm: do not schedule on _no_schedule hosts

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

common/options: set default variable of osd_erasure_code_plugins

5ddda38da4f introduced a regression which failed to substitude
osd_erasure_code_plugins's default value with the the corresponding
CMake variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: less noise about osd specs

We don't want this message every time around the serve() loop

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: kick serve loop when adding/removing labels

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not place osds on _no_schedule hosts

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm: document _no_schedule label

Signed-off-by: Sage Weil <sage@newdream.net>

osd: remove references to _delete_some()

It was replaced by do_delete_work() in b927419dd40987728684cc773e85130d29d06c39

Signed-off-by: Neha Ojha <nojha@redhat.com>

python-common: use OrderedDict instead of Set to remove duplicates from host labels list

this preserves the order of the list instead of randomizing the order

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

Merge pull request #40969 from zdover23/wip-doc-cephadm-osd-filters-colons-2021-apr-21

doc/cephadm: osd.rst -- removing colons

Reviewed-by: Sage Weil <sweil@redhat.com>

doc/cephadm: osd.rst -- removing colons

This PR removes colons that weren't necessary.
This is a companion PR to PR#40996.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40705 from orozery/libcryptsetup-return-checks

librbd/crypto: fix bad return checks from libcryptsetup

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #40966 into master

* refs/pull/40966/head:
doc/cephadm: osd: rewrite "additional opts"

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/common: fix forwarding in non_futurized_call_with_interruption().

This commit fixes a memory corruption problem found on the intersection
of `seastar::thread` and `interruptor` by a teuthology job [1].
It turns out to be caused by a use-after-free which happens due to
an issue with type deduction in `non_futurized_call_with_interruption`.
However, it might be beneficial to audit other members of the family too.

If `Result` is deduced to be a non-reference type (i.e. the `Func`
returns e.g. `ceph::bufferlist`), `std::forward<Result>` forwards
it as a r-value reference (`Result&&`). `decltype(auto)` makes this
reference the return type of the entire function which isn't what
we want.

Debugging
=========

Instrumentation
---------------
```diff
--- a/src/crimson/osd/ops_executer.cc
+++ b/src/crimson/osd/ops_executer.cc
@@ -28,6 +28,21 @@ namespace {

 namespace crimson::osd {

+struct scope_dbgr {
+  scope_dbgr() {
+    logger().debug("{}:{} on {}", __func__, __LINE__, (void*)this);
+  }
+  scope_dbgr(const scope_dbgr& src) {
+    logger().debug("{}:{} copy on {}; src {}", __func__, __LINE__, (void*)this, (void*)&src);
+  }
+  scope_dbgr(scope_dbgr&& src) noexcept {
+    logger().debug("{}:{} move on {}; src {}", __func__, __LINE__, (void*)this, (void*)&src);
+  }
+  ~scope_dbgr() {
+    logger().debug("{}:{} on {}", __func__, __LINE__, (void*)this);
+  }
+};
+
 OpsExecuter::call_ierrorator::future<> OpsExecuter::do_op_call(OSDOp& osd_op)
 {
   std::string cname, mname;
@@ -84,14 +99,16 @@ OpsExecuter::call_ierrorator::future<> OpsExecuter::do_op_call(OSDOp& osd_op)
       ceph::bufferlist outdata;
       auto cls_context = reinterpret_cast<cls_method_context_t>(this);
       const auto ret = method->exec(cls_context, indata, outdata);
-      return std::make_pair(ret, std::move(outdata));
+      logger().debug("do_ops_call: outdata just after exec: {}", outdata);
+      return std::make_tuple(ret, std::move(outdata), scope_dbgr{});
     }
   ).then_interruptible(
     [this, prev_rd, prev_wr, &osd_op, flags]
     (auto outcome) -> call_errorator::future<> {
-      auto& [ret, outdata] = outcome;
+      auto& [ret, outdata, dbgr] = outcome;
       osd_op.rval = ret;
-
+      logger().debug("do_ops_call: outdata just after then_interruptible: {}",
+                     outdata);
       logger().debug("do_op_call: method returned ret={}, outdata.length()={}"
                      " while num_read={}, num_write={}",
                      ret, outdata.length(), num_read, num_write);
```

After the fix
-------------
```
DEBUG 2021-04-21 15:32:30,012 [shard 0] bluestore - get_attr
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - do_ops_call: outdata just after exec: buffer::list(len=18,
        buffer::ptr(0~18 0x6000000ef000 in raw 0x6000000ef000 len 4000 nref 1)
)
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:33 on 0x60000085feb3
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000085ff00; src 0x60000085feb3
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085feb3
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000085fed0; src 0x60000085ff00
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085ff00
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000085ff40; src 0x60000085fed0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fed0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x60000015a740; src 0x60000085ff40
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000085ff40
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018df0; src 0x60000015a740
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x60000015a740
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x6000005196d8; src 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018df0; src 0x6000005196d8
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x6000005196d8
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018db0; src 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018df0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x600000807b58; src 0x7ffc4d018db0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018db0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - scope_dbgr:39 move on 0x7ffc4d018d30; src 0x600000807b58
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - do_ops_call: outdata just after then_interruptible: buffer::list(len=18,
        buffer::ptr(0~18 0x6000000ef000 in raw 0x6000000ef000 len 4000 nref 1)
)
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x7ffc4d018d30
DEBUG 2021-04-21 15:32:30,012 [shard 0] osd - ~scope_dbgr:42 on 0x600000807b58
```

Before
------
```
DEBUG 2021-04-21 16:06:24,354 [shard 0] bluestore - get_attr
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_ops_call: outdata just after exec: buffer::list(len=18,
        buffer::ptr(0~18 0x6000000f7000 in raw 0x6000000f7000 len 4000 nref 1)
)
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:33 on 0x60000085fe84
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x60000085fed0; src 0x60000085fe84
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fe84
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fed0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x60000085fea0; src 0x60000085fed0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x60000085ff40; src 0x60000085fea0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085fea0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x6000008360b0; src 0x60000085ff40
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x60000085ff40
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3ee00; src 0x6000008360b0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x6000008360b0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x600000801a38; src 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3ee00; src 0x600000801a38
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x600000801a38
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3edc0; src 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3ee00
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x600000807bf8; src 0x7ffe48b3edc0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3edc0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - scope_dbgr:39 move on 0x7ffe48b3ed40; src 0x600000807bf8
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_ops_call: outdata just after then_interruptible: buffer::list(len=18,
        buffer:ptr(0~18 no raw)
)
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_op_call: method returned ret=0, outdata.length()=18 while num_read=1, num_write=0
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x7ffe48b3ed40
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - ~scope_dbgr:42 on 0x600000807bf8
DEBUG 2021-04-21 16:06:24,354 [shard 0] osd - do_osd_ops: osd_op(client.4131.0:4 1.e 1:70fc7f5f:test-rados-api-o06-82166-2::foo:head {call lock.get_info in=31b out=18b}
 snapc 0={} ondisk+read+known_if_redirected e6) v8 - object 1:70fc7f5f:test-rados-api-o06-82166-2::foo:head all operations successful
Reactor stalled for 521 ms on shard 0.
Backtrace:
  0x00000000017101b8
  0x00000000016d7960
  0x00000000016d7edc
  0x00000000016d810f
  /lib64/libpthread.so.0+0x0000000000012b2f
  0x00000000016b563e
  0x00000000016b801c
  0x00000000016b8504
  0x0000000000d68120
```

[1]: http://pulpito.front.sepia.ceph.com/rzarzynski-2021-04-19_12:11:51-rados-master-distro-basic-smithi/6058778/

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: osd: rewrite "additional opts"

This PR rewrites the "Additional Options"
subsection of the OSD chapter of the cephadm
guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>