RGW - Don't move attrs before setting them

Moving the attrs into s->bucket_attrs before setting them results in
setting empty attrs into the bucket.  This means that reading them back
later gets empty attrs, which can cause a segfault.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 4d38154f5779b01ed93430ef7405417820214796)

Conflicts:
src/rgw/rgw_op.cc

Merge pull request #42264 from yuriw/wip-yuriw-p2p-pacific

qa/tests: advanced pacific version to reflect the latest 16.2.5 point

Merge pull request #40694 from singuliere/wip-50209-pacific

pacific: qa/rgw: move ignore-pg-availability.yaml out of suites/rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42093 from thotz/rgwkmsvaultsslsupport-pacific

pacific: rgw: extending existing ssl support for vault KMS

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #42265 from neha-ojha/wip-rn-cleanup

pacific: doc/releases/pacific.rst: remove notes about autoscaler

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/tests: advanced pacific version to reflect the latest 16.2.5 point

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc/releases/pacific.rst: remove notes about autoscaler

This was reverted in 504d0cb120f02b1940f81505bd37a54773625af0, but
got added back by mistake in 6a3dde8b498af5e33242955a0609b0e27adf4877.

The relevant code in master was not reverted and is being reworked, we'll
update PendingReleaseNotes in master after that.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42250 from paulreece42/pacific

pacific: doc/rados/operations: s/max_misplaced/target_max_misplaced_ratio/

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40692 from singuliere/wip-50240-pacific

pacific: Warning Cleanup and Clang Compile Fix

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc: added documentation on additional throttling options for the PG balancer module

Signed-off-by: Paul Reece <paul@reece.cc>
(cherry picked from commit eafa979b7972be72f7bde1dd29f40b403f29a496)

doc/rados/operations: s/max_misplaced/target_max_misplaced_ratio/

max_misplaced with replaced by in target_max_misplaced_ratio
edbd592ee44e02a5328e1510879555c2f9dcfc9e, but the document was not
sync'ed. let's update it accordingly.

Fixes: https://tracker.ceph.com/issues/50745
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cc43462ab3d4b38cf0c496ae084b82c20ba449ae)

16.2.5

Merge pull request #41774 from trociny/wip-51143-pacific

pacific: cls/rgw: look for plain entries in non-ascii plain namespace too

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42153 from rhcs-dashboard/wip-51478-pacific

pacific: mgr/dashboard: fix OSD out count

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: huww98 <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42156 from rhcs-dashboard/wip-51489-pacific

pacific: mgr/dashboard: run cephadm-backend e2e tests with KCLI

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #42159 into pacific

* refs/pull/42159/head:
pybind/ceph_volume_client: use cephfs mkdirs api

Reviewed-by: Ramana Raja <rraja@redhat.com>

pybind/ceph_volume_client: use cephfs mkdirs api

This _mkdir_p should never have worked as the first directory it tries
to stat/mkdir is "", the empty string. This causes an assertion in the
client. I'm not sure how this code ever functioned without causing
faults. They look like:

    2021-07-01 02:15:04.449 7f7612b5ab80  3 client.178735 statx enter (relpath  want 2047)

The assertion is caused by a C++ exception:

/usr/include/c++/8/string_view:172: constexpr const _CharT& std::basic_string_view<_CharT, _Traits>::operator[](std::basic_string_view<_CharT, _Traits>::size_type) const [with _CharT = char$_Traits = std::char_traits<char>; std::basic_string_view<_CharT, _Traits>::size_type = long unsigned int]: Assertion '__pos < this->_M_len' failed.
        Aborted (core dumped)

Where relpath is just the path passed to Client::stat.

This commit only applies to Pacific and older because master no longer
has this library.

Fixes: https://tracker.ceph.com/issues/51492
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: run cephadm-backend e2e tests with KCLI

Fixes: https://tracker.ceph.com/issues/51300
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 5c03b49c4da55cf8d0c679ecb2c58182e4d3361a)

mgr/dashboard: fix OSD out count

Think we have 3 OSDs out but up (prepare for re-formatting to change min_alloc_size), and another OSD down but in
(during reboot). The dashboard will display "1 down, 2 out", which is obviously incorrect. It should be "1 down, 3 out"

Fixes: https://tracker.ceph.com/issues/51376
Signed-off-by: 胡玮文 <huww98@outlook.com>
(cherry picked from commit 921b48234777eba1a15c126f964cc47c0932879a)

Merge pull request #42140 from rhcs-dashboard/wip-51475-pacific

pacific: mgr/dashboard: User database migration has been cut out

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #42120 from neha-ojha/wip-51453-pacific

pacific: qa/suites/rados: add simultaneous scrubs to the thrasher

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #42065 from batrick/i51232

pacific: qa: scrub code does not join scrubopts with comma

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42067 from batrick/i51251

pacific: qa: fs:upgrade uses teuthology default distro

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: User database migration has been cut out

This PR will revert changes done by https://tracker.ceph.com/issues/49645 to auto-migrate user database v1 to v2.

Fixes: https://tracker.ceph.com/issues/51443
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 82922811612813840a684af993ca7c95a0b48d53)

Merge pull request #41842 from Daniel-Pivonka/pacific-backport-41740

pacific: mgr/nfs: do not depend on cephadm.utils

Merge pull request #42117 from liewegas/cephadm-june-last-pacific

pacific: cephadm june final batch

Merge pull request #42119 from rhcs-dashboard/wip-51447-pacific

pacific: mgr/dashboard: Fix bucket name input allowing space in the value

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42061 from batrick/i50913

pacific: MDS heartbeat timed out between during executing MDCache::start_files_to_recover()

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #42068 from batrick/i51286

pacific: MDSMonitor: crash when attempting to mount cephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42097 from vshankar/wip-51413

pacific: cephfs-mirror: reopen logs on SIGHUP

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42072 from batrick/i51414

pacific: mds: "FAILED ceph_assert(r == 0 || r == -2)"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42071 from batrick/i51412

pacific: mds: mkdir on ephemerally pinned directory sometimes blocked on journal flush

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42070 from batrick/i51324

pacific: pacific: client: abort after MDS blocklist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42069 from batrick/i51322

pacific: qa: test_data_scan.TestDataScan.test_pg_files AssertionError: Items in the second set but not the first:

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42066 from batrick/i51235

pacific: doc: pacific updates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42064 from batrick/i51231

pacific: pybind/mgr/snap_schedule: Invalid command: Unexpected argument 'fs=cephfs'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42063 from batrick/i51230

pacific: qa: fs:bugs does not specify distro

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pacific: qa: add missing package exclusions

ceph-immutable-object-cache does not exist on nautilus.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/pybind/test_ceph_argparse: add a test where args contains comma

to ensure the support for the new syntax of "prefix --name bon,jour"
does not break existing behavior of "prefix hello cruel,world" where value2
contains reads "hello,world", and the parsed result should be

prefix="prefix"
value=["hello", "cruel,world"]

instead of something like

prefix="prefix"
value=["hello", "cruel", "world"]

or

prefix="prefix"
value=["cruel", "world"]

the above test only applies to the test where "value" is a CephString.
if "value" is a CephChoices, the parsed argument should be

prefix="prefix"
value=["hello", "cruel", "world"]

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 9b6be59d6fa6ff7f6c5d773b6774728cf828b768)

cmake: run nose test using "python3 -m nose"

otherwise the tests are not performed at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 49be76ce11085bd333b5b8b281d8676c5ed450c2)

test/pybind/test_ceph_argparse: check for expected argument

instead of comparing the validated argument with empty dict, check for
its content for better coverage of testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cf4570d663d9c0fb656a54adc03fdfe83ead6ed0)

test/pybind/test_ceph_argparse: decorate helper method with @nottest

otherwise nose tries to run it as a test as its name contains "test", like

E.
======================================================================
ERROR: Runs validate in different arg/kargs ways.
----------------------------------------------------------------------
TypeError: arg_kwarg_test() missing 3 required positional arguments: 'prefix', 'args', and 'sig'

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3d7516f8daefcefa06a46981a62442d9193164f6)

test/pybind: do not test obsoleted command

"scrub" command was marked obsoleted in
e9a5ce0897efc6126caeebea9900bf05ec3d2174, so the test_ceph_argparse
cannot retrieve its command description using "get_command_descriptions"
cli tool anymore, let's drop the related test accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 77cb1fed0c30eec83ee0e4e0899dfdc1d4c5cc51)

test/pybind: drop forward and readforward from test

these two write modes were dropped in
a0a3ed324a130a9934735927600f01d1fa65bc0d, so update the test
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b8389557ad6297256f6a793723a5b601de739f2d)

test/pybind: do not use tab for indent

the python source file is idented using spaces.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 952285dcde501b38a63533768705526d3c5f02dc)

pybind/ceph_argparse: validate csv if desc.N

if desc.N is not None, we should take the argument as a comma separated
values, and validate the values individually.

restructure the validate() function and its helpers to pass the
validated args if desc.N explicitly, as desc.instance.val should only
hold a single value of desc.instance type, otherwise we need to reset
it after collecting all the argument in a CSV string value is parsed.

this change would be less consistent and hackish.

and update a test to verify this behavior.

Fixes: https://tracker.ceph.com/issues/51145
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0948a78ee6524911bd1fb76f1bf285e90e3cd349)

test/pybind: pep8 cleanups

* do not import unused symbols
* add two empty lines before method
* aligned indents

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit bc182888a3ca557e330e4317e0c50c1e80dbde4d)

Merge pull request #42045 from tchaikov/pacific-51354

pacific: osd/osd_type: use f->dump_unsigned() when appropriate

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #42019 from neha-ojha/wip-50791-pacific

pacific: osd: clear data digest when write_trunc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41944 from ronen-fr/wip-ronenf-pac-50346

pacific: osd/scrub: replace a ceph_assert() with a test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42062 from batrick/i51203

pacific: mds: CephFS kclient gets stuck when getattr() on a certain file

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #42060 from batrick/i50875

pacific: mds: MDSLog::journaler pointer maybe crash with use-after-free

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #42059 from batrick/i50848

pacific: mds: "cluster [ERR]   Error recovering journal 0x203: (2) No such file or directory" in cluster log"

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #42058 from batrick/i50846

pacific: mds: journal recovery thread is possibly asserting with mds_lock not locked

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #42057 from batrick/i50636

pacific: session dump includes completed_requests twice, once as an integer and once as a list

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #42056 from batrick/i50630

pacific: mds: Error ENOSYS: mds.a started profiler

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #41460 from idryomov/wip-qa-focal-pacific

pacific: qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41052 from lxbsz/wip-50445-pacific

pacific: client: make Inode to inherit from RefCountedObject

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #40856 from batrick/i50282

pacific: MDS slow request lookupino #0x100 on rank 1 block forever on dispatched

Reviewed-by: Jeff Layton <jlayton@redhat.com>

mgr: move GIL helpers to Gil.{h,cc}

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 88db7b190a974d70c08a1951b06b7fc9957b1b60)

qa/suites/rados: add simultaneous scrubs (multiple options) to the thrasher

Setting osd-max-scrubs to either 2 or 3.

Triggered by https://tracker.ceph.com/issues/50346

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
(cherry picked from commit d232c4e8d8e4ca0ff428960235b61ea398bbec07)

mgr/dashboard: Fix bucket name input allows space in the value

Fixes: https://tracker.ceph.com/issues/51368
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 2e94611a65c26f1db79d535f2a32e8bc747ecdd7)

mgr/ActivePyModules: streamline mgr_ips a bit

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 49d5a5bdea2ca925ee5030d19d3793927c2a1fbd)

mgr/ActivePyModules: don't take unnecessary lock

This isn't needed.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 3ee6f97b3ed172be98dd429a4aef54de5fb689f7)

pybind/mgr/mgr_module: use get("mgr_ips") for standby get_mgr_ip()

This is necessary to allow the standby mgr to bind to the right IP.

Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit d159adc20e005dc7f7a7cea06c2ee7aa6f06d559)

mgr: add get() for standby modules

Fixes: https://tracker.ceph.com/issues/51446
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit 8b8e7522982476d25685ad055d4a89f6173041f8)

doc/cephadm: improving "Monitoring the Upgrade"

This PR improves the section "Monitoring the Upgrade"
in the "Upgrading Ceph" chapter of the cephadm documentation.

This PR introduces a couple of section breaks with signposting
information in their titles, and rewrites some sentences in order
to reduce the cognitive load of the reader.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 2dcdef0d43fb269447b5a00c8c29dc508919768d)

doc/cephadm: improve "Upgrading Ceph" (main)

This PR makes a couple of minor improvements to the text under the
top-level section "Upgrading Ceph" in the "Upgrading Ceph" chapter of
the cephadm documentation.

This one, mercifully, contains only a couple of changes.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit b4bfb3efb6d61cc19b286bb6a43c4d7db643df59)

cephadm: Fix normalize_image_digest for local registries

Cause they typically don't have dots in it.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 8ef5657cf6aa3558a20516cfa1a793dfba513db4)
 - conflict with adjacent test for the journald logging, which was not backported

doc/cephadm: enrich "Disabling Automatic Deploy..."

This PR rewrites and reformats the section "Disabling Automatic
Deployment of Daemons" in the "Service Management" chapter of the
cephadm guide.

I've rewritten some sentences, removed some "please"s, and added
some section titles so that the content in this is better
signposted.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit fee80411297589f80ff458bd20902e547e41c603)

doc/cephadm: enrich "deployment of daemons"

service-management.rst contained a section called
"Deployment of Daemons" that needed a rewrite by
a native English speaker.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 14dfbff2cd844885ab521de944ebeb0bb92bb72d)

src/vstart: create rgw user with system flag

As dashboard requires the system flag to be enabled.

Signed-off-by: Varsha Rao <varao@redhat.com>
(cherry picked from commit 0d0cd4c6b0e8bbeaa71f54c831d67ef880e58016)

src/vstart: deploy rgw service with cephadm

Signed-off-by: Varsha Rao <varao@redhat.com>
(cherry picked from commit 63f5fa2e5e8946be4b4123c588bb133f1b18412b)

doc/install/get-packages: point to current stable release

Point to pacific for downloading the cephadm script

Co-authored-by: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: Sandro Bonazzola <sbonazzo@redhat.com>
(cherry picked from commit 136dbdef663a5d1c658dad52a794941ab6e6eb07)

doc: Add kcli utilization for development environments

How to use kcli to create ceph clusters (cephadm) and made easy code changes in
manager modules.

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit 3b82624e0cf0babec6868c1a999c5740fe04302f)

Merge pull request #42027 from rhcs-dashboard/wip-51148-pacific

pacific: mgr/dashboard: Include Network address and labels on Host Creation form

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

logrotate: include cephfs-mirror daemon

Fixes: http://tracker.ceph.com/issues/51318
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 957bae255c9f6fba8a4ccfcb885c2687784444aa)

cephfs-mirror: reopen logs on SIGHUP

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 68ad416766e2c73109dbf44f179d1b85b9d61994)

doc/radosgw/vault: add documentation for ssl authentication

Now RGW server can use custom ssl certs to authenticate with vault
server, adding steps for the same

Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
(cherry picked from commit c2af1cdc132da6b56e7f902e009dfda3ec0e2066)

rgw: add support for client cert and key for vault

Authenticate vault with help of user supplied client cert and keys.

Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
(cherry picked from commit 5b9139bd366d029871d29ca90971c5ad058fcf01)

Conflicts:
src/common/options/rgw.yaml.in
- added required changes in options.cc and legacy_config_opts.h

rgw: update debug log for curl error

The req_data->error_buf stores error returned by the curl command, add
that information to existing debug log.

Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
(cherry picked from commit d88ce5100eed90584791b74085a8d864a3796fab)

rgw: add support to consume user given ca cert for vault

Currently RGW can authenticate with vault via SSL using system certs.
With this patch user can provide custom ca cert and location of the file
can be specified in ceph.conf like this :
rgw_crypt_require_ssl = <file path>

Fixes: https://tracker.ceph.com/issues/47776
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
(cherry picked from commit 424dca0f3866c62e5c51cc8b2a080d680fddbe7f)
Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
Conflicts:
src/common/options/rgw.yaml.in
- added required changes in options.cc and legacy_config_opts.h

rgw: add seperate option for verify ssl for vault KMS engine

Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>
(cherry picked from commit 4a93faa2eec8077edb4c35a20b6b66bc610a6e4a)

Conflicts:
src/common/options/rgw.yaml.in
- added required changes in options.cc and legacy_config_opts.h

Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>

mgr/dashboard: Include Network address and labels on Host Creation form

The ability to create host by specifying network address and also create
labels.

https://tracker.ceph.com/issues/50318
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 7c1df692f270ad0e7c80559e5b190c98eea44981)

Conflicts:
qa/tasks/mgr/dashboard/test_host.py
   -  Keep the {'status': ''} parameter on the request body of _post
src/pybind/mgr/dashboard/controllers/host.py
   -  Added the addr and labels as parameter in the create() method
      as well as the add_host() method.
src/pybind/mgr/dashboard/services/orchestrator.py
   -  Added  the addr and labels as parameter in the add() method and
      removed the status parameter
src/pybind/mgr/dashboard/openapi.yaml
   -  Regenerated the openapi.yaml spec

mds: just respawn mds daemon when osd op requests timeout

Fixes: https://tracker.ceph.com/issues/51280
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit c854a4eea44a631079dfe481c235a323fae54b74)

mds: try to flush the mdlog when requesting the rdlock

After client requests' early_reply() the mdlog won't be flushed
immediately, but it will hold the write lock. So if the client
sends another request to a different MDS daemon, which then needs
to request read lock, then it maybe stuck at most for 5 seconds.
Let's try to flush the mdlog which might release the write locks
after mdlog is successfully flushed.

Fixes: https://tracker.ceph.com/issues/51069
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 63f7a165ec6bbf9cb1bded29f8bc5c017c01c768)

client: check if a mds rank is `up` before fetching connection addr

Client segfaults when trying to infer which mds rank a connection
reset call is coming from. It does this by iterating `mds_sessions`
and checking the mds addr (in mdsmap) to the `Connection *`.

However, cases where the mds is blocklisted, the client receives
an updated mdsmap in which the corresponding mds is not in `up`
set thereby resulting in a segfault when calling `mdsmap->get_addrs`
since it expects that the mds should be in `up` state.

Note that this leaves the `Connection *` as it is and does not clean
it up. That needs to fixed separately probably by maintaining a map
of `Connection *` to mds rank for lookup.

Fixes: http://tracker.ceph.com/issues/50530
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 951483fd36c28c65f12da7c058132d14a99ac5a9)

qa: always format the pgid in hex

If the pg number is larger than 9, this won't match the array index,
which was in dec just before this.

Fixes: https://tracker.ceph.com/issues/50808
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 2b9b2a3fbd5d4e1f01b96a96d4abaccffb8b9e3a)

mon/MDSMonitor: check fscid exists for legacy case

If a client does not have permission to see the legacy fs, the monitor
will throw an exception when looking up the mdsmap later in the code.
We need to check existence for both code paths.

Fixes: https://tracker.ceph.com/issues/51077
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 75ccd9d49c78bacf73b9e61d99a75fb455a8b85f)

qa: use centos latest for fs:upgrade

Fixes: https://tracker.ceph.com/issues/51250
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit b946d5a6b202f3770b3d4a5d34dd8b31b27d8498)

doc: add missing note about restoring allow_standby_replay

Fixes: https://tracker.ceph.com/issues/51187
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 103bb6b6bb3b5350946985eb1546fdeac1fa454f)

doc: clarify distributed ephemeral pins new mechanism

This documentation was for the old code, the new code (by Zheng)
fragments the directory and distributes those fragments.

Fixes: https://tracker.ceph.com/issues/51187
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit dcb340f3d82984055f488501cbfe4c267be4f98a)

doc: update mds_autoscaler documentation to reflect recent changes

Notably, Sage changed how this worked in [1].

[1] 6d8c0e4722eb5f3014dc76f6733afea08bcc8e72

Fixes: https://tracker.ceph.com/issues/51187
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 570c7ffc0149b9fbd20b50b54632f37990b27128)

qa: update scrub start code to use comma sep scrubopts

The documentation specifies this in [1] and yet we were using (I
believe) an older syntax:

    ceph tell mds.foo:0 scrub start / recursive force

instead of

    ceph tell mds.foo:0 scrub start / recursive,force

Oddly the former works at least as recently as in [2]:

    2021-06-03T07:11:42.071 DEBUG:teuthology.orchestra.run.smithi025:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / recursive force
    ...
    2021-06-03T07:11:42.268 INFO:teuthology.orchestra.run.smithi025.stdout:{
    2021-06-03T07:11:42.268 INFO:teuthology.orchestra.run.smithi025.stdout:    "return_code": 0,
    2021-06-03T07:11:42.268 INFO:teuthology.orchestra.run.smithi025.stdout:    "scrub_tag": "cf7a74b2-3eb2-4657-9274-ea504b1ebf8f",
    2021-06-03T07:11:42.269 INFO:teuthology.orchestra.run.smithi025.stdout:    "mode": "asynchronous"
    2021-06-03T07:11:42.269 INFO:teuthology.orchestra.run.smithi025.stdout:}

[1] https://docs.ceph.com/en/latest/cephfs/scrub/
[2] /ceph/teuthology-archive/pdonnell-2021-06-03_03:40:33-fs-wip-pdonnell-testing-20210603.020013-distro-basic-smithi/6148097/teuthology.log

Fixes: https://tracker.ceph.com/issues/51146
See-also: https://tracker.ceph.com/issues/51145
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0f505dc299fd10f27bab74b02356c1164814d0f5)

qa: update cli syntax to conventional

This was using an obscure syntax that worked at one time and wasn't
documented (AFAIK).

Fixes: https://tracker.ceph.com/issues/51182
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit a402b23c84aa399a989e1a6d9aa97046481c32c5)

qa: specify distro for fs:bugs

Fixes: https://tracker.ceph.com/issues/51184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 8eeb1455ee48c059a44e18b3fbae0c6a54d5d506)

mds: progress the recover queue immediately after the inode is enqueued

Fixes: https://tracker.ceph.com/issues/50840
Signed-off-by: "Yan, Zheng" <ukernel@gmail.com>
(cherry picked from commit 616ebc3f04620c22439a62f3af35845fa98e508f)

mds: switch to mdr->get_client()

The get_client() will check the mdr's type, if not a client request
it will return -1.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 4357fa8028f320e973644b7b74aedab567afde2e)

mds: add heartbeat_reset() in start_files_to_reover() to avoid timedout issue

Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>
(cherry picked from commit f43f1d85cf162a3204cd42af620303067c9d162f)

mds: place the journaler pointer under the mds_lock

When the _recovery_thread is trying to reformat the journal, it
will delete the old journal pointer and assign with a new one,
during this the mds_lock is unlocked. That means in other threads,
such as when 'flush journal', who are using the MDSLog::journaler
pointer will potentially hit use-after-free bug.

Fixes: https://tracker.ceph.com/issues/50807
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 0c143922f8a7bad0f19a0a2c97121a470b8fcba5)