Merge pull request #46425 from lxbsz/wip-55660

Pacific: qa: add filesystem/file sync stuck test support

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46424 from lxbsz/wip-55659

Pacific: mds: trigger to flush the mdlog in handle_find_ino()

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46423 from lxbsz/wip-55757

Pacific: mds: flush mdlog if locked and still has wanted caps not satisfied

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #46688 from trociny/wip-56053-pacific

pacific: mgr: relax "pending_service_map.epoch > service_map.epoch" assert

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46775 from adk3798/wip-56042-pacific

pacific: mgr/cephadm: try to get FQDN for active instance

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #46810 from rhcs-dashboard/wip-56164-pacific

pacific: mgr/dashboard: dashboard help command showing wrong syntax for login-banner

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: dashboard help command showing wrong syntax for login-banner

Signed-off-by: Sarthak0702 <sarthak.dev.0702@gmail.com>
(cherry picked from commit 826741f192e754ed32b1636b06b9939efa5c5226)

Merge pull request #46529 from rhcs-dashboard/wip-54623-pacific

pacific: mgr/dashboard: BDD approach for the dashboard cephadm e2e

Reviewed-by: Sarthak Gupta <sarthak.dev.0702@gmail.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #46817 from zdover23/wip-doc-2022-06-23-backport-46769-pacific-dev-guide-command-context

pacific: doc/dev: add context note to dev guide config

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46622 from cfsnyder/wip-55998-pacific

pacific: rgw: log access key id in ops logs

Merge pull request #46737 from rhcs-dashboard/wip-iops-pacific

pacific: mgr/dashboard: iops optimized option enabled

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: save mgr logs inside a folder on kcli e2e

Save the mgr container logs of cephadm inside a folder and later on
archive it and get it as an artifact on the cephadm dashboard e2e jobs

Fixes: https://tracker.ceph.com/issues/55247
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit d74defbbb3282f056e26619ecd65fc63216f2f59)

Merge pull request #46570 from guits/wip-55945-pacific

pacific: cephadm: add 'is_paused' field in orch status output

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #46776 from adk3798/wip-55948-pacific

pacific: mgr/cephadm: capture exception when not able to list upgrade tags

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #46779 from adk3798/wip-55950-pacific

pacific: mgr/cephadm: check if a service exists before trying to restart it

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #46780 from adk3798/wip-55963-pacific

pacific: mgr/cephadm: adding logic to close ports when removing a daemon

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #46619 from cfsnyder/wip-55996-pacific

pacific: rgw: reopen ops log file on sighup

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #46597 from idryomov/wip-qa-rwl-on-tmpfs-pacific

pacific: qa/suites/rbd: place cache file on tmpfs for xfstests

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #46593 from idryomov/wip-rbd-unlink-newest-snap-at-capacity-pacific

pacific: librbd: unlink newest mirror snapshot when at capacity, bump capacity

Reviewed-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #46590 from idryomov/wip-rbd-preserve-non-primary-snap-pacific

pacific: rbd-mirror: don't prune non-primary snapshot when restarting delta sync

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

Merge pull request #46668 from nmshelke/wip-56014-pacific

pacific: mgr/volumes: subvolumegroup quotas

Reviewed-by: Venky Shankar vshankar@redhat.com
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46515 from nmshelke/wip-55802-pacific

pacific: mgr/volumes: set, get, list and remove metadata of snapshot

Reviewed-by: Venky Shankar vshankar@redhat.com

doc/dev: add context note to dev guide config

This PR adds a note directing first-time cloners of
their Ceph git forks to make sure to cd into the ceph/
directory before trying to run the "git config" commands.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 3e6bcd7f0be2d0f896146c563a3381c9b7ea0e83)

doc: Update quota kclient limitation

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit cd92ea0cb459ffd31d164a661634f913eab59594)

qa: Add tests which validates quota limitation for subvolumegroup quota

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 256c75e260dde91ab71fc0c43407578ad2e9f553)

doc: Document subvolumegroup quota commands

Add documentation for subvolume group quota along
with 'subvolumegroup resize' and 'subvolumegroup info'
commands

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 168b912b026700644416390caf3540c1aaa75a62)

qa: Add subvolume removal tests when it's group quota exceeds

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit e9fe37b8e1f451c94ce3d1a63c090fd57444ab3f)

qa: Add tests for subvolumegroup idempotence creation

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 12e003f980cee4c2cc5b9c6f9dbf2d8d82920e1e)

qa: Add tests for subvolume removal on group quota set

Validates the subvolume removal is successful if the
corresponding group's quota is set.

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit c006181803bcc2c3e26fc81dbfc8226f0fabe753)

qa: Add tests for subvolumegroup resize

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit dbe8cf17926c296399391674ceed8d5deb033996)

qa: Add subvolume group quota tests

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 085700bf194558bc1469bc98fe8234e46252e146)

mgr/volumes: Disable quota for mgr libcephfs connection

This is done to give 'mgr' libcephfs connection right to bypass
quota. The mgr/volumes plugin maintains configuration files
with in the directory where the user has enforced quota. So
when the quota is met, certain mgr/volumes apis don't work as
intended. e.g., When subvolumegroup quota is met, the group's
subvolume removal with '--retain-snapshots' fails.

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 3cfcbef284f371bfbec01311e6dad0172572fe9c)

client: Makes quota optional

Make quota optional. This commit basically reverts the
commit da9f1d829484fe35e6d5839c19afa371e444c194

This is done to give 'mgr' libcephfs connection right to bypass
quota. The mgr/volumes plugin maintains configuration files
with in the directory where the user has enforced quota. So
when the quota is met, certain mgr/volumes apis don't work as
intended. e.g., When subvolumegroup quota is met, the group's
subvolume removal with '--retain-snapshots' fails.

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit f365a9a5fe465b5d5cc81e5d4dd136bd8989cc20)

Conflicts:
src/common/options/mds-client.yaml.in
- pacific release not having yamls to store config hence added option in
  src/common/options.cc

mgr/volumes: Add subvolumegroup resize cmd

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 346964a89ce3b1ad80f8513387e7d84b036016b5)

mgr/volumes: Add subvolumegroup info cmd

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 9e1b7ed39e25f28fdfc66b394d1f3ee61c7ce0ba)

mgr/volumes: Support subvolumegroup quota

Fixes: https://tracker.ceph.com/issues/53509
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 06a869cb8ad1a9e5f966fa49046f6eb6674c29fe)

Conflicts:
src/pybind/mgr/volumes/fs/operations/group.py
- Updates in defination of create_groups
src/pybind/mgr/volumes/fs/volume.py
- Added set_group_attrs in import list and split long line

mgr/cephadm: try to get FQDN for active instance

Fixes: https://tracker.ceph.com/issues/55674
Signed-off-by: Tatjana Dehler <tdehler@suse.com>
(cherry picked from commit d0385e030b391f588b4ec0dc707d5d46778a2aaa)

Conflicts:
src/pybind/mgr/cephadm/tests/test_services.py

mgr/cephadm: adding logic to close ports when removing a daemon
Fixes: https://tracker.ceph.com/issues/52906
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 4deb546ffd67ac8f05d2788150764a26b5671b87)

Conflicts:
src/pybind/mgr/cephadm/serve.py

mgr/cephadm: check if a service exists before trying to restart it
Fixes: https://tracker.ceph.com/issues/55800
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 6b76753c3cabf9663fa1daa47c7bcb7df110a94c)

mgr/cephadm: capture exception when not able to list upgrade tags
Fixes: https://tracker.ceph.com/issues/55801
Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit 0e7a4366c0c1edd74d52acad5ed4dc3df0ef7679)

Merge pull request #46762 from idryomov/wip-wnbd-main-branch-pacific

pacific: win32_deps_build.sh: master -> main for wnbd

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

win32_deps_build.sh: master -> main for wnbd

wnbd is switching, see https://github.com/cloudbase/wnbd/pull/67.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 27048bf2b29a7060958a2b6dda8247cfad8a0334)

Merge pull request #46348 from zdover23/wip-disk-benchmarking-and-cache-pacific-backport-2022-05-19

pacific: doc: add disk benchmarking and cache recommendations

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46040 from cbodley/wip-55457

pacific: rgw: RGWCoroutine::set_sleeping() checks for null stack

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #46631 from rzarzynski/wip-pglog-trim-dups-pacific

pacific: tools: ceph-objectstore-tool is able to trim solely pg log dups' entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #46596 from kotreshhr/wip-55927-pacific

pacific: client/fuse: Fix directory DACs overriding for root

Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #46567 from vshankar/tr-55932

pacific: mds: do not assert early on when issuing client leases

Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #45293 from joscollin/wip-54479-pacific

pacific: mgr/stats: be resilient to offline MDS rank-0

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #46608 from rzarzynski/wip-55982-pacific

pacific: osd: log the number of 'dups' entries in a PG Log

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: iops optimized option enabled

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit 86cb558a7772df587c8eca7bdfecc95928dc7730)

Merge pull request #46690 from rhcs-dashboard/wip-55941-pacific

pacific: mgr/dashboard: Feature 54330 osd creation workflow

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #46691 from sseshasa/wip-56059-pacific

pacific: mon/OSDMonitor: Ensure kvmon() is writeable before handling "osd new" cmd

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: reopen ops log file on sighup

Handles radosgw SIGHUP such that ops log file is reopened if applicable.

Fixes: https://tracker.ceph.com/issues/53788
Signed-off-by: Cory Snyder <csnyder@iland.com>
(cherry picked from commit f26523f59add768d3d2305642bb641d6675f8f82)

Conflicts:
src/rgw/rgw_main.cc

Cherry-pick notes:
- OpsLogRados initialized differently in Pacific

Merge pull request #46722 from zdover23/wip-doc-2022-06-15-backport-46712-pacific-master-to-main-dev-guide-basic-workflow

pacific: doc/dev: s/master/main/ in title

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: s/master/main/ in title

This changes "master" to "main" in a title. If we lived in an
ideal world, this would have been a part of PR#46678.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit f5fd158bea2a54b83c7f75cd806885de44045ad6)

Merge pull request #45928 from kotreshhr/wip-55349-pacific

pacific: mgr/volumes: Show clone failure reason in clone status command

Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

qa: set, get, list and remove custom metadata for snapshot

Following test are added:
1. Set custom metadata for subvolume snapshot.
2. Set custom metadata for subvolume snapshot(Idempotency).
3. Get custom metadata for specified key.
4. Get custom metadata if specified key not exist (Expecting error ENOENT).
5. Get custom metadata if no any key-value is added means section not exist (Expecting error ENOENT).
6. Update value for existing key in custom metadata.
7. List custom metadata of subvolume snapshot.
8. List custom metadata of subvolume snapshot if no any key-value is added (Expect empty json/dictionary)
9. Remove custom metadata for specified key.
10. Remove custom metadata if specified key not exist (Expecting error ENOENT).
11. Remove custom metadata if no any key-value is added means section not exist (Expecting error ENOENT).
12. Remove custom metadata with --force option.
13. Remove custom metadata with --force option if specified key not exist (Expecting command to succeed because of '--force' option)
14. Remove subvolume snapshot and verify whether metadata for snapshot is removed or not

Fixes: https://tracker.ceph.com/issues/55401
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
(cherry picked from commit 6fd28cc9d67b96ba87f0dffbf41d626229e904e3)

docs: set, get, list and remove custom metadata for snapshot

Set custom metadata on the snapshot as a key-value pair using
    $ ceph fs subvolume snapshot metadata set <vol_name> <subvol_name> <snap_name> <key_name> <value> [--group_name <subvol_group_name>]
    note: If the key_name already exists then the old value will get replaced by the new value.
    note: The key_name and value should be a string of ASCII characters (as specified in python's string.printable). The key_name is case-insensitive and always stored in lower case.
    note: Custom metadata on a snapshots is not preserved when snapshotting the subvolume, and hence, is also not preserved when cloning the subvolume snapshot.

Get custom metadata set on the snapshot using the metadata key::
    $ ceph fs subvolume snapshot metadata get <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>]

List custom metadata (key-value pairs) set on the snapshot using::
    $ ceph fs subvolume snapshot metadata ls <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]

Remove custom metadata set on the snapshot using the metadata key::
    $ ceph fs subvolume snapshot metadata rm <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>] [--force]
    Using the '--force' flag allows the command to succeed that would otherwise fail if the metadata key did not exist.

Fixes: https://tracker.ceph.com/issues/55401
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
(cherry picked from commit 59a0cbc14bf2832080e983729de5c462ddc70bb3)

mgr/volumes: set, get, list and remove custom metadata for snapshot

If CephFS in ODF configured in external mode, user like to use
subvolume snapshot metadata to store some Openshift specific
information, as the PVC/PV/namespace the subvolumes/snapshot
are coming from. For RBD volumes, it's possible to add metadata
information to the images using the 'rbd image-meta' command.
However, this feature is not available for CephFS volumes.
We'd like to request this capability.

Adding following commands:
    ceph fs subvolume snapshot metadata set <vol_name> <sub_name> <snap_name> <key_name> <value> [<group_name>]
    ceph fs subvolume snapshot metadata get <vol_name> <sub_name> <snap_name> <key_name> [<group_name>]
    ceph fs subvolume snapshot metadata ls <vol_name> <sub_name> <snap_name> [<group_name>]
    ceph fs subvolume snapshot metadata rm <vol_name> <sub_name> <snap_name> <key_name> [<group_name>] [--force]

Fixes: https://tracker.ceph.com/issues/55401
Signed-off-by: Nikhilkumar Shelke <nshelke@redhat.com>
(cherry picked from commit 559222cfe8d552cd2d7aef7361de4140820ae74a)

Merge pull request #45961 from vshankar/tr-55375

pacific: mgr/volumes: set, get, list and remove metadata of subvolume

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #46710 from zdover23/wip-doc-2022-06-15-backport-pacific-46705-master-to-main-dev-guide-merging

pacific: doc/dev_guide: s/master/main in merging.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev_guide: s/master/main in merging.rst

This changes the branch name "master" to the branch name
"main" in merging.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 52da71f0ab0a4606a73be749b3d983969bd34972)

Merge pull request #46704 from zdover23/wip-doc-2022-06-15-backport-pacific-46678-master-to-main-dev-guide-basic-workflow

pacific: doc/dev: s/master/main/ in basic workflow

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46662 from zdover23/wip-doc-2022-06-13-backport-pacific-46651-master-to-main

pacific: doc/dev: s/master/main/ essentials.rst dev guide

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46663 from zdover23/wip-doc-2022-06-13-backport-pacific-46659-hardware-recs-polish

pacific: doc/start: Polish network section of hardware-recommendations.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: s/master/main/ in basic workflow

This PR changes "master" to "main" in the
basic_workflow.rst file. I have even changed
"master" to "main" in some terminal output from
several years ago. This isn't historically ac-
curate, of course, but my hope is that this change
will prevent someone in the future from being con-
fused about why an antiquated branch name is ref-
erred to.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit f5cfc22445a3e454e572873cdcfbc1852a38aa17)

Merge pull request #46039 from cbodley/wip-55456

pacific: rgw: OpsLogFile::stop() signals under mutex

Reviewed-by: Cory Snyder <csnyder@iland.com>

Merge pull request #45410 from cfsnyder/wip-54537-pacific

pacific: rgw: fix segfault in OpsLogRados::log when realm is reloaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46112 from Matan-B/wip-55514-pacific

pacific: mount/conf: Fix IPv6 parsing

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #45893 from nkshirsagar/wip-55296-pacific

pacific: Catch exception if thrown by __generate_command_map()

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #45847 from dvanders/wip-55250-pacific

pacific: rgw_reshard: drop olh entries with empty name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45486 from cfsnyder/wip-53868-pacific

pacific: rgw: return OK on consecutive complete-multipart reqs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #46139 from kotreshhr/wip-55335-pacific

pacific: mgr/volumes: Fix idempotent subvolume rm

Reviewed-by: Venky Shankar vshankar@redhat.com
Reviewed-by: Nikhilkumar Shelke <nshelke@redhat.com>

Merge pull request #46138 from nmshelke/wip-55412-pacific

pacific: mds: add a perf counter to record slow replies

Reviewed-by: Venky Shankar vshankar@redhat.com
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #45968 from vshankar/tr-55384

pacific: mgr/snap_schedule: add time zone suffix to snapshot dir name

Reviewed-by: Venky Shankar vshankar@redhat.com

Merge pull request #45792 from lxbsz/wip-55192

pacific: client: buffer the truncate if we have the Fx caps

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45900 from ivancich/wip-55044-pacific

pacific: cls/rgw: rgw_dir_suggest_changes detects race with completion

Reviewed-by: casey Bodley <cbodley@redhat.com>

rgw: RGWCoroutine::set_sleeping() checks for null stack

users of the RGWOmapAppend coroutine don't manage the lifetime of its
underlying coroutine stack, so end up making calls on RGWOmapAppend
after its stack goes away. this null check is a band-aid, and there are
still several other calls in RGWCoroutine that don't check for null
stack

Fixes: https://tracker.ceph.com/issues/49302
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3f0f831d66c7d43c9872f5de2aceb68aef4004d8)

mon/OSDMonitor: Ensure kvmon() is writeable before handling "osd new" cmd

Before proceeding to handle "osd new" mon command as part of
OSDMonitor::prepare_command_impl(), a check is made to verify if the
authmon is writeable. Later on, prepare_command_osd_new() invokes
KVMonitor::do_osd_new() to create pending dmcrypt keys and calls
propose_pending(). The propose could fail (with an assertion failure)
if there was a prior mon command that resulted in the kvmon invoking
propose_pending().

In order to avoid such a situation, introduce a check to verify that
kvmon is also writeable in OSDMonitor::prepare_command_impl(). If it
is not writeable, the op is pushed into the wait_for_active context
queue to be retried later.

Fixes: https://tracker.ceph.com/issues/55773
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 9a0d42c9fbcf64711a791edcf9cafdba57d901d7)

mgr/dashboard: test througput deployment option

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit d58b9499208cd611897518009495f9e80515349d)

Conflicts:
src/pybind/mgr/dashboard/tests/test_osd.py

Besides test deployment options, there were extra changes that came
from main which had to be removed in favor of specific deployment
options tests.

mgr/dashboard: throughput optimized option enabled

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit f2474bcb767893dc750b8f1231b4583925f9bfb1)

mgr/dashboard: OSD Creation Workflow initial works

Introducing the Cost/Capacity Optimized deployment option
Used bootstrap accordion
Adapted the e2e but not written new tests for the deployment option

Fixes: https://tracker.ceph.com/issues/54340
Fixes: https://tracker.ceph.com/issues/54563
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>
(cherry picked from commit 6c2dcb740efb793a3f6ef593793151a34c19ca01)

mgr/dashboard: retrieve disk status

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
(cherry picked from commit a1d1c853a5e4ff9a317591b99b75e005ccc862c9)

Conflicts:
src/pybind/mgr/dashboard/tests/test_osd.py

There were some extra tests that came with this commit. This was
resolved by adding test_deployment_options alone instead of all the
incoming changes.

mgr: relax "pending_service_map.epoch > service_map.epoch" assert

When we are activating we may receive several service map updates
initiated by the previous active mgr. Treat them all as initial map.

The code also adds "pending_service_map_dirty == 0" assert, which we
expect is true when receiving an initial map -- otherwise we can't
just initialize pending_service_map with received map.

Fixes: https://tracker.ceph.com/issues/51835
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit cc2721ccdb33248a732abd1919df808ef8a1f80f)

mgr/dashboard: fix drain e2e failure

Cypress sometimes fail to register the click and that causes the
deselect/select to not happen properly. Deselecting the row immediately
after performing the action makes it pass from cypress.

Fixes: https://tracker.ceph.com/issues/55741
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit dcf0445153694b87ac685f4cfc4094d7d1a708da)

Merge pull request #46456 from rhcs-dashboard/wip-55590-pacific

pacific: mgr/dashboard:  WDC multipath bug fixes

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: nSedrickm <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

rgw: fix segfault in OpsLogRados::log when realm is reloaded

We weren't previously handling the deallocation of the store when
a realm was reloaded. Now passing a const reference to the pointer.

Fixes: https://tracker.ceph.com/issues/54130
Signed-off-by: Cory Snyder <csnyder@iland.com>
(cherry picked from commit 0713f65355586b2f6ceeb6bbce8763158847e5ed)

Conflicts:
src/rgw/rgw_log.cc
src/rgw/rgw_log.h

Cherry-pick notes:
- conflicts due to rename of RGWRados to rgw::sal::Store after Pacific

(cherry picked from commit 44a37dc0023a69da33620dc7b6e112221f7a6a0b)

doc/start: Polish network section of hardware-recommendations.rst

Harmonize network throughput notation, minor tweaks to wording.
Followup to #46637

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 2eb173fef9dc9ff34078b0ae13e9fcc398cdbaa3)

doc/dev: s/master/main/ essentials.rst dev guide

This PR changes all reference to the "master" branch
to references to the "main" branch (because we renamed
"master" to main", and the docs now need to reflect that).

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 728b8f26746c733988302d52cce4c46ebc607f63)

Merge pull request #46653 from zdover23/wip-doc-2022-06-13-backport-46637-pacific-hardware-recs-network

pacific: doc/start: rewrite hardware-recs networks section

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46656 from zdover23/wip-doc-2022-06-13-backport-46583-pacific-hardware-recs-osd-and-mds-acros

pacific: doc/start: make OSD and MDS structures parallel

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46657 from zdover23/wip-doc-2022-06-13-backport-46633-pacific-start-intro-crush-para-rewrite

pacific: doc/start: rewrite CRUSH para

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #46368 from tchaikov/pacific-pr-46355

pacific: ceph.spec.in: disable annobin plugin if compile with gcc-toolset

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #46365 from tchaikov/pacific-pr-44007

pacific: cmake: check for python(\d)\.(\d+) when building boost

Reviewed-by: Dirk Mueller <dmueller@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #46312 from b-ranto/wip-alerts-pacific

[pacific] mgr/alerts: Add Message-Id and Date header to sent emails

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #46255 from tserong/wip-55616-pacific

pacific: test: No direct use of nose

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/start: rewrite CRUSH para

This PR supersedes https://github.com/ceph/ceph/pull/46584
and makes changes suggested by Anthony D'Atri that improve
the coherence and consistency of the paragraph that explains
the basics of the CRUSH algorithm.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit ba1a85b292fcd30020897a486cb452381c998690)

doc/start: make OSD and MDS structures parallel

This PR makes the "Ceph OSDs" and "MDSs" bullet points
parallel by naming "object storage daemon" before referring
to the (admittedly more common and colloquial, but surely
unknown to people who genuinely require a document called
'Intro') acronym "OSD".

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 4f6edb92b9c4b7c47354d8ad66fdee6c06389541)

doc/start: rewrite hardware-recs networks section

This rewrites the first two-thirds of the "Networks"
section of the Hardware Recommendations page in the
Intro to Ceph document. I have tried to divide the
techincal content in this section into subsections
that foreground the various subjects covered.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit 778d3c0b59148e2a3b6bb662437778692b1988e3)

tools: ceph-objectstore-tool is able to trim pg log dups' entries.

The main assumption is trimming just dups doesn't need any update
to the corresponding pg_info_t.

Testing:

1. cluster without the autoscaler
```
rzarz@ubulap:~/dev/ceph/build$ MON=1 MGR=1 OSD=3 MGR=1 MDS=0 ../src/vstart.sh -l -b -n -o "osd_pg_log_dups_tracked=3000000" -o "osd_pool_default_pg_autoscale_mode=off"
```

2. 8 PGs in the testing pool.
```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph osd pool create test-pool 8 8
```

3. Provisioning dups with rados bench
```
bin/rados bench -p test-pool 300 write -b 4096  --no-cleanup
...
Total time run:         300.034
Total writes made:      103413
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.34637
Stddev Bandwidth:       0.589071
Max bandwidth (MB/sec): 2.4375
Min bandwidth (MB/sec): 0.902344
Average IOPS:           344
Stddev IOPS:            150.802
Max IOPS:               624
Min IOPS:               231
Average Latency(s):     0.0464151
Stddev Latency(s):      0.0183627
Max latency(s):         0.0928424
Min latency(s):         0.0131932
```

4. Killing osd.0
```
rzarz@ubulap:~/dev/ceph/build$ kill 2572129 # pid of osd.0
```

5. Listing PGs on osd.0 and calculating number of pg log's entries and
dups:

```
rzarz@ubulap:~/dev/ceph/build$ bin/ceph-objectstore-tool --data-path dev/osd0 --op list-pgs --pgid 2.c > osd0_pgs.txt
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
3100
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

6. Trimming dups
```
rzarz@ubulap:~/dev/ceph/build$ CEPH_ARGS="--osd_pg_log_dups_tracked 2500 --osd_pg_log_trim_max=100" bin/ceph-objectstore-tool --data-path dev/osd0 --op trim-pg-log-dups --pgid 2.7
max_dup_entries=2500 max_chunk_size=100
Removing keys dup_0000000020.00000000000000000001 - dup_0000000020.00000000000000000100
Removing keys dup_0000000020.00000000000000000101 - dup_0000000020.00000000000000000200
Removing keys dup_0000000020.00000000000000000201 - dup_0000000020.00000000000000000300
Removing keys dup_0000000020.00000000000000000301 - dup_0000000020.00000000000000000400
Removing keys dup_0000000020.00000000000000000401 - dup_0000000020.00000000000000000500
Removing keys dup_0000000020.00000000000000000501 - dup_0000000020.00000000000000000600
Finished trimming, now compacting...
Finished trimming pg log dups
```

7. Checking number of pg log's entries and dups
```
rzarz@ubulap:~/dev/ceph/build$ for pgid in `cat osd0_pgs.txt`; do echo $pgid; bin/ceph-objectstore-tool --data-path dev/osd0 --op log --pgid $pgid | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; done
2.7
10020
2500
2.6
10100
3000
2.3
10012
2800
2.1
10049
2900
2.2
10057
2700
2.0
10027
2900
2.5
10077
2700
2.4
10072
2900
1.0
97
0
```

Conflicts:
        src/tools/ceph_objectstore_tool.cc -- undetected conflict
        with d5445b8f113797718a0dbb05e884a6bffbfed76a. Fixed by
        adopting the patch no not require the `unique_ptr<T>::get()`.

Fixes: https://tracker.ceph.com/issues/53729
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit a2190f901abf2fed20c65e59f53b38c10545cb5a)