mgr/cephadm: add ingress support for ssl rgw service

This commit adds support for ssl backend rgw. This allows for full
network encryption.

Fixes: https://tracker.ceph.com/issues/51972
Signed-off-by: Frank Ederveen <frank.ederveen@gmail.com>
(cherry picked from commit ab9d6ea42453225189f3ded8863c00bc55b71189)

Merge pull request #49857 from adk3798/wip-58446-quincy

quincy: mgr/cephadm: make logging refresh metadata to debug logs configurable

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49853 from adk3798/wip-58538-quincy

quincy: mgr/cephadm: increase ingress timeout values

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50356 from zdover23/wip-doc-2023-03-03-backport-50351-to-quincy

quincy: doc/radosgw: format admonitions

Merge pull request #50090 from mchangir/wip-58599-quincy

quincy: mon: disable snap id allocation for fsmap pools

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49957 from dparmar18/wip-58604-quincy

quincy: qa/fs/mixed-clients: specify distros for tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49942 from lxbsz/wip-58602

quincy: ceph_fuse: retry the test_dentry_handling if fails

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49939 from lxbsz/wip-58609

quincy: mds: fix and skip submitting invalid osd request

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49655 from dparmar18/wip-58253-quincy

quincy: mds/PurgeQueue: don't consider filer_max_purge_ops when _calculate_ops

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49654 from dparmar18/wip-58348-quincy

quincy: mgr/nfs: handle bad cluster name during info command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50151 from neesingh-rh/wip-57970-quincy

quincy: cephfs-top: addition of sort feature and limit option

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #49863 from adk3798/wip-58197-quincy

quincy: mgr/cephadm: save host cache data after scheduling daemon action

doc/radosgw: format admonitions

Break up the text of two similar admonitions into three paragraphs (in
each of the two instances). This makes the content of the admonition
much easier to read at a glance.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4db68f5711f48b190f0135a3a7a6df8dfcf5abfa)

Merge pull request #49284 from adk3798/wip-58147-quincy

quincy: mgr/cephadm: some master -> main cleanup

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #50336 from zdover23/wip-doc-2023-03-02-backport-50335-to-quincy

quincy: doc/cephadm: update cephadm compatability and stability page

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephadm: update cephadm compatability and stability page

This page is very out of date. This commit probably doesn't
cover everything there is to say about stability and compatability
in cephadm, but it at least gets it noticeably closer to reality

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 383f46aa79e7a6f5571405a352ad023983560625)

Merge pull request #50331 from zdover23/wip-doc-2023-03-02-backport-50316-to-quincy

quincy: doc/rgw: improve diagram caption

doc/rgw: improve diagram caption

Improve the syntax and semantics of the caption of the diagram
"Replication of Object Data Between Zones" in doc/radosgw/multisite.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e8aa50e240141949e2cf91c3b216808f4e055887)

Merge pull request #50241 from cbodley/wip-58768

quincy: cls/rgw: remove index entry after cancelling last racing delete op

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #50240 from cbodley/wip-58772

quincy: rgw: Fix segfault due to concurrent socket use at timeout

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #50239 from cbodley/wip-58770

quincy: rgw/beast: StreamIO remembers connection errors for graceful shutdown

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49689 from joscollin/wip-58409-quincy

quincy: doc: document the relevance of mds_namespace mount option

Merge pull request #50304 from idryomov/wip-qa-workunits-windows-ownership-quincy

quincy: CODEOWNERS: assign qa/workunits/windows to RBD

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

CODEOWNERS: assign qa/workunits/windows to RBD

Assume ownership of qa/workunits/windows.  Despite the generic name,
currently it has just rbd-wnbd tests.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 38898dfeb8e9ced2603b599d1b590200de54150a)

Merge pull request #50297 from zdover23/wip-doc-2023-02-28-backport-50295-to-quincy

quincy: doc/radosgw: s/zone group/zonegroup/g et alia

Merge pull request #50231 from rhcs-dashboard/node-14-quincy

quincy: cmake: bump node version to 14

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc/radosgw: s/zone group/zonegroup/g et alia

s/zone group/zonegroup/ where simple greps failed to find instances of
"zone group" that were spread across two lines; break a paragraph into
two paragraphs so that each paragraph has a thematic idea of its own.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d1f2e9f3a6837f2b19acaabfb3261849d547d5f8)

Merge pull request #50293 from zdover23/wip-doc-2023-02-28-backport-50276-to-quincy

quincy: doc/rgw: caption a diagram

doc/rgw: caption a diagram

Add an explanation of a diagram that shows how object data is replicated
between zones within a zonegroup.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ce2bd7cb2603c479c17ac18b9277b903a02bb06a)

Merge pull request #49883 from petrutlucian94/quincy_rbd_wnbd_test

quincy: qa/workunits/windows: backport rbd-wnbd tests

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #50287 from zdover23/wip-doc-2023-02-28-backport-50274-to-quincy

quincy: doc/rgw: remove "tertiary", link to procedure

Merge pull request #49856 from adk3798/wip-58447-quincy

quincy: mgr/cephadm: improve offline host handling, mostly around upgrade

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49862 from adk3798/wip-58201-quincy

quincy: mgr/cephadm: specify ports for iscsi

Reviewed-by: Teoman ONAY <tonay@ibm.com>

Merge pull request #50277 from zdover23/wip-doc-2023-02-27-backport-50264-to-quincy

quincy: doc/radosgw: multisite - edit "functional changes"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: remove "tertiary", link to procedure

Remove the term "tertiary zone" and replace it with "second secondary
zone" (because there is no such thing as a tertiary zone). Link to the
procedure for creating a secondary zone in a place where such a link is
helpful to the reader.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8e273199d09a670578b0c1076bc18dbd113d42dd)

Merge pull request #50271 from zdover23/wip-doc-2023-02-27-backport-50267-to-quincy

quincy: doc/glossary: Add "zone" to glossary.rst

Merge pull request #50234 from rhcs-dashboard/wip-58830-quincy

quincy: mgr/dashboard: fix constraints.txt

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc/radosgw: multisite - edit "functional changes"

Improve the wording of the section "Functional Changes from Infernalis"
in doc/radosgw/multisite.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 35edcc17c96010328e8ee85648a3b74887aef7b9)

doc/glossary: Add "zone" to glossary.rst

Add "Zone" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 495eb97fd7fa3d53d27ad8d7df993df80b708ca1)

mgr/dashboard: fix constraints.txt again

The previous attempt to fix this issue
(https://github.com/ceph/ceph/pull/50207) was not successful, since the
PIP syntax for compatible versions (~=) behaves differently compared to
other package management tools (e.g.: npm).

In this case, package~=x.y.z equasl to package>=x.y.z;package==x.y.*,
instead of the desired package>=x.y.z;package==x.*.

Fixes: tracker.ceph.com/issues/58827
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 1c7fb66856d14a02bf9d4935c7e6bd1832bb3aae)

Merge pull request #50262 from zdover23/wip-doc-2023-02-27-backport-50260-to-quincy

quincy: doc/glossary: add "client" to glossary

doc/glossary: add "client" to glossary

Add a definition of "client" to doc/glossary.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 2dc6f56f26c55f89f3445ff97a88a704622b55c0)

Merge pull request #50258 from zdover23/wip-doc-2023-02-26-backport-50257-to-quincy

quincy: doc/glossary: add "application" to the glossary

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/glossary: add "application" to the glossary

Add an admittedly-stub definition of "application" to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 235d2e8b67b1700a55819a14eb26f00de86ba554)

Merge pull request #48452 from lxbsz/wip-57837

quincy: mds: wait unlink to finish to avoid conflict when creating same entries

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #50254 from zdover23/wip-doc-2023-02-24-backport-50250-to-quincy

quincy: doc/cephadm/host-management: add service spec link

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #50209 from cfsnyder/wip-57965-quincy-backport

quincy: rgw: add 'inline_data' zone placement info option

Merge pull request #50252 from zdover23/wip-doc-2023-02-24-backport-50179-to-quincy

quincy: doc/cephfs: describe conf opt "client quota df" in quota doc

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

rgw: add 'inline_data' zone placement info option

Adds a new RGW zone placement info option to control whether
an object's first data chunk is placed in the head object.
This allows admins to make a tradeoff between optimizing for
PUT/GET performance vs. DELETE performance for some cluster
configurations.

Fixes: https://tracker.ceph.com/issues/57965
Signed-off-by: Cory Snyder <csnyder@iland.com>
(cherry picked from commit 9052ca420f003af4b89b568c50bd5083333d3010)

Conflicts:
src/rgw/rgw_zone.cc
src/rgw/rgw_zone_types.h

Cherry-pick notes:
- Conflicts due to rgw_zone_types.h moving to rgw_zone.h

doc/cephadm/host-management: add service spec link

The old "(below)" text is not accurate, the service spec definition is
not in the same file at this point in time. This commit adds a ref link
to the actual service specification section.

Signed-off-by: thomas <me@thomasanderson.cloud>
(cherry picked from commit bb5837f049d9eb46f8d5c63c4dfec09fd94424f3)

doc/cephfs: describe conf opt "client quota df" in quota doc

The ceph config file option (from the client section) "client quota df"
is mentioned in "CephFS Client Capabilities" document but not in the
"CephFS Quota" document. Adding information about this option to this
document too would make it easier for CephFS users to discover,
understand and use this option.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit cd3f9575afb33bdb0a311a3f27801879a3254824)

Merge pull request #50247 from zdover23/wip-doc-2023-02-24-backport-50246-to-quincy

quincy: doc/glossary: add AWS/OpenStack bucket info

doc/glossary: add AWS/OpenStack bucket info

Add links to AWS's documentation of buckets, in accordance with Casey
Bodley's suggestions here:
https://github.com/ceph/ceph/pull/50221#discussion_r1115900879

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d16ed970e9ac76d6320f0ae26fa3ddd745dbd429)

test/cls/rgw: test that index entry is removed after racing deletes

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5ddde6eb3d6a9e387b904fa4986bfb8e2ccbef7a)

cls/rgw: remove index entry after cancelling last racing delete op

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 08efcc2536c34935a516bf850a9f909c0a3b3f52)

Conflicts:
src/cls/rgw/cls_rgw.cc CLS_LOG_BITX() -> CLS_LOG()

rgw: Fix segfault due to concurrent socket use at timeout

This commit fixes a potential segfault risk when
rgw timeout handler works on the socket in one
thread while it is concurrently used by another.
The details of the fix are:

1. Instead of calling socket close(), which resets
descriptor_data in boost::asio socket and risks
segfault due to concurrent use of the socket,
the timeout handler now calls cancel() to abort
all pending ops followed by shutdown() to disable
the underlying transport. The eventual closure of
the socket will be done in the socket destructor.

2. Expose the actual boost::asio socket via get_socket()
from Connection so that the timeout handler can call
cancel() and shutdown() on it, although the socket data
member is already accessible. It allows future expansion
that wants to hide the socket even though it renders the
existing close() less useful.

Fixes: https://tracker.ceph.com/issues/58670
Signed-off-by: Yixin Jin <yjin77@yahoo.ca>
(cherry picked from commit 6d5988cbdab48705414e311217c61d3c798085d8)

rgw/beast: StreamIO remembers connection errors for graceful shutdown

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 525899d4608203ea4236a816a78cf871cadceea6)

Merge pull request #50184 from cbodley/wip-58118

quincy: rgw: default-initialize delete_multi_obj_op_meta

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49902 from ifed01/wip-ifed-fix-cls-queue-list-qui

quincy: cls/queue: use larger read chunks in queue_list_entries

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49840 from k0ste/wip-58554-quincy

quincy: rgw/beast: fix interaction between keepalive and 100-continue

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49794 from dang/wip-58509-quincy

quincy: Fix chown to unlink

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49775 from liangmingyuanneo/wip-rgw-backport-49636-to-quincy

quincy: rgw: remove guard_reshard in bucket_index_read_olh_log

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49404 from lxbsz/wip-58251

quincy: mount: fix mount failure with old kernels

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49092 from cfsnyder/wip-54496-quincy

quincy: rgw: prevent spurious/lost notifications in the index completion thread

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49090 from cfsnyder/wip-54157-quincy

quincy: rgw: "reshard cancel" errors with "invalid argument"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48845 from nkshirsagar/wip-58006-quincy

quincy: mon: bail from handle_command() if _generate_command_map() fails

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #49967 from vshankar/tr-57729

quincy: pybind/mgr: check for empty metadata mgr_module:get_metadata()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #49831 from tserong/wip-58534-quincy

quincy: ceph.spec.in: Replace %usrmerged macro with regular version check

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #48719 from idryomov/gcc-13-quincy

quincy: common: fix build with GCC 13 (missing <cstdint> include)

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #48222 from mchangir/wip-55749-quincy

quincy: mgr/snap_schedule: remove subvol interface

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #48028 from tserong/wip-57391-quincy

quincy: ceph-dencoder: Add erasure_code to denc-mod-osd's target_link_libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #48466 from neesingh-rh/wip-57849-quincy

quincy: mgr/volumes: Add human-readable flag to volume info command

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #48252 from dparmar18/wip-57666-quincy

quincy: mds/Server: Do not abort MDS on unknown messages

Reviewed-by: Jos Collin <jcollin@redhat.com>

mgr/dashboard: fix constraints.txt

Replaces strict version matching with compatible version syntax.

Reference: https://peps.python.org/pep-0440/#version-specifiers

Fixes: https://tracker.ceph.com/issues/58827
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit d5f8927a5cd4bfd9870f02fea1c66b32191d1fba)

Merge pull request #50224 from zdover23/wip-doc-2023-02-23-backport-50221-to-quincy

quincy: doc/glossary: add "Bucket"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #50228 from zdover23/wip-doc-2023-02-23-backport-50226-to-quincy

quincy: doc: preen cephadm/troubleshooting.rst and radosgw/placement.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

quincy: qa/workunits/windows: backport rbd-wnbd tests

The quincy branch doesn't currently include the rbd-wnbd python
tests. For this reason, the ceph-build scripts are fetching a
certain version from the main branch and then run each test
individually [1].

One issue is that the fio test is failing on quincy and we aren't
getting any information. It seemed to be a time out, but it still
fails even after the timeout was increased.

By backporting the tests to quincy, the tests will be invoked
through the "run-tests.ps1" script, hopefully receiving more
information.

Note that we're skipping the live resize test since this feature
isn't available in Quincy.

This will also allow us to simplify the ceph-build script.

[1] https://github.com/ceph/ceph-build/blob/b5adddec2cb03059d0bc0b5d4ffa7aac7a248fc0/scripts/ceph-windows/run_tests#L70-L85

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

cmake: bump node version to 14

to update the angular to 13, we need a node version of either 12.20.x or
14.15.x or 16.10.x. But inorder to not break the angular 12 run for now
   we  need to choose the node which works for both 12 and 13. which is
14.15.x. So I went with that node version here.

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 5f95db33e912ca2b4a8a9d265d8cb537a6ceab03)
(cherry picked from commit 34dc871be599a44eb813b10fb49cf89a64a996c1)

doc: preen cephadm/troubleshooting.rst and radosgw/placement.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit d5bf55bcd6fcec43d2b728a2b8cd2f35b42e3b7e)

Merge pull request #50222 from zdover23/wip-doc-2023-02-23-backport-50220-to-quincy

quincy: doc/cephadm/troubleshooting: remove word repeat

doc/glossary: add "Bucket"

Add "Bucket" to the glossary. The definition added here is just a first
attempt, but a man's reach must exceed his grasp, or what's a heaven
for?

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ffbc5ea2b9347e2600df6aea9cdf895529fbf857)

doc/cephadm/troubleshooting: remove word repeat

The original text was "In case the cluster the last MGR of a cluster"

Signed-off-by: thomas <me@thomasanderson.cloud>
(cherry picked from commit 4e9eaf8b5d81b58ad49ed6ecbce38db98b985fc1)

Merge pull request #50218 from zdover23/wip-doc-2023-02-23-backport-50213-to-quincy

quincy: doc/rados: update OSD_BACKFILLFULL description

doc/rados: update OSD_BACKFILLFULL description

Document that this warning can happen even when the OSDs' limits
haven't hit the backfill_full_ratio.

Signed-off-by: Ponnuvel Palaniyappan <pponnuvel@gmail.com>
(cherry picked from commit 198a9e02cb934802fcb5d5d06b05e3662a4b5cab)

Merge pull request #50097 from sseshasa/wip-58708-quincy

quincy: osd: Restore defaults of mClock built-in profiles upon modification

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #50091 from petrutlucian94/wip-58702-quincy

quincy: os/memstore: Fix memory leak

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

rgw: initialize RGWIndexCompletionManager thread after related members

resolves valgrind issues about RGWIndexCompletionManager::process()
using uninitialized memory

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 112c2608750cf8db2c52c8e89f5897080501f6bd)

rgw: prevent spurious/lost notifications in the index completion thread

this was happening when asyn completions happened during reshard.
more information about testing:
https://gist.github.com/yuvalif/d526c0a3a4c5b245b9e951a6c5a10517

we also add more logs to the completion manager.
should allow finding unhandled completions due to reshards.

Fixes: https://tracker.ceph.com/issues/54435
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit e72b6510a402cbafa6c99475626f15d97fd00f86)

Conflicts:
- src/rgw/rgw_rados.cc

Cherry-pick notes:
- Conflicts within RGWIndexCompletionThread methods, which were removed by this commit

Merge pull request #48312 from k0ste/wip-57196-quincy

quincy: rgw: x-amz-date change breaks certain cases of aws sig v4.

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #49096 from cfsnyder/wip-57237-quincy

quincy: rgw/coroutine: check for null stack on wakeup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49094 from cfsnyder/wip-55227-quincy

quincy: rgw: adding BUCKET_REWRITE and OBJECT_REWRITE OPS to

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49088 from cfsnyder/wip-55506-quincy

quincy: rgw: RGWPutLC does not require Content-MD5

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50204 from zdover23/wip-doc-2023-02-22-backport-50203-to-quincy

quincy: doc/rgw: clarify multisite.rst top matter

doc/rgw: clarify multisite.rst top matter

Improve the pragmatics of the top matter of multisite.rst. Organize the
text into sections, where doing so makes the nature of multi-site
configurations clearer.

Co-authored-by: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit bbc229340f77ba8ec7fa967a59664caf3c07d030)

Merge pull request #49085 from cfsnyder/wip-57410-quincy

quincy: rgw: bucket list operation slow down in special scenario

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #48914 from yuvalif/wip-57561-quincy

quincy: tests: remove pubsub tests from multisite

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #48239 from trociny/wip-56406-quincy

quincy: rgw: splitting gc chains into smaller parts to prevent

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #50002 from smanjara/wip-57702-quincy

quincy: qa/suites/rgw: fix and update tempest and barbican tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49828 from ivancich/wip-58543-quincy

quincy: rgw: fix the problem of duplicate idx when bi list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49641 from dang/wip-58388-quincy

quincy: RGW - Make sure PostObj set bucket on s->object

Reviewed-by: Casey Bodley <cbodley@redhat.com>