Merge pull request #54844 from cbodley/wip-63775-reef

reef: add checking for rgw frontend init

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54843 from cbodley/wip-63773-reef

reef: rgw/iam: admin/system users ignore iam policy parsing errors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54794 from k0ste/wip-62412-reef

reef: rgw/auth: Fix the return code returned by AuthStrategy,

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54791 from k0ste/wip-63626-reef

reef: rgw: SignatureDoesNotMatch for certain RGW Admin Ops endpoints w/v4 auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54736 from yuvalif/wip-63701-reef

reef: rgw/kafka/amqp: fix race conditionn in async completion handlers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54674 from cbodley/wip-63628-reef

reef: rgw: object lock avoids 32-bit truncation of RetainUntilDate

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54328 from petrutlucian94/wip-63430-reef

reef: rgw: fix cloud-sync multi-tenancy scenario

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54129 from ivancich/wip-63247-reef

reef: rgw: only buckets with reshardable layouts need to be considered for resharding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

encoding: use concepts for chrono encoding

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b33e08657ecb7949505c6eae920c83e1e84096bb)

rgw: object lock uses 64-bit encoding for RetainUntilDate

the default encoding of ceph::real_time truncates seconds to uint32_t,
so stores the wrong timestamp for object lock enforcement

Fixes: https://tracker.ceph.com/issues/63537
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1fd4309fbbebf0d3e67aa2800d5fb3c7de19dcc7)

Merge pull request #55128 from zdover23/wip-doc-2024-01-10-backport-55106-to-reef

reef: doc/rgw: edit admin.rst - rate limit management

doc/rgw: edit admin.rst - rate limit management

Edit the "Rate Limit Management" section in doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 47fabe757cff630aaf23a85d268b7406057244e2)

Merge pull request #55091 from cbodley/wip-63964-reef

reef: make-dist: don't use --continue option for wget

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #55084 from petrutlucian94/wip-63955-reef

reef: win32_deps_build.sh: change Boost URL

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

make-dist: don't use --continue option for wget

the boost jfrog mirror is broken and returns an HTML error page instead
of the archive. the file size of this page is 11534 bytes

when download_from() retries the download from download.ceph.com, the -c
option tells it to resume the download of the existing file. the
resulting boost_1_82_0.tar.bz2 ends up with the correct total file size
of 121325129 bytes, but the first 11534 bytes still correspond to the
HTML from jfrog. that causes the sha256sum mismatch

remove the -c option so that wget fetches the archive in its entirety

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ba7d6e59e8750867059c29a33c22bcb3e358b65)

Merge pull request #54818 from baergj/wip-63746-reef

reef: librbd: Append one journal event per image request

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54807 from trociny/wip-63733-reef

reef:  pybind/rbd: fix compilation with cython3

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54769 from ajarr/wip-63715-reef

reef: qa/workunits/rbd/cli_generic.sh: narrow race window when checking that rbd_support module command fails after blocklisting the module's client

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #55082 from zdover23/wip-doc-2024-01-08-backport-55075-to-reef

reef: doc/radosgw: edit admin.rst - quota

win32_deps_build.sh: change Boost URL

The Boost download URL is broken again [1], we'll switch to a
working mirror.

[1] https://github.com/boostorg/boost/issues/842

Fixes: https://tracker.ceph.com/issues/63952
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 594d1e5e83e3d4c318ab1dd79660670a0852f7e6)

doc/radosgw: edit admin.rst - quota

Edit the "Quota Management" section (all of it) in
doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 22fce3c7a55591f6755355751157bc7231776137)

Merge pull request #55071 from zdover23/wip-doc-2024-01-06-backport-55069-to-reef

reef: doc: update rgw admin api req params for get user info

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55073 from zdover23/wip-doc-2024-10-06-backport-55065-to-reef

reef: doc/radosrgw: edit admin.rst

doc/radosrgw: edit admin.rst

Edit "Add / Remove Admin Capabilities" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1f0ff27898dfd06a17b18fadd52f68dfb73ccb94)

doc: update rgw admin api req params for get user info

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit e6c58dcd614f71efa4de2f781b8425c480af285e)

Merge pull request #55064 from zdover23/wip-doc-2024-01-05-backport-55053-to-reef

doc/releases: edit reef.rst

doc/releases: edit reef.rst

Make minor corrections to doc/releases/reef.rst. These corrections were
suggested by Anthony D'Atri in https://github.com/ceph/ceph/pull/55049.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d02178fdbd5707c6469fdd678f42a4575a3c10b4)

Merge pull request #55055 from zdover23/wip-doc-2024-01-04-backport-55048-to-reef

reef: doc/radosgw: edit "Add/Remove a Key"

doc/radosgw: edit "Add/Remove a Key"

Edit the section "Add/Remove a Key" in doc/radosgw/admin.rst. Each
operation (e.g. "Adding an S3 key pair for a user", "Removing an S3 key
pair for a user") now has its own subsection. This increased granularity
should make it easier in the future to link to each of these specific
operations, if needed.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f62e93cbe73cd8f624a6c99497051c1a0aaf3ab6)

Merge pull request #55049 from zdover23/wip-doc-2024-01-03-backport-reef-releases-page-from-latest-to-reef

reef: doc/releases: specify dashboard improvements

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/releases: specify dashboard improvements

...and address other nits

Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit d5ab720dc54e19904267fbde5ff54260eff9bd60)
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55044 from zdover23/wip-doc-2024-01-02-backport-55038-to-reef

reef: doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 6601e7f38c0b2c1326d40f3bb0599069e38421ff)

Merge pull request #55034 from zdover23/wip-doc-2024-01-01-backport-55030-to-reef

reef: doc/radosgw: edit "remove a subuser"

Merge pull request #55032 from zdover23/wip-doc-2024-01-01-backport-55031-to-reef

reef: doc/mgr: document lack of MSWin NFS 4.x support

doc/radosgw: edit "remove a subuser"

Edit the English language in the section "Remove a Subuser" in
doc/radosgw/admin.rst. This commit is made in response to Matt
Benjamin's request for improvement of this section
(https://github.com/ceph/ceph/pull/55028#discussion_r1438599833).

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8fc3c488861887490f6ec6bac8f49770664b49a6)

doc/mgr: document lack of MSWin NFS 4.x support

Document that no version of Microsoft Windows supports mounting an NFS
v4.x export natively, as of the time of this commit.

This commit has been made in response to an anonymous note left on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 03656c02a70d4ab719cbcf605517f8a2e9e81117)

Merge pull request #55027 from zdover23/wip-doc-2023-12-30-backport-55023-to-reef

reef: doc/radosgw: edit sections

doc/radosgw: edit sections

Edit the following sections in doc/radosgw/admin.rst:

* Remove a User
* Remove a Subuser

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8631dc58f7144b32febadd9072c6691456ac8f28)

Merge pull request #55020 from zdover23/wip-doc-2023-12-28-backport-55011-to-reef

reef: doc/radosgw: admin.rst - edit "Create a Subuser"

doc/radosgw: admin.rst - edit "Create a Subuser"

Edit the section "Create a Subuser" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 3c7a6c41df540f935b0b0b2c4fdc7375327a7662)

Merge pull request #55017 from zdover23/wip-doc-2023-12-28-backport-55016-to-reef

reef: doc/radosgw: admin.rst - edit sections

doc/radosgw: admin.rst - edit sections

Edit the following sections in doc/radosgw/admin.rst:

- Get User Info
- Modify User Info
- User Suspend/Enable (I split this into two sections)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 21ebd2e855a4dd2d2d1244bb3688db0c6613dc30)

Merge pull request #55004 from zdover23/wip-doc-2023-12-26-backport-55003-to-reef

reef: doc/radosgw: admin.rst - edit "Create a User"

doc/radosgw: admin.rst - edit "Create a User"

Edit the section "Create a User" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 46139c8fa9d4ae0e493bca17ba1199e3a67c39f8)

Merge pull request #55000 from zdover23/wip-doc-2023-12-25-backport-54997-to-reef

reef: doc/radosgw: edit admin.rst 1 of x

doc/radosgw: edit admin.rst 1 of x

Edit doc/radosgw/admin.rst to the end of the section "User Management".
Correct grammar (mostly awkward prepositions) and tidy up weird clauses.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 819b2974e64548fcf66423cca94a785887fd7d07)

Merge pull request #54995 from zdover23/wip-doc-2023-12-23-backport-54989-to-reef

reef: doc/radosgw: list supported plugins-compression.rst

doc/radosgw: list supported plugins-compression.rst

List supported compression plugins in doc/radosgw/compression.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 64b0eb7d050755aab58a4ec7bcb218f9ced19921)

Merge pull request #54985 from zdover23/wip-doc-2023-12-21-backport-54970-to-reef

reef: doc/radosgw: edit compression.rst

doc/radosgw: edit compression.rst

Improve the grammar and simplify the sentence structure of
doc/radosgw/compression.rst. This commit is made in anticipation of a
near-future commit that will list the compression algorithms available
to users of Ceph.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 84c5d2c828c2fbd70bdeadedd341ca42ddb1c20c)

Merge pull request #53340 from trociny/wip-62724-reef

reef: mds: optionally forbid to use standby for another fs as last resort

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54961 from zdover23/wip-doc-2023-12-19-backport-54960-to-reef

reef: doc/install: update "update submodules"

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/install: update "update submodules"

Remove misleading material that would give readers the wrong idea about
when stale submodules are present. This commit is made in response to
information given to me by Ilya Dryomov here: https://github.com/ceph/ceph/pull/54929#issuecomment-1859237986.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f5c87693abcfc9a6a3869a3f52549384807fba9d)

Merge pull request #54532 from ceph/reef-release

v18.2.1

Merge pull request #54938 from zdover23/wip-doc-2023-12-18-backport-54929-to-reef

reef: docs: Add information about OpenNebula integration

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

docs: Add information about OpenNebula integration

- Exclude doc build output from git
- Fix missing doc build dependency
- Also includes some involuntary automatically persistent linting by vscode

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: Daniel Clavijo <dclavijo@opennebula.io>
(cherry picked from commit ee2ee31d3f72a2ee63f9d28bbaf6045657f43ee6)

Merge pull request #54933 from zdover23/wip-doc-2023-12-17-backport-52354-to-reef

reef: doc/radosgw/admin.rst: use underscores in config var names

doc/radosgw/admin.rst: use underscores in config var names

Following the current policy, config var names in `ceph.conf` etc. should use underscores instead of spaces.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
(cherry picked from commit cd323cb664a125834e41b77f711eac898e548382)

Merge pull request #54905 from zdover23/wip-doc-2023-12-15-backport-54893-to-reef

reef: doc/radosgw: format commands in role.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: format commands in role.rst

Format approximately one-hundred lines of doc/radosgw/role.rst to
include proper command prompts. I also made one small English usage
improvement.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 31d72b8ecc1b75b7996027418614c6e2e6a1d0e7)

mds: encode `bal_rank_mask` using a higher (next) version

This is required for a reef client to work with a higher revision
MDS, since essentially, this happens:

reef(client):

if (version >=17) {
    decode(bal_rank_mask, p);
}

and higher-revision MDS (say, upcoming squid):

version = 17
encode(version, bl);
...
...
encode(max_xattr_size, bl);
encode(bal_rank_mask, bl);

The client incorrectly decodes max_xattr_size (type: uint64_t) into
bal_rank_mask (type: string).

This situation ended up due to a couple of reasons:

* the kclient patchset hanlding `max_xattr_size` was merged early on
  and another MDS side change that bumped the MDSMap encoding version
  to 17 got merged in the midst (PR #43284). Details in comment:

        https://github.com/ceph/ceph/pull/46357#issuecomment-1293556227

* The reef backport for PR #46357 got delayed (and, reef branched out).

Which means reef(18.2.0) user-space clients are broken with higher version
MDSs.

Fixes: https://tracker.ceph.com/issues/63713
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 36ee8e7ed365933b264321c15f77c0ed1e352d8c)

Merge pull request #54886 from zdover23/wip-doc-2023-12-13-backport-54875-to-reef

reef: doc/radosgw: format "Create a Role"

doc/radosgw: format "Create a Role"

Properly format the section "Create a Role" in the file
doc/radosgw/role.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit fa6ad94f3afc6d11c3ed243d66c96a666e973bde)

Merge pull request #54865 from batrick/wip-63782-reef

reef: qa: add support/qa for cephfs-shell on CentOS 9 / RHEL9

Reviewed-by: Venky Shankar <vshankar@redhat.com>

18.2.1

Signed-off-by: Ceph Release Team <ceph-maintainers@ceph.io>

reef: qa: only test cephfs-shell with ubuntu or centos 9.stream

Because reef is tested on RHEL/centos 8, we need to remove those jobs from
testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

PendingReleaseNotes: announce cephfs-shell avail. on rhel9

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 04a54fbd8297e1027e1a8d0259ff406b41a2ae4c)

qa: test fs:shell on all distros

This is a simple sub-suite that has one job. Always schedule on all supported distros.

Fixes: https://tracker.ceph.com/issues/43393
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 215d96c1d1533836d313be54864c1b0525440b4c)

qa: add cephfs-shell to installed rpm packages

For CephFS QA.

Fixes: https://tracker.ceph.com/issues/43393
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit b12bdfeee3647b495db87be5f3d0abcb3ca77034)

ceph.spec.in: enable support for cephfs-shell by default via EPEL9

The required packages [1,2] are now at the right version and readily available.

[1] https://packages.fedoraproject.org/pkgs/python-cmd2/python3-cmd2/epel-9.html
[2] https://packages.fedoraproject.org/pkgs/python-colorama/python3-colorama/epel-9.html

Fixes: https://tracker.ceph.com/issues/43393
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d63d96f8bcda08990bc3be28c9a835d6c836f0a3)

ceph-volume: fix mpath device support

commit [1] broke mpath devices support in `disk.is_device()`

[1] https://github.com/ceph/ceph/commit/4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc

Fixes: https://tracker.ceph.com/issues/62722
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 28a12f8ef703a7349d9f2de40251c22e018fc475)
(cherry picked from commit 1e7223281fa044c9653633e305c0b344e4c9b3a4)

test/librbd: actually alternate overlaps in DiscardWithPruneWriteOverlap

Make sense of the inner loop in write_thread.  The crash on
"it != m_events.end()" assert reproduces even faster this way.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit d790b57b4bd3868cf12941c565b6906534e19f91)

test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap

For reasons that I think no longer apply today, set_val() and
set_val_or_die() refuse to set "type: str" config options that aren't
marked as "can be changed at runtime" -- set_val() returns an error and
set_val_or_die() terminates the process.  What is and isn't marked as
"can be changed at runtime" seems to be pretty much random both within
and outside of RBD, so let's just refactor how config is set here.

While at it, I realized that reproducer config is underspecified:

- for rbd_cache_policy and rbd_cache_writethrough_until_flush settings
  to matter, rbd_cache must be set to true and rbd_cache_max_dirty must
  be set to a positive number

- order should be set explicitly, because rbd_default_order can be as
  low as 12 (for 4096-byte objects), interfering with the logic of the
  test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit de397f7588897fb0a3f15dcddf660c8e569b1e4e)

Merge pull request #53539 from guits/wip-62893-reef

reef: ceph-volume: fix mpath device support

Merge pull request #54854 from zdover23/wip-doc-2023-12-11-backport-54845-to-reef

reef: doc/radosgw: edit front matter - role.rst

doc/radosgw: edit front matter - role.rst

Clarify the front matter in doc/radosgw/role.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 9d10361961e01c4d60f600f6e7ed106d6203e554)

Merge pull request #54849 from zdover23/wip-doc-2023-12-10-backport-54846-to-reef

reef: doc/radosgw: format POST statements

doc/radosgw: format POST statements

Format the POST methods so that they appear in the rendered text as
examples of POST API calls and not as plain old unformatted text, which
is how they looked before this commit. The content of these API calls
remains to be tested and confirmed to work, but this is a first step.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1c0901cf8503ca8486dee8875ce85adaa464b127)

add checking result for rgw  frontend init
fixes: https://tracker.ceph.com/issues/63644

Signed-off-by: zhipeng li <qiuxinyidian@gmail.com>
(cherry picked from commit 5189c2e457bbbc0c7f430d3fa810b59896c8bb44)

rgw/iam: admin/system users ignore iam policy parsing errors

allow admin users to repair broken iam policies that would otherwise
reject access on PutBucketPolicy/DeleteBucketPolicy requests

Fixes: https://tracker.ceph.com/issues/63485
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d962dbeb9040a07d0a2c9db8ab9a2cf1fc320cc7)

Merge pull request #54833 from zdover23/wip-doc-2023-12-08-backport-54751-to-reef

reef: doc/radosgw: add gateway starting command

doc/radosgw: add gateway starting command

Add a command that properly starts (or restarts) the RADOS gateway after
RGW settings have been changed. This commit has been added in response
to an issue reported anonymously on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ec7c515490c2ade44d886e423a6601c7ef0cf5e8)

os/bluestore: get rid off resulting lba alignment in allocators

Fixes: https://tracker.ceph.com/issues/62815
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
(cherry picked from commit b0cb41a761dd7c32bde8e29eb12d44519a2988a5)
(cherry picked from commit a585ac27e151e2ffe1e0a347d1a09fb4c752e809)

os/bluestore: adjust and validate bluefs_shared_alloc_size

Make sure it's in-sync (meaning it's higher or equal and properly aligned)
with bluestore_min_alloc_size into account

Fixes: https://tracker.ceph.com/issues/63618
Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
(cherry picked from commit 334e7fa0879512790e89d1f4d30cd6aaa81f28fa)
(cherry picked from commit f7b8b4d14b2baba0aef5563f4e0abe9d780d290a)

Merge pull request #54772 from ifed01/wip-ifed-no-alloc-lba-align-reef

reef: os/bluestore: get rid off resulting lba alignment in allocators

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #54822 from zdover23/wip-doc-2023-12-07-backport-54809-to-reef

reef: doc: add supported file types in cephfs-mirroring.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: add supported file types in cephfs-mirroring.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 3d44e5dfdc83e33167ee2b63caf7c4fb2d34f6ba)

test/librbd: Add workaround for a journaling deadlock that this test exposes

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
(cherry picked from commit 7436b4c99034dd19b2ff1e759603779061cbfdb6)

librbd: Append one journal event per image request

In the case where an image request is split across multiple object
extents and journaling is enabled, multiple journal events are appended.
Prior to this change, all object requests would wait for the last
journal event to complete, since journal events complete in order and
thus the last one completing implies that all prior journal events were
safe at that point.

The issue with this is that there's nothing stopping that last journal
event from being cleaned up before all object requests have stopped
referring to it. Thus, it's entirely possible for the following sequence
to occur:
1. An image request gets split into two image extents and two object
   requests. Journal events are appended (one per image extent).
2. The first object request gets delayed due to an overlap, but the
   second object request gets submitted and starts waiting on the last
   journal event (which also causes a C_CommitIOEvent to be instantiated
   against that journal event).
3. Journaling completes, and the C_CommitIOEvent fires. The
   C_CommitIOEvent covers the entire range of data that was journaled in
   this event, and so the event is cleaned up.
4. The first object request from above is allowed to make progress; it
   tries to wait for the journal event that was just cleaned up which
   causes the assert in wait_event() to fire.

As far as I can tell, this is only possible on the discard path today,
and only recently. Up until 21a26a752843295ff946d1543c2f5f9fac764593
(librbd: Fix local rbd mirror journals growing forever), m_image_extents
always contained a single extent for all I/O types; this commit changed
the discard path so that if discard granularity changed the discard
request, m_image_extents would be repopulated, and if the request
happened to cross objects then there would be multiple m_image_extents.

It appears that the intent here was that there should be one journal
event per image request and the pending_extents kept track of what had
completed thus far. This commit restores that 1:1 relationship.

Fixes: https://tracker.ceph.com/issues/63422
Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
(cherry picked from commit 4a8fa2da72fe64109073fddca0d4cfd99aeb9c77)

librbd: Change append_io_events() to take Extents

An upcoming commits will use this to change how multi-extent image
requests are appended to the journal.

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
(cherry picked from commit 9fae091ea22c627cb18cc68055afa7acf634c1b7)

librbd: Eliminate unused m_synchronous from AbstractImageWriteRequest

This has been unused since its introduction years ago and so isn't worth
keeping.

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
(cherry picked from commit a0ccd8bb3ffbecb3313c0cdd7d3e6091fc2a9721)

test/librbd: Add a stress test that reproduces a crash during discard journaling

See the comments in DiscardWithPruneWriteOverlap for details.

Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
(cherry picked from commit e8d54e3c9faeddedc2890294556cd66095b83be4)

Merge pull request #54805 from zdover23/wip-doc-2023-12-06-backport-54790-to-reef

reef: doc/radosgw: update link in rgw-cache.rst

pybind/rbd: make cdef functions not propagate exceptions by default

Setting legacy_implicit_noexcept compiler directive to True will cause
Cython 3.0 to have the same semantics as Cython 0.x.

Fixes: https://tracker.ceph.com/issues/62140
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit e3156050d0ce9b504ee40d30e98f49a860b7dde5)

test/pybind/rbd: test callbacks raising exceptions

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 55b3be5234f1c670b0c7d3f3a1584af2573d9288)

doc/radosgw: update link in rgw-cache.rst

Update link in doc/radosgw/rgw-cache.rst. The link updated here is a
link to all the Nginx configuration files. The old link was broken. This
update comes to us from an anonymous report on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit e80d560d97bd44f9a308916a099bf9391813d95b)

rgw/auth: Fix the return code returned by AuthStrategy,
in case of LocalEngine authentication, when the previous
auth engine is Not Applicable. The error code now returned
is the one returned by LocalEngine.

fixes: https://tracker.ceph.com/issues/61916

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit 6f7ef8620e52be776ef633bb962a4309996529df)

SignatureDoesNotMatch for certain RGW Admin Ops endpoints when using v4 auth

https://tracker.ceph.com/issues/62105
Change from std::map<> to std::multimap<> to allow for duplicates
rgwadmin submits duplicates in a very few cases, so we need to handle them.

Signed-off-by: David.Hall <mcfadden7586@gmail.com>
(cherry picked from commit 3758f6e7433c58b9e62ae35184659cffabdbd133)

Merge pull request #54753 from zdover23/wip-doc-2023-12-02-backport-54752-to-reef

reef: doc/radosgw: fix formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>