mgr/dashboard: Fixes multisite topology page breadcrumb

The multi-site topology page just says object in breadcrumb.
The fix adds the missing "multi-site" breadcrumb.

Fixes https://tracker.ceph.com/issues/63635

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>
(cherry picked from commit f75a9da2871113a156092ff28f29b9d12f465975)

Merge pull request #55104 from rkachach/fix_issue_63957_63958

reef: backport of fixes for 63678 and 63694

Merge pull request #55194 from zdover23/wip-doc-2024-01-16-backport-55169-to-reef

reef: docs/radosgw: edit admin.rst "enable/disable user rate limit"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55191 from rhcs-dashboard/reef-docs-build-fix

reef: admin/doc-requirements: bump Sphinx to 5.0.2

Reviewed-by: Matan-B <NOT@FOUND>

Merge pull request #55124 from rhcs-dashboard/wip-59470-reef

reef: mgr/dashboard: fix e2e failure related to landing page

docs/radosgw: edit admin.rst "enable/disable user rate limit"

Edit "Enable/Disable User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1cd996b6fe2e3c923001a69772c7b7635d058f2b)

admin/doc-requirements: bump Sphinx to 5.0.2

```
Running Sphinx v4.5.0

Sphinx version error:
The sphinxcontrib.applehelp extension used by this project needs at least Sphinx v5.0; it therefore cannot be built with this version.
```

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit a916feeee757e4e196967944feeb1e9b1f92c398)

Merge pull request #55163 from zdover23/wip-doc-2024-01-13-backport-54530-to-reef

reef: ceph-menv:fix typo in README

ceph-menv:fix typo in README

Signed-off-by: yu.wang <wangyu@wyu.space>
(cherry picked from commit 4efd9317b83eca48c1a92724e7ebe304650fbc04)

Merge pull request #55157 from zdover23/wip-doc-2024-01-12-backport-55146-to-reef

reef: doc/radosgw: edit admin.rst "get user rate limit"

doc/radosgw: edit admin.rst "get user rate limit"

Edit "Get User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1c561e2beac67c85c40f2adf387c42aba8175ecb)

Merge pull request #55150 from zdover23/wip-doc-2024-01-12-backport-55142-to-reef

reef: doc/radosgw - edit admin.rst "set user rate limit"

doc/radosgw - edit admin.rst "set user rate limit"

Edit "Set User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8660b4edc71113d5d2d51438c9fd6f388b35a214)

Merge pull request #54296 from yuvalif/wip-59498-reef

reef: rgw/lua: fix CopyFrom crash

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54127 from BBoozmen/wip-63255-reef

reef: RGW/STS: when generating keys, take the trailing null character into account

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54862 from idryomov/wip-63654-reef

reef: librbd: fix regressions in ObjectListSnapsRequest

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #54852 from mkogan1/wip-63753-reef

reef: rgw: d3n: fix valgrind reported leak related to libaio worker threads

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54844 from cbodley/wip-63775-reef

reef: add checking for rgw frontend init

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54843 from cbodley/wip-63773-reef

reef: rgw/iam: admin/system users ignore iam policy parsing errors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54794 from k0ste/wip-62412-reef

reef: rgw/auth: Fix the return code returned by AuthStrategy,

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54791 from k0ste/wip-63626-reef

reef: rgw: SignatureDoesNotMatch for certain RGW Admin Ops endpoints w/v4 auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54736 from yuvalif/wip-63701-reef

reef: rgw/kafka/amqp: fix race conditionn in async completion handlers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54674 from cbodley/wip-63628-reef

reef: rgw: object lock avoids 32-bit truncation of RetainUntilDate

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54328 from petrutlucian94/wip-63430-reef

reef: rgw: fix cloud-sync multi-tenancy scenario

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54129 from ivancich/wip-63247-reef

reef: rgw: only buckets with reshardable layouts need to be considered for resharding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

encoding: use concepts for chrono encoding

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b33e08657ecb7949505c6eae920c83e1e84096bb)

rgw: object lock uses 64-bit encoding for RetainUntilDate

the default encoding of ceph::real_time truncates seconds to uint32_t,
so stores the wrong timestamp for object lock enforcement

Fixes: https://tracker.ceph.com/issues/63537
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1fd4309fbbebf0d3e67aa2800d5fb3c7de19dcc7)

Merge pull request #55128 from zdover23/wip-doc-2024-01-10-backport-55106-to-reef

reef: doc/rgw: edit admin.rst - rate limit management

doc/rgw: edit admin.rst - rate limit management

Edit the "Rate Limit Management" section in doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 47fabe757cff630aaf23a85d268b7406057244e2)

mgr/dashboard: fix e2e failure related to landing page

Fixes: https://tracker.ceph.com/issues/59142
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
(cherry picked from commit 6ec48eeb261bc8aceb1bfa3ee3732acfb0974478)

Merge pull request #55091 from cbodley/wip-63964-reef

reef: make-dist: don't use --continue option for wget

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #55084 from petrutlucian94/wip-63955-reef

reef: win32_deps_build.sh: change Boost URL

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

reef: backport of fixes for 63678 and 63694
Fixes: https://tracker.ceph.com/issues/63957
Fixes: https://tracker.ceph.com/issues/63958
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

make-dist: don't use --continue option for wget

the boost jfrog mirror is broken and returns an HTML error page instead
of the archive. the file size of this page is 11534 bytes

when download_from() retries the download from download.ceph.com, the -c
option tells it to resume the download of the existing file. the
resulting boost_1_82_0.tar.bz2 ends up with the correct total file size
of 121325129 bytes, but the first 11534 bytes still correspond to the
HTML from jfrog. that causes the sha256sum mismatch

remove the -c option so that wget fetches the archive in its entirety

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ba7d6e59e8750867059c29a33c22bcb3e358b65)

Merge pull request #54818 from baergj/wip-63746-reef

reef: librbd: Append one journal event per image request

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54807 from trociny/wip-63733-reef

reef:  pybind/rbd: fix compilation with cython3

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #54769 from ajarr/wip-63715-reef

reef: qa/workunits/rbd/cli_generic.sh: narrow race window when checking that rbd_support module command fails after blocklisting the module's client

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #55082 from zdover23/wip-doc-2024-01-08-backport-55075-to-reef

reef: doc/radosgw: edit admin.rst - quota

win32_deps_build.sh: change Boost URL

The Boost download URL is broken again [1], we'll switch to a
working mirror.

[1] https://github.com/boostorg/boost/issues/842

Fixes: https://tracker.ceph.com/issues/63952
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
(cherry picked from commit 594d1e5e83e3d4c318ab1dd79660670a0852f7e6)

doc/radosgw: edit admin.rst - quota

Edit the "Quota Management" section (all of it) in
doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 22fce3c7a55591f6755355751157bc7231776137)

Merge pull request #55071 from zdover23/wip-doc-2024-01-06-backport-55069-to-reef

reef: doc: update rgw admin api req params for get user info

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #55073 from zdover23/wip-doc-2024-10-06-backport-55065-to-reef

reef: doc/radosrgw: edit admin.rst

doc/radosrgw: edit admin.rst

Edit "Add / Remove Admin Capabilities" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 1f0ff27898dfd06a17b18fadd52f68dfb73ccb94)

doc: update rgw admin api req params for get user info

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit e6c58dcd614f71efa4de2f781b8425c480af285e)

Merge pull request #55064 from zdover23/wip-doc-2024-01-05-backport-55053-to-reef

doc/releases: edit reef.rst

doc/releases: edit reef.rst

Make minor corrections to doc/releases/reef.rst. These corrections were
suggested by Anthony D'Atri in https://github.com/ceph/ceph/pull/55049.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d02178fdbd5707c6469fdd678f42a4575a3c10b4)

Merge pull request #55055 from zdover23/wip-doc-2024-01-04-backport-55048-to-reef

reef: doc/radosgw: edit "Add/Remove a Key"

doc/radosgw: edit "Add/Remove a Key"

Edit the section "Add/Remove a Key" in doc/radosgw/admin.rst. Each
operation (e.g. "Adding an S3 key pair for a user", "Removing an S3 key
pair for a user") now has its own subsection. This increased granularity
should make it easier in the future to link to each of these specific
operations, if needed.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f62e93cbe73cd8f624a6c99497051c1a0aaf3ab6)

Merge pull request #55049 from zdover23/wip-doc-2024-01-03-backport-reef-releases-page-from-latest-to-reef

reef: doc/releases: specify dashboard improvements

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/releases: specify dashboard improvements

...and address other nits

Signed-off-by: Laura Flores <lflores@ibm.com>
(cherry picked from commit d5ab720dc54e19904267fbde5ff54260eff9bd60)
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55044 from zdover23/wip-doc-2024-01-02-backport-55038-to-reef

reef: doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

doc/cephfs: Update disaster-recovery-experts.rst to mention Slack

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>
(cherry picked from commit 6601e7f38c0b2c1326d40f3bb0599069e38421ff)

Merge pull request #55034 from zdover23/wip-doc-2024-01-01-backport-55030-to-reef

reef: doc/radosgw: edit "remove a subuser"

Merge pull request #55032 from zdover23/wip-doc-2024-01-01-backport-55031-to-reef

reef: doc/mgr: document lack of MSWin NFS 4.x support

doc/radosgw: edit "remove a subuser"

Edit the English language in the section "Remove a Subuser" in
doc/radosgw/admin.rst. This commit is made in response to Matt
Benjamin's request for improvement of this section
(https://github.com/ceph/ceph/pull/55028#discussion_r1438599833).

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8fc3c488861887490f6ec6bac8f49770664b49a6)

doc/mgr: document lack of MSWin NFS 4.x support

Document that no version of Microsoft Windows supports mounting an NFS
v4.x export natively, as of the time of this commit.

This commit has been made in response to an anonymous note left on
https://pad.ceph.com/p/Report_Documentation_Bugs.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 03656c02a70d4ab719cbcf605517f8a2e9e81117)

Merge pull request #55027 from zdover23/wip-doc-2023-12-30-backport-55023-to-reef

reef: doc/radosgw: edit sections

doc/radosgw: edit sections

Edit the following sections in doc/radosgw/admin.rst:

* Remove a User
* Remove a Subuser

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8631dc58f7144b32febadd9072c6691456ac8f28)

Merge pull request #55020 from zdover23/wip-doc-2023-12-28-backport-55011-to-reef

reef: doc/radosgw: admin.rst - edit "Create a Subuser"

doc/radosgw: admin.rst - edit "Create a Subuser"

Edit the section "Create a Subuser" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 3c7a6c41df540f935b0b0b2c4fdc7375327a7662)

Merge pull request #55017 from zdover23/wip-doc-2023-12-28-backport-55016-to-reef

reef: doc/radosgw: admin.rst - edit sections

doc/radosgw: admin.rst - edit sections

Edit the following sections in doc/radosgw/admin.rst:

- Get User Info
- Modify User Info
- User Suspend/Enable (I split this into two sections)

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 21ebd2e855a4dd2d2d1244bb3688db0c6613dc30)

Merge pull request #55004 from zdover23/wip-doc-2023-12-26-backport-55003-to-reef

reef: doc/radosgw: admin.rst - edit "Create a User"

doc/radosgw: admin.rst - edit "Create a User"

Edit the section "Create a User" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 46139c8fa9d4ae0e493bca17ba1199e3a67c39f8)

Merge pull request #55000 from zdover23/wip-doc-2023-12-25-backport-54997-to-reef

reef: doc/radosgw: edit admin.rst 1 of x

doc/radosgw: edit admin.rst 1 of x

Edit doc/radosgw/admin.rst to the end of the section "User Management".
Correct grammar (mostly awkward prepositions) and tidy up weird clauses.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 819b2974e64548fcf66423cca94a785887fd7d07)

Merge pull request #54995 from zdover23/wip-doc-2023-12-23-backport-54989-to-reef

reef: doc/radosgw: list supported plugins-compression.rst

doc/radosgw: list supported plugins-compression.rst

List supported compression plugins in doc/radosgw/compression.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 64b0eb7d050755aab58a4ec7bcb218f9ced19921)

Merge pull request #54985 from zdover23/wip-doc-2023-12-21-backport-54970-to-reef

reef: doc/radosgw: edit compression.rst

doc/radosgw: edit compression.rst

Improve the grammar and simplify the sentence structure of
doc/radosgw/compression.rst. This commit is made in anticipation of a
near-future commit that will list the compression algorithms available
to users of Ceph.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 84c5d2c828c2fbd70bdeadedd341ca42ddb1c20c)

Merge pull request #53340 from trociny/wip-62724-reef

reef: mds: optionally forbid to use standby for another fs as last resort

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54961 from zdover23/wip-doc-2023-12-19-backport-54960-to-reef

reef: doc/install: update "update submodules"

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/install: update "update submodules"

Remove misleading material that would give readers the wrong idea about
when stale submodules are present. This commit is made in response to
information given to me by Ilya Dryomov here: https://github.com/ceph/ceph/pull/54929#issuecomment-1859237986.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit f5c87693abcfc9a6a3869a3f52549384807fba9d)

Merge pull request #54532 from ceph/reef-release

v18.2.1

Merge pull request #54938 from zdover23/wip-doc-2023-12-18-backport-54929-to-reef

reef: docs: Add information about OpenNebula integration

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

docs: Add information about OpenNebula integration

- Exclude doc build output from git
- Fix missing doc build dependency
- Also includes some involuntary automatically persistent linting by vscode

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: Daniel Clavijo <dclavijo@opennebula.io>
(cherry picked from commit ee2ee31d3f72a2ee63f9d28bbaf6045657f43ee6)

Merge pull request #54933 from zdover23/wip-doc-2023-12-17-backport-52354-to-reef

reef: doc/radosgw/admin.rst: use underscores in config var names

doc/radosgw/admin.rst: use underscores in config var names

Following the current policy, config var names in `ceph.conf` etc. should use underscores instead of spaces.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
(cherry picked from commit cd323cb664a125834e41b77f711eac898e548382)

Merge pull request #54905 from zdover23/wip-doc-2023-12-15-backport-54893-to-reef

reef: doc/radosgw: format commands in role.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: format commands in role.rst

Format approximately one-hundred lines of doc/radosgw/role.rst to
include proper command prompts. I also made one small English usage
improvement.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 31d72b8ecc1b75b7996027418614c6e2e6a1d0e7)

mds: encode `bal_rank_mask` using a higher (next) version

This is required for a reef client to work with a higher revision
MDS, since essentially, this happens:

reef(client):

if (version >=17) {
    decode(bal_rank_mask, p);
}

and higher-revision MDS (say, upcoming squid):

version = 17
encode(version, bl);
...
...
encode(max_xattr_size, bl);
encode(bal_rank_mask, bl);

The client incorrectly decodes max_xattr_size (type: uint64_t) into
bal_rank_mask (type: string).

This situation ended up due to a couple of reasons:

* the kclient patchset hanlding `max_xattr_size` was merged early on
  and another MDS side change that bumped the MDSMap encoding version
  to 17 got merged in the midst (PR #43284). Details in comment:

        https://github.com/ceph/ceph/pull/46357#issuecomment-1293556227

* The reef backport for PR #46357 got delayed (and, reef branched out).

Which means reef(18.2.0) user-space clients are broken with higher version
MDSs.

Fixes: https://tracker.ceph.com/issues/63713
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 36ee8e7ed365933b264321c15f77c0ed1e352d8c)

Merge pull request #54886 from zdover23/wip-doc-2023-12-13-backport-54875-to-reef

reef: doc/radosgw: format "Create a Role"

doc/radosgw: format "Create a Role"

Properly format the section "Create a Role" in the file
doc/radosgw/role.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit fa6ad94f3afc6d11c3ed243d66c96a666e973bde)

Merge pull request #54865 from batrick/wip-63782-reef

reef: qa: add support/qa for cephfs-shell on CentOS 9 / RHEL9

Reviewed-by: Venky Shankar <vshankar@redhat.com>

18.2.1

Signed-off-by: Ceph Release Team <ceph-maintainers@ceph.io>

reef: qa: only test cephfs-shell with ubuntu or centos 9.stream

Because reef is tested on RHEL/centos 8, we need to remove those jobs from
testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

PendingReleaseNotes: announce cephfs-shell avail. on rhel9

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 04a54fbd8297e1027e1a8d0259ff406b41a2ae4c)

qa: test fs:shell on all distros

This is a simple sub-suite that has one job. Always schedule on all supported distros.

Fixes: https://tracker.ceph.com/issues/43393
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 215d96c1d1533836d313be54864c1b0525440b4c)

qa: add cephfs-shell to installed rpm packages

For CephFS QA.

Fixes: https://tracker.ceph.com/issues/43393
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit b12bdfeee3647b495db87be5f3d0abcb3ca77034)

ceph.spec.in: enable support for cephfs-shell by default via EPEL9

The required packages [1,2] are now at the right version and readily available.

[1] https://packages.fedoraproject.org/pkgs/python-cmd2/python3-cmd2/epel-9.html
[2] https://packages.fedoraproject.org/pkgs/python-colorama/python3-colorama/epel-9.html

Fixes: https://tracker.ceph.com/issues/43393
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d63d96f8bcda08990bc3be28c9a835d6c836f0a3)

ceph-volume: fix mpath device support

commit [1] broke mpath devices support in `disk.is_device()`

[1] https://github.com/ceph/ceph/commit/4fc6bc394dffaf3ad375ff29cbb0a3eb9e4dbefc

Fixes: https://tracker.ceph.com/issues/62722
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>
(cherry picked from commit 28a12f8ef703a7349d9f2de40251c22e018fc475)
(cherry picked from commit 1e7223281fa044c9653633e305c0b344e4c9b3a4)

test/librbd: close image in DiffIterateDeterministic

... to avoid valgrind reporting a memory leak on ImageCtx.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit be40bbbb12823721f8fb1cac4db681f5c22825ea)

test/librbd: drop DiffIterateTest.DiffIterateRegression6926

This was added to test [1].  It's duplicated by several cases in
DiffIterateTest.DiffIterateDeterministicPP now.  Specifically, the
issue could be reproduced by any of:

    (8) beginning of time -> snap2
    (9) snap1 -> snap2
    (10) beginning of time -> snap1

[1] https://tracker.ceph.com/issues/6926

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 93ff7fe6e4b1232a97f669f3dce6653cc08638a8)

test/librbd: drop TestLibRBD.SnapDiff

This was added to integration test [1], separate from the fix which
went in only with unit test adjustments.  It's duplicated by several
cases in DiffIterateTest.DiffIterateDeterministic now.  Specifically,
the issue could be reproduced by any of:

    (3) snap2 -> HEAD
    (4) snap3 -> HEAD
    (7) snap2 -> snap3

[1] https://tracker.ceph.com/issues/50787

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 356ac6a5477061389e673053f178d1ec6a9b9427)

test/librbd: add DiffIterateTest.DiffIterateDeterministic{,PP}

scribble()-based DiffIterate tests are too weak: at least two
regressions that should been caught by DiffIterate.DiffIterate or
DiffIterate.DiffIterateStress were missed [1][2].  Aside from the
randomness which can be both a good and a bad thing, asserts there
ensure only that the returned diff covers all changes that were made.
If the returned diff is too excessive or otherwise bogus, this isn't
detected [3].

Add a deterministic test to systematically cover the most common cases
that don't involve discards.  A similar test for discards will be added
with the fix for [4].

Comment out debug log in vector_iterate_cb() like it's done in
iterate_cb().

[1] https://tracker.ceph.com/issues/50787
[2] https://tracker.ceph.com/issues/63654
[3] https://tracker.ceph.com/issues/63719
[4] https://tracker.ceph.com/issues/53897

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit f5e3f263724d04be181225d9e24cbe2f4669e8bc)

librbd: fix read_whole_object handling in ObjectListSnapsRequest

Originally, in commit 2be4840afd4f ("librados/snap_set_diff: don't
assert on empty snapset"), exists was set to true.  This didn't make
ObjectListSnapsRequest, causing the following deep-copy tests to fail
when run against calc_snap_set_diff() rigged to return "whole object"
as described in [1]:

    TestDeepCopy.Snaps
    TestDeepCopy.SnapDiscard
    TestDeepCopy.CloneHideParent
    TestDeepCopy.Snaps_LargerDstObjSize
    TestDeepCopy.Snaps_SmallerDstObjSize

This is a regression introduced in commit cc87a8bd697e ("librbd:
deep-copy object utilizes image-extent IO methods") by way of commit
11923e234efc ("librbd: generic object list snapshot request").

[1] https://github.com/ceph/ceph/pull/20648#issuecomment-369292309

Fixes: https://tracker.ceph.com/issues/63654
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 0a1f633e0240b4a7cfbcddd96d53fbf4b17f0b28)

librbd: fix LIST_SNAPS_FLAG_WHOLE_OBJECT behavior

Bundling read_whole_object and LIST_SNAPS_FLAG_WHOLE_OBJECT cases
together is wrong:

- In read_whole_object case, calc_snap_set_diff() sets just
  read_whole_object.  Everything else is zeroed out and may require
  resetting to fit with the rest of ObjectListSnapsRequest logic.

- In LIST_SNAPS_FLAG_WHOLE_OBJECT case, only the diff should be
  expanded.  Everything else is set by calc_snap_set_diff() and should
  be used as is.  This goes for end_size in particular -- if it's reset
  to object size, bogus zero extents may be returned as the object
  would appear to have grown.

This is a regression introduced in commit 4429ed4f3f4c ("librbd: switch
diff iterate API to use new snaps list dispatch methods") by way of
commit 66dd53d9c4d9 ("librbd: optionally return full object extent for
any snapshot deltas").

Fixes: https://tracker.ceph.com/issues/63654
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 8f86d80614680afecbfe82b2a6e965678a3c6034)

test/librbd: make ListSnapsWholeObject actually test stuff

Despite being added in commit 66dd53d9c4d9 ("librbd: optionally return
full object extent for any snapshot deltas") ostensibly to test the new
LIST_SNAPS_FLAG_WHOLE_OBJECT code, it surely doesn't do that because
the flag isn't even passed to MockObjectListSnapsRequest::create().

I can only guess, but it looks like snap ID 3 was intended to be
a starting point.  Otherwise, with 0 and CEPH_NOSNAP passed as snap
IDs, the overlap that is set up for the clone wouldn't affect the
computation in any way.

Use snap ID 3 as a starting point and run both with and without
LIST_SNAPS_FLAG_WHOLE_OBJECT on the same snapset to pinpoint the
difference.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit bd52297a71100dd35874fa3aeda81fca1d828b38)

librados/snap_set_diff: set end_size only if end object exists

Since commit 73f50a13109f ("rbd-mirror: use generalized deep copy for
image sync"), the only user of calc_snap_set_diff() immediately unsets
end_size otherwise.

calc_snap_set_diff() semantics are clearer if end_size is set together
with end_exists and clone_end_snap_id.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit c0747922040841c371dbe2706354c08c73d8e59f)

test/librbd: actually alternate overlaps in DiscardWithPruneWriteOverlap

Make sense of the inner loop in write_thread.  The crash on
"it != m_events.end()" assert reproduces even faster this way.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit d790b57b4bd3868cf12941c565b6906534e19f91)