scribble()-based DiffIterate tests are too weak: at least two
regressions that should been caught by DiffIterate.DiffIterate or
DiffIterate.DiffIterateStress were missed [1][2]. Aside from the
randomness which can be both a good and a bad thing, asserts there
ensure only that the returned diff covers all changes that were made.
If the returned diff is too excessive or otherwise bogus, this isn't
detected [3].
Add a deterministic test to systematically cover the most common cases
that don't involve discards. A similar test for discards will be added
with the fix for [4].
Comment out debug log in vector_iterate_cb() like it's done in
iterate_cb().
Ilya Dryomov [Mon, 27 Nov 2023 10:59:26 +0000 (11:59 +0100)]
librbd: fix read_whole_object handling in ObjectListSnapsRequest
Originally, in commit 2be4840afd4f ("librados/snap_set_diff: don't
assert on empty snapset"), exists was set to true. This didn't make
ObjectListSnapsRequest, causing the following deep-copy tests to fail
when run against calc_snap_set_diff() rigged to return "whole object"
as described in [1]:
This is a regression introduced in commit cc87a8bd697e ("librbd:
deep-copy object utilizes image-extent IO methods") by way of commit 11923e234efc ("librbd: generic object list snapshot request").
Ilya Dryomov [Mon, 27 Nov 2023 09:11:52 +0000 (10:11 +0100)]
librbd: fix LIST_SNAPS_FLAG_WHOLE_OBJECT behavior
Bundling read_whole_object and LIST_SNAPS_FLAG_WHOLE_OBJECT cases
together is wrong:
- In read_whole_object case, calc_snap_set_diff() sets just
read_whole_object. Everything else is zeroed out and may require
resetting to fit with the rest of ObjectListSnapsRequest logic.
- In LIST_SNAPS_FLAG_WHOLE_OBJECT case, only the diff should be
expanded. Everything else is set by calc_snap_set_diff() and should
be used as is. This goes for end_size in particular -- if it's reset
to object size, bogus zero extents may be returned as the object
would appear to have grown.
This is a regression introduced in commit 4429ed4f3f4c ("librbd: switch
diff iterate API to use new snaps list dispatch methods") by way of
commit 66dd53d9c4d9 ("librbd: optionally return full object extent for
any snapshot deltas").
Ilya Dryomov [Sun, 19 Nov 2023 21:44:28 +0000 (22:44 +0100)]
test/librbd: make ListSnapsWholeObject actually test stuff
Despite being added in commit 66dd53d9c4d9 ("librbd: optionally return
full object extent for any snapshot deltas") ostensibly to test the new
LIST_SNAPS_FLAG_WHOLE_OBJECT code, it surely doesn't do that because
the flag isn't even passed to MockObjectListSnapsRequest::create().
I can only guess, but it looks like snap ID 3 was intended to be
a starting point. Otherwise, with 0 and CEPH_NOSNAP passed as snap
IDs, the overlap that is set up for the clone wouldn't affect the
computation in any way.
Use snap ID 3 as a starting point and run both with and without
LIST_SNAPS_FLAG_WHOLE_OBJECT on the same snapset to pinpoint the
difference.
Ilya Dryomov [Sat, 11 Nov 2023 13:15:49 +0000 (14:15 +0100)]
librados/snap_set_diff: set end_size only if end object exists
Since commit 73f50a13109f ("rbd-mirror: use generalized deep copy for
image sync"), the only user of calc_snap_set_diff() immediately unsets
end_size otherwise.
calc_snap_set_diff() semantics are clearer if end_size is set together
with end_exists and clone_end_snap_id.
Zac Dover [Wed, 1 Nov 2023 01:53:59 +0000 (11:53 +1000)]
doc/cephadm: edit troubleshooting.rst (1 of x)
Edit doc/cephadm/troubleshooting.rst. This commit and the PR of which it
is a part was raised in response to
https://github.com/ceph/ceph/pull/53976. The limits of reStructuredText
are particularly visible here in every instance of a BASH for-loop and
in every instance of a command stretched over multiple lines.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
Rishabh Dave [Thu, 26 Oct 2023 10:48:31 +0000 (16:18 +0530)]
cmake: add --progress flag to git submodule update commands
Ceph has lots of submodules that needs to be cloned before building
binaries from the repository. Seeing the progress when these submodules
are being cloned is useful, especially when developers/users have a
network issue or a slow network.
Zac Dover [Mon, 30 Oct 2023 02:37:39 +0000 (12:37 +1000)]
doc/glossary: improve "BlueStore" entry
Initially s/backend/back end/ but then I added a little more information
about BlueStore's use of RocksDB to map object names to block locations
on disk.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
There is no need for CreateSnapshotRequests.__del__() that calls
CreateSnapshotRequests.wait_for_pending().
MirrorSnapshotScheduleHandler.shutdown() already calls
CreateSnapshotRequests.wait_for_pending().
Ramana Raja [Thu, 26 Oct 2023 17:18:52 +0000 (13:18 -0400)]
mgr/rbd_support: fix recursive locking on CreateSnapshotRequests lock
The MirrorSnapshotScheduleHandler's run thread issues asynchronous
create snapshot requests using a CreateSnapshotRequests instance. When
the thread invokes a CreateSnapshotRequests instance's get_ioctx(),
the instance's class variable lock is acquired. With the class
variable lock held, the garbage collection of a CreateSnapshotRequests
instance may race in the thread. The thread would then call
CreateSnapshotRequests __del__() that tries to acquire the class
variable lock that the thread already holds. Fix this
recursive deadlock by converting the CreateSnapshotRequests lock from
a class variable to an instance variable. There is no need to share
the lock across CreateSnapshotRequests instances.
Also convert MirrorSnapshotScheduleHandler, PerfHandler and
TrashPurgeScheduleHandler class variables to instance variables
that don't need to be shared across the instances.
Fixes: https://tracker.ceph.com/issues/62994 Signed-off-by: Ramana Raja <rraja@redhat.com> Co-Authored-By: Ilya Dryomov <idryomov@gmail.com>
Adam Emerson [Sat, 28 Oct 2023 17:29:59 +0000 (13:29 -0400)]
build: Fix fmt version check
Currently, when attempting to build ceph on a system with fmt
installed, we try to build against it whatever the version. This
constantly breaks people's builds, since newer versions of fmt often
change the API.
This change specifies that versions must be below 10 as well as at or
above 8.1.1, so that on systems with a new format, we fall back to
using the submodule.
It also removes the `Findfmt.cmake` module, as that does not check
the installed version. Instead, we use the cmake config file installed by
the system package of fmt and does support version checking.
Aashish Sharma [Mon, 30 Oct 2023 07:47:37 +0000 (13:17 +0530)]
mgr/dashboard: update rgw multisite import form helper info
Change 'To obtain the token, generate it from your secondary Ceph cluster' to 'To obtain the token, generate it from your primary Ceph cluster' in rgw multisite import form helper
Ronen Friedman [Sat, 28 Oct 2023 16:42:34 +0000 (11:42 -0500)]
osd/scrub: do not clear PG_STATE_REPAIR unconditionally
As we now call clear_pgscrub_state() at the end of each
'Session' state, we must not clear PG_STATE_REPAIR
unconditionally.
Previously - scrubs that reached normal completion, i.e.
reached PgScrubber::scrub_finish(), would have only cleared
that PG flag under specific conditions. That was changed in
previous commits of this PR, and is now fixed.
Zac Dover [Fri, 27 Oct 2023 06:58:28 +0000 (16:58 +1000)]
doc/rados: remove cache-tiering-related keys
Remove information related to cache-tiering-related keys from
doc/rados/operations/pools.rst. Cache-tiering is deprecated in Reef.
This PR is suitable for backporting to the Reef release branch, but not
to release branches prior to Reef.
test/librbd/fsx: wait for resize to propagate in krbd_resize()
With this changes resize request will not be blocked until the resize is
completed. Because of this the fsx test fails as it assumes that the
request to resize immediately implies changes on the device size.
Hence we have to add a wait in resize handler of fsx for the device to
actually get resized.
Zac Dover [Wed, 25 Oct 2023 23:48:57 +0000 (09:48 +1000)]
doc/rados: remove HitSet-related key information
Remove HitSet-related key information from
doc/rados/operations/pools.rst. HitSet-related keys are relevant only to
releases of Ceph that support cache tiering. Only Quincy and earlier
(inclusive) releases of Ceph support cache tiering. Backport this commit
from main to Reef, but not to Quincy or to release branches earlier than
Quincy.
John Mulligan [Wed, 11 Oct 2023 18:05:17 +0000 (14:05 -0400)]
cephadm: add a --dry-run option to cephadm shell
Instead of creating the shell, the --dry-run option prints the container
command that would be used. This can be used as a starting point for
creating custom container commands similar to what cephadm shell would
generate but with tweaks.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Redouane Kachach [Mon, 23 Oct 2023 15:18:57 +0000 (17:18 +0200)]
mgr/rook: remove all the hardcoded refs to the 'rook-ceph' namespace Fixes: https://tracker.ceph.com/issues/63291 Signed-off-by: Redouane Kachach <rkachach@redhat.com>
John Mulligan [Sun, 22 Oct 2023 12:14:24 +0000 (08:14 -0400)]
cephadm: add deployment test for osd
Add a deployment test case for OSD. OSD has some special properties that
we have extra assertions for.
Part of a series of commits to increase coverage of deployment
path features with regards to container engine options, env vars
and mounts. This will serve future refactoring efforts.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 21 Oct 2023 20:27:13 +0000 (16:27 -0400)]
cephadm: add test assertions for nvmeof options, mount
Part of a series of commits to increase coverage of deployment
path features with regards to container engine options, env vars
and mounts. This will serve future refactoring efforts.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
projects / ceph.git / log