Creating a global, pool or namespace level mirror snapshot
schedule shouldn't schedule mirror snapshots of images that
are part of a group and reside in the pool or namespace.
Also disallow directly scheduling mirror image snapshots on
images that are part of a group.
Ramana Raja [Tue, 5 Nov 2024 16:12:54 +0000 (11:12 -0500)]
qa/workunits/rbd: add basic tests for mirror group snapshot scheduler
Add tests to check the basic functionality of the
mirror_group_snapshot_schedule module. Check that
- `rbd mirror group snapshot schedule add/rm/status/ls` commands work
- the module can recover from blocklisting of its client and continue
to process requests
rbd-mirror: more improvements in the group replayer
- Arresting the interruptions after shutdown
- Restart GroupReplayer if bootstrap returns remote group id as empty
- Fix a test failure in "regular group snapshots test"
rbd-mirror: create group snapshots only if remote group snapshot is complete
Also do not create a regular group snapshot until the next mirror group
snapshot is created. Otherwise we simply have an incomplete regular group
snapshot until the next mirror group snapshot is created.
* fix braces in the imageMap update_images_added & update_images_removed
* do not allow image add from non-primary
* `down+unknown` status shown on querying individual images which are part
of group enabled for mirroring
* `mirror pool status` shows down+unknown status
* fix imageMap being overwritten when multiple images are enabled for mirroring
* fix misleading error msg when getting status of a non-mirror enabled group
$ rbd --cluster site-a group snap ls pool1/test_group --debug-rbd=0
NAME STATUS
group_snap1 ok
group_snap2 ok
.mirror.2_10416b8b4567_10536b8b4567 ok
$ rbd --cluster site-b group snap ls pool1/test_group --debug-rbd=0
NAME STATUS
.mirror.2_10376b8b4567_1037327b23c6 ok
group_snap1 ok
group_snap2 ok
.mirror.2_10416b8b4567_10536b8b4567 ok
rbd-mirror: add undo code, exclusive locking and quiescing
* add essential logic to undo partially succeeded API's like, group promote,
group demote, group enable, group disable, group image add and
group image remove
* add exclusive locking and quiescing with-in all the required group API's
* adress code duplication and optimization with in the group API's
Make sure group snapshots doesn't get copied to secondary if the group snapshot
is incomplete on primary. On creation time of a group snapshots on primary,
make sure to delete the previous snapshot in case it is incomplete.
Max Kellermann [Thu, 24 Apr 2025 05:17:48 +0000 (07:17 +0200)]
mds/Locker: use ceph_abort_msg() instead of ceph_assert()
This ceph_assert() always fails, but depending on the configuration
value `ceph_assert_supresssions`, execution may continue, but the
`dir` variable is left uninitialized. This leads to a compiler
warning:
/home/jenkins-build/build/workspace/ceph-api/src/mds/Locker.cc:451:22: error: variable 'dir' is used uninitialized whenever 'if' condition is false [-Werror,-Wsometimes-uninitialized]
clang then suggests to nullptr-initialize the variable:
/home/jenkins-build/build/workspace/ceph-api/src/mds/Locker.cc:447:11: note: initialize the variable 'dir' to silence this warning
447 | CDir *dir;
| ^
| = nullptr
This, however, is a very bad idea because all this does is suppress
the warning; it still crashes the process.
Since there's no recovery from this problem, let's switch to
ceph_abort_msg() which is [[noreturn]] and the compiler can deduce
that `dir` is always initialized when it's used.
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Merge pull request #62693 from ronen-fr/wip-rf-iocnt
osd/scrub: performance counters for I/O performed by the scrubber
Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com> Reviewed-by: Bill Scales <bill_scales@uk.ibm.com> Reviewed-by: Samuel Just <sjust@redhat.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
N Balachandran [Mon, 21 Apr 2025 11:34:08 +0000 (17:04 +0530)]
rbd: display correct mirror state when creating
The mirror image state is set to MIRROR_IMAGE_STATE_CREATING
when the image is first created on the secondary, but was displayed
as "unknown" by the rbd info command. This has been fixed.
Fixes: https://tracker.ceph.com/issues/70963 Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
HealthMonitor: Add topology-aware netsplit detection and warning
Problem:
Currently, Ceph cannot detect and report network partitions (netsplits)
between monitors in different topology locations in a consolidated way.
While stretch mode can handle partitions through monitor elections,
users lack visibility into the topology-level view of network
disconnections, making troubleshooting difficult.
Solution:
This implementation adds a hierarchical netsplit detection mechanism that:
- Uses DirectedGraph structure for netsplit detection
- Maps monitor disconnections to relevant CRUSH topology levels
- Aggregates individual disconnections into location-level reports when appropriate
- Detects complete location-level netsplits when ALL monitors between locations
cannot communicate
- Reports specific topology locations experiencing complete communication failures
- Falls back to individual monitor-level reporting for partial disconnections
- Handles monitors with missing location data gracefully
- Leverages HealthMonitor::check_for_mon_down to receive a set of down monitors,
efficiently avoiding false netsplit reports for monitors already known to be down
- Implements smart filtering that correctly excludes down monitors from location-based
analysis, ensuring accurate netsplit reporting at both individual and topology levels
The implementation produces user-friendly health warnings:
1. For complete location netsplits: "Netsplit detected between dc1 and dc2"
2. For individual monitor disconnections: "Netsplit detected between mon.a and mon.d"
Performance considerations:
- Time complexity: O(m²) where m is the number of monitors
- Space complexity: O(m²) for connection tracking
- Practical impact is minimal as monitor count is typically small (3-7)
librbd: disallow "rbd trash mv" if image is in a group
Removing an image that is a member of a group has always been
disallowed. However, moving an image that is a member of a group to
trash is currently allowed and this is deceptive -- the only reason for
a user to move an image to trash should be the intent to remove it.
More importantly, group APIs operate in terms of image names -- there
are no corresponding variants that would operate in terms of image IDs.
For example, even though internally GroupImageSpec struct stores an
image ID, the public rbd_group_image_info_t struct insists on an image
name. When rbd_group_image_list() encounters a trashed member image
(i.e. one that doesn't have a name), it just fails with ENOENT and no
listing gets produced at all until the offending image is restored from
trash. Something like this can be very hard to debug for an average
user, so let's make rbd_trash_move() fail with EMLINK the same way as
rbd_remove() does in this scenario.
The one case where moving a member image to trash makes sense is live
migration where the source image gets trashed to be almost immediately
replaced by the destination image as part of preparing migration.
EMLINK is returned by rbd_remove() if the image is a member of a group.
Add a dedicated exception similar to ImageBusy or ImageHasSnapshots and
a test for it.
projects / ceph.git / log