rbd-mirror: do not prune primary snapshots as part of the secondary daemon
Problem:
In a relocate situation there is a demote involved and we have a demote
snapshot both on primary with peer UUID attached and on secondary with peer UUID
attached. Later a promote say on site-b triggered primary demote group snap
removal on site-a which started a image snap removal, so it started a request to
image replayer (idle at that time), but the request didn't yet went
through.
As a next command we had --force promote on site-a but since the request on
site-a for image snap removal is already in place it somehow went through and
removed the image snaps leaving the group snap on site-a primary demote snapshot
partially deleted with peer UUID attached. And now note site-a is again primary.
In the later test, it involved demote on site-b and followed by a resync on it.
But since the primary demote snapshot is lingering in the partially deleted
state and since peer UUID is still attached, there is no way the resync will
go through, leaving the syncing snaps in incomplete state forever.
Note: Any CLI command on new primary site-a cannot delete the partially deleted
snap because there is a peer UUID attached to it.
Solution:
Do not prune any primary snapshot (including primary demote snapshots) as part
of the secondary daemon operations.
It is safe, however, to unlink remote group snapshots (such as non-primary
demote snapshots). Just ensure that primary demote snapshots are not pruned by
the secondary daemon.
Even in scenarios where a relocation back to the initial cluster occurs in the
future, the primary demoted snapshot on the current secondary (which becomes
primary during relocation) will not be synced again — even though it retains
with a peer UUID. This is because the scan_for_unsynced_group_snapshots()
function uses the last local group snapshot ID to locate the corresponding
snapshot on the remote side and continues syncing from that point onward, rather
than starting from the beginning.
However, in the event of a full resync request, this snapshot will be synced.
This is expected behavior and should not pose any issues. This actually
helps in pruning the primary demote snap on primary quickly later.
librbd/mirror: remove peer UUID of group snapshot before attempting removal
Problem:
When the number of snapshots exceeds the configured limit
(rbd_mirroring_max_mirroring_snapshots), the cleanup process remove the last
group snapshot on the primary. If this removal fails partially, the
group snapshot with some or empty image snap may still remain.
As a result:
* The snapshot is incorrectly considered for synchronization to the secondary.
* The secondary snapshot can remain forever in the incomplete state, leading
to stuck of daemon progress.
Solution:
Before attempting to remove the group snapshot:
* Explicitly remove the peer UUID associated with the snapshot.
* This ensures that the snapshot is no longer considered eligible for
synchronization.
* Even if the actual snapshot deletion fails partially, the snapshot is not
tried for mirroring, avoiding daemon stalls or inconsistencies.
rbd_mirror: order the user group snaps and the mirror group snaps syncing
In case of user group snap addition:
* Wait for the user group snap to sync to complete and only then mark
the mirror group snap to complete
In case of user group snap removal:
* Wait for the user group snap to get pruned and only then mark the
mirror group snap to complete
Updated tests:
• test_create_group_with_images_then_mirror_with_regular_snapshots
enable scenario 1 as with the recent changes it works, also remove
additional mirror group snapshot as it is no more required.
Ramana Raja [Tue, 26 Aug 2025 19:22:12 +0000 (15:22 -0400)]
qa/workunits/rbd: fix race in test_force_promote_delete_group
... that is a part of the rbd_mirror_group_simple test suite.
In "test_force_promote_delete_group," the secondary group is
force-promoted, but the original primary is not demoted. The
force-promoted group is disabled for mirroring. One of its member images
is removed. The group is then re-enabled for mirroring before being
removed.
Sometimes the check to confirm that the group has been removed fails.
This happens because the rbd-mirror daemon’s group replayer, after
restarting, recreates the group immediately after it has been explicitly
removed and before the check for group removal.
To fix this race, remove the check to confirm that the group has been
removed as it is not necessary. The successful execution of the group
removal command should be sufficient.
N Balachandran [Mon, 2 Jun 2025 04:08:52 +0000 (09:38 +0530)]
librbd/api/Mirror: enhancements to group_enable API
Notable enhancements have been made to the mirror group_enable API:
* Previously, member images were enabled for mirroring synchronously
and one after another. They are now enabled asynchronously and
concurrently, making the time to enable a group independent of the
number of member images it contains.
* Intermediate steps involved in enabling an image, such as fetching
mirror peers, acquiring an exclusive lock, creating a mirror snapshot,
among other operations, run asynchronously but are synchronized
across all the group member images using C_Gather callbacks. An
asynchronous operation on an image proceeds to the next step only
after that operation completes for all member images. This allows for
easier cleanup of member images and the group when an intermediate
operation fails for one or more member images.
* The exclusive locks on member images are now held for a shorter
duration. All member image locks are acquired just before taking
image snapshots and released after retrieving the snapshot IDs.
Previously, the locks were obtained earlier in the sequence of
operations, even for steps that did not need them, and were not
explicitly released.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com> Co-Authored-by: Ramana Raja <rraja@redhat.com>
VinayBhaskar-V [Fri, 8 Aug 2025 13:00:34 +0000 (13:00 +0000)]
rbd-mirror: Allow group disable without --force when in PROMOTION_STATE_UNKNOWN
With the earlier version disabling a primary group with promotion state PROMOTION_STATE_UNKNOWN
requires a force flag which deviates from the existing behavior of standalone images.
This change aligns group disable behavior with that of standalone images by allowing primary
groups in PROMOTION_STATE_UNKNOWN state to be disabled without the force flag.
rbd-mirror: fix group replayer deleting the previous mirror group snapshot
The prune logic in the group replayer currently holds the previous
group snapshot until the current group snapshot syncing is complete.
Hence there is always a previous group snapshot (and the respective
image snapshots).
But when there are user group snapshots group replayer takes them to
account and prunes the previous mirror snapshots, which shouldn't be done.
The group replayer should always hold the previous mirror group snapshot
(irrespective of the user group snap).
Also rename unlink_group_snapshots() -> prune_group_snapshots() and
substitute the use of unlink with prune as required.
1. move prepare_non_primary_mirror_snap_name to anonymous namespace
2. call is_rename_requested() in handle_load_remote_group_snapshots()
3. restructure various routines for better readability
rbd-mirror: allow removal of user group snapshot if id do not match
Currently in the unlink_group_snapshots() we check for regular group snapshot
names on remote group and local group. If the same name user snapshot exists on
remote group we do not remove it locally. There might be a case, user deleted
the user group snapshot on the remote group and recreated it with the same name
again (before the delete update via a mirror group snapshot is conveyed locally),
even in such case unfortunately currently we still don't delete the group
snapshot locally as the names still matching with the locally existing snapshot.
Hence we need to preserve the user snapshot locally only if their id's match and
not the group snapshot names.
Without this fix the mirroring daemon stays stuck at the same snapshot.
librbd: unlink the individual image snapshots taken as part of group snapshot
... but are not linked to group snapshot. This can happen when a mirror group
snapshot process crashes and the group snapshot is left in INCOMPLETE
state. The fix is to crawl through the list of images in the group and
match the individual image snapshots with a matching group snap id in
them and removing them.
VinayBhaskar-V [Tue, 27 May 2025 13:28:51 +0000 (18:58 +0530)]
rbd-mirror: filter group snapshots by mirror_peer_uuid in group_replayer
The group_replayer::Replayer was previously including all remote group
snapshots during synchronization, regardless of whether they were
associated with the current peer's mirror_peer_uuid. This could result
in syncing snapshots that belong to other peers, leading to inconsistent
behavior.
This commit ensures that only remote group snapshots containing the
expected mirror_peer_uuid are processed. Snapshots that do not match
are filtered out early in the replay logic, preventing cross-peer
interference.
VinayBhaskar-V [Tue, 27 May 2025 13:19:08 +0000 (18:49 +0530)]
rbd-mirror: ensure safe access to m_prune_snap_ids_by_gr
The ImageReplayer m_prune_snap_ids_by_gr set is modified under a lock
but accessed from external calls without a lock. This unsynchronized access can
lead to undefined behavior if the set is accessed while the set is being modified.
This commit ensures that ImageReplayer m_prune_snap_ids_by_gr are
safely access under a lock.
qa/workunits/rbd: bypass regular group snapshot removal checks
Currently there is a delay in the deletion of the regular group snaps that
needs fixing, so just commenting the checks to avoid the temporary test
failures.
Ilya Dryomov [Mon, 16 Jun 2025 20:46:38 +0000 (22:46 +0200)]
librbd: allow group remove when group mirroring isn't supported
Ignore a potential EOPNOTSUPP error from Mirror::group_disable() in
Group::remove() -- this would come up if the client side (including rbd
CLI) is upgraded before the OSDs. The same is done for standalone
images in RemoveRequest::handle_disable_mirror().
group get info shouldn't look at incomplete group snapshot if group is
primary or primary demoted. For example a failed force promote can leave an
incomplete primary group snapshot on the group, it doesn't mean that group
is primary yet.
create a dummy/orphan snapshot, on a force promote operation. It is a
special non-primary snapshot without a mirror uuid and any image snapshots
within it. Its existence denotes that a force promote had begun. It is
created when a force promote was issued and no demote snapshot exists locally.
N Balachandran [Mon, 12 May 2025 15:35:24 +0000 (21:05 +0530)]
rbd-mirror: ignore EOPNOTSUPP for group listing
pool_watcher::RefreshEntitiesRequest errored out if listing mirrored
groups on a remote cluster failed with EOPNOTSUPP. In mixed clusters, this
prevented image replayers from starting up and hence syncing standalone images.
EOPNOTSUPP is now ignored when listing mirror groups.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
N Balachandran [Fri, 25 Apr 2025 11:31:58 +0000 (17:01 +0530)]
rbd-mirror: fix missing mirror group status
The recent changes to the GroupReplayer Bootstrap caused the
mirror group status for a newly enabled group to not be written to
the rbd_mirroring object on the secondary for upto 10 seconds.
The cls_rbd group_status_set did not write the status to disk if
the group does not exist or if its state is not enabled. This has been
modified to write the status even if the status is creating.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
librbd: let Mirror::group_get_info() return ENOENT only if group DNE
Getting ENOENT when the group doesn't exist is very much desired,
especially considering the fact that groups can't be opened like images
are and that group APIs operate in terms of group names. However, in
case the group exists but mirroring is disabled, it's better to remain
consistent with Mirror::image_get_info() and return a partially formed
struct where state is set to RBD_MIRROR_GROUP_DISABLED -- this is the
behavior that API users have grown to rely on.
John Agombar [Mon, 28 Apr 2025 12:29:46 +0000 (13:29 +0100)]
qa/workunits/rbd: update to mirror group snapshot tests
Updated tests:
• test_odf_failover_failback - Update retry_promote screnario to limit retries to
10000 attempts
Disabled tests:
• test_create_group_with_images_then_mirror_with_regular_snapshots scenario 1
test sometimes fails as removed regular snapshot remains on secondary cluster
after new mirror group snapshot is sync complete
Helpers:
- get_pool_image_count() $XMLSTARLET variable shouldn't be used anymore
* do not group/image demote on undo (promote failure)
* wait for demote snapshot sync to complete locally before accepting and continue
to actually perform group promote.
N Balachandran [Wed, 23 Apr 2025 15:52:34 +0000 (21:22 +0530)]
rbd-mirror: fix remove_mirror_peer_uuid
A race between the remove_mirror_peer_uuid() function in the group Replayer,
which called group_snap_set without checking if there were any
peer_uuids on the remote snap, and a snapshot cleanup on the primary
could lead to a case where the group snapshot is recreated on the
primary without the snap_order key. This causes all mirroring operations
on the group to fail on the primary as ListSnapshotsRequest fails when
it cannot find the snap_order key.
This has been fixed to update the group snapshot only if it contains the
peer_uuid.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
N Balachandran [Sat, 19 Apr 2025 11:26:49 +0000 (16:56 +0530)]
rbd-mirror: group replayer bootstrap changes
The group replayer BootstrapRequest has been refactored to make
it easier to handle various scenarios. It now mimics the ImageReplayer
Bootstrap sequence to a great degree.
Credit to Ilya Dryomov <idryomov@gmail.com> for the help with
integrating GroupMirrorStateUpdateRequest into the Replayer.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
Ramana Raja [Tue, 15 Apr 2025 19:16:34 +0000 (15:16 -0400)]
librbd/api: propagate ENOENT error in Mirror::group_get_info() API
The ENOENT error was ignored by the librbd API that retrieves mirror
group information. This manifests as a bug in the mirror group
snapshot scheduler, where the scheduler utilized this API to prevent
scheduler commands on groups that are not enabled for snapshot-based
mirroring. Since the librbd API masked the ENOENT error, the
scheduler's check for mirror group mode was compromised, resulting
in scheduler commands unexpectedly succeeding on groups that were not
enabled for snapshot-based mirroring. Therefore, allow the ENOENT error
from the API that retrieves mirror group information to surface. This
fixes the spurious mirror group mode check in the mirror snapshot
scheduler. Additionally, add a test to ensure that group snapshot
schedule commands fail on a group not enabled for mirroring.
John Agombar [Tue, 15 Apr 2025 15:37:24 +0000 (16:37 +0100)]
qa/workunits/rbd: update to rbd_mirror_group_simple
Added -p option to rbd_mirror_group_simple.sh. This can be used to print the enabled tests
and scenarios (for use by Jenkins) when producing a list of tests to run
Changed order of tests so that long running tests are run later
Updated tests:
• test_group_rename - workaround for intermittent failure
New tests:
• test_invalid_actions - test is currently disabled as it is not complete
N Balachandran [Tue, 15 Apr 2025 11:34:41 +0000 (17:04 +0530)]
rbd-mirror: fix crash in group Replayer
The rbd-mirror daemon crashed if the Replayer was destroyed while
an on-going remote group snap listing operation was in progress. The
callback would attempt to access members of the Replayer instance which
no longer existed.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
John Agombar [Thu, 10 Apr 2025 18:29:48 +0000 (19:29 +0100)]
qa/workunits/rbd: update to mirror group snapshot tests
Added -d <filename> option to rbd_mirror_group_simple.sh. This can be used to
save the stdout and stderr from commands run during a test+scenario into $TEMPDIR/filename.
After a successful test completion the contents of this file are deleted prior to running
the next test to prevent the file from becoming too large.
Updated tests:
• various - Added step after a resync request to wait for the group id to change before
continuing. This ensures that the group delete/recreate step has been completed and
prevents later commands from failing with group doesn't exist type errors.
• test_enable_mirroring_when_duplicate_group_exists - Added checks of the "state" and
"description" fields for the peer site in the group status output. Test is disabled
as it currently fails
• test_enable_mirroring_when_duplicate_image_exists_scenarios - simplified test to only
have a single duplicate named image. Test fails still and is disabled.
• test_remote_namespace - added steps to take new snapshot on primary after failover
and check that this syncs successfully.
• test_group_and_standalone_images_do_io - merged 2 scenarios to remove duplication
New tests:
Updated tests:
• test_demote_snap_sync - Checks that a demote snap is correctly synced to the secondary
after the deamon is restarted
rbd: don't fail "rbd group snap ls" if namespace details aren't available
Treat namespace details as an optional set of details, the same as "rbd
snap ls --all" command does. This avoids sporadic failures with ENOENT
when snapshot listing races snapshot removal which "rbd group snap ls"
command is actually more vulnerable to than "rbd snap ls --all" because
namespace details for group snapshots aren't cached internally.
Credit to N Balachandran <nibalach@redhat.com> for root causing this.
rbd-mirror: don't reset m_on_start_finish in GroupReplayer::restart()
If restart() is called while a previous start() is still in progress,
resetting m_on_start_finish before calling stop() from inside restart()
causes on_finish that was passed to start() to be leaked. One of the
ways this manifests in is a hang in InstanceReplayer::stop() on any
kind of shutdown -- due to InstanceReplayer::start_group_replayer()
keeping track of its start() invocations through C_TrackedOp
Ramana Raja [Thu, 27 Mar 2025 17:09:47 +0000 (13:09 -0400)]
rbd_mirror: add fields to mirror group status's description str
Add the following fields to the mirror group status's description string:
- last_snasphot_bytes
- last_snasphot_complete_seconds
- local_snapshot_timestamp
- remote_snapshot_timestamp
John Agombar [Thu, 20 Mar 2025 20:48:57 +0000 (20:48 +0000)]
qa/workunits/rbd: update to mirror group snapshot tests
Added new environment variable RBD_MIRROR_GLOBAL_DEBUG to allow additional debug to be turned on
after a cluster has been created.
Updated tests:
• test_group_rename - disabled some checks that are failing as the fix is deferred
• test_empty_group and test_empty_groups - added some snapshot, demote, promote steps
New tests:
• test_remote_namespace - Added new scenarios that test default and non-default
namespaces on the local and remote clusters.
rbd_mirror: fix cross namespace group snapshot mirroring
In case if group belongs to non-default namespace, the mirror daemon on
secondary is setting an empty mirror peer uuid on non primary demote snapshot.
And when previous secondary turns into primary (by an explicit promote request)
the previous i.e. non primary demote snapshot is getting unlinked as part of
the promote request as there is no mirror peer uuid set on it. Because the
previous dependent snapshot is removed, there are split-brain errors leading
to recent test failures.
This fix makesure to set the right mirror peer uuid on the non primary demote
snapshot even if the group belong to non-default namespace.
Ilya Dryomov [Sun, 30 Mar 2025 09:09:33 +0000 (11:09 +0200)]
qa/workunits/rbd: stick to RBD_MIRROR_MODE for mirror image mode
MIRROR_IMAGE_MODE was dropped in favor of RBD_MIRROR_MODE in commit 9b773eec4a8c ("qa/suites/rbd: Cleanup of MIRROR_IMAGE_MODE").
The check in mirror_group_snapshot_and_wait_for_sync_complete() is
redundant because "rbd mirror group snapshot" command would fail anyway
and mirror_group_internal() isn't used.
Ilya Dryomov [Sun, 30 Mar 2025 09:09:33 +0000 (11:09 +0200)]
qa/workunits/rbd: use xmlstarlet directly in rbd_mirror_helpers.sh
Commit e09f04669053 ("qa/workunits/rbd: mirror group functional tests")
mistakenly resurrected a redundant variable which was dropped in commit 4f309603caa3 ("qa: drop XMLSTARLET variable, use xmlstarlet directly").
Ilya Dryomov [Sun, 30 Mar 2025 09:09:33 +0000 (11:09 +0200)]
qa/workunits/rbd: stop_mirror() should send a given signal just once
Commit e09f04669053 ("qa/workunits/rbd: mirror group functional tests")
moved the kill invocation in stop_mirror() from outside of the wait loop
to inside. This is wrong because the signal should be sent just once:
rbd-mirror daemon installs a oneshot handler for SIGINT and SIGTERM and
a subsequent signal immediately kills the process.
The same commit also erroneously changed RBD_MIRROR_INSTANCES default
for all rbd_mirror_helpers.sh users instead of just group tests.
This fixes sporadic failures in "TEST: no blocklists".
librbd/api: don't mask images in group with read-only as part of image_demote()
if the images are part of a group wait until group_demote() is finally done
with GroupUnlinkPeerRequest() and then mask the images part of the group with
IMAGE_READ_ONLY_FLAG_NON_PRIMARY.
Thanks to Nithya for working along for a better fix here.
librbd/api: finalize the API's about skip-quiesce and ignore-quiesce-error flags
* leave --skip-quiesce and --ignore-quiesce-error options only on
rbd mirror group snapshot command
* drop flags argument from mirror_group_enable(), mirror_group_promote() and
mirror_group_demote() APIs, it will remain only on
mirror_group_create_snapshot() and aio_mirror_group_create_snapshot() APIs
* mirror_group_promote() and mirror_group_demote() should behave as if
RBD_SNAP_CREATE_SKIP_QUIESCE flag was passed
* mirror_group_enable() should use get_default_snap_create_flags() to get flags
-- it will be governed by rbd_default_snapshot_quiesce_mode config option
* make each of the mentioned APIs explicitly do either
a) snap_create_flags_api_to_internal(<flags passed by the user>, &snap_create_flags),
b) snap_create_flags_api_to_internal(get_default_snap_create_flags(), &snap_create_flags)
Credits to Ilya Dryomov <idryomov@gmail.com> for the above finalisation.
rbd-mirror: don't call group_snap_set for every image snap for regular group snap
It looks like we fixed avoiding of calling group_snap_set() for each image
snapshot update for mirror group snapshot, but for regular group snapshot,
it is still happening. This commit will fix it.
For 1 & 2 cases, we can simply delete the so far created group snapshot
in the respective callback handler which is basically an empty INCOMPLETE
group snapshot and let the state machine recreate it again later.
For 3 & 4 cases, we are cannot delete the created snapshot, because the
image snapshots whould have synced/syncing by now, deleting the group
snapshot will bring additional comlications (if there is a failover at
the same time). Hence setting m_retry_validate_snap flag in this case,
this would all the rescan even for regular group snapshots, if the
snapshot is yet INCOMPLETE on disk the validate_image_snaps_sync_complete()
will be called again.
For case 5, added logic to retry remove_mirror_peer_uuid() again.
rbd-mirror: fix m_stop_requested leading to a race
* if m_stop_requested is set then is_replay_interrupted return true.
* also shut_down should set m_stop_requested to false, it is instead
setting it to true this will lead to race and a possible crash accessing GR
b/w shut_down() and notify_group_listener_stop()
librbd/api: fail group promote when there is no previous snapshot
If the group enable time initial snapshot didn't sync to the secondary and
is in incomplete state, but then there happens a force promote on secondary,
there is no previous snapshot for that force promote to rollback to.
John Agombar [Thu, 20 Mar 2025 20:48:57 +0000 (20:48 +0000)]
qa/workunits/rbd: update to mirror group snapshot tests
Updated status() helper function to dump contents of stderr and stdout for last command
New helper functions to check for image snaps existence
Added new environment variable RBD_MIRROR_HIDE_BASH_DEBUGGING to turn off set -x output.
Previously RBD_MIRROR_SHOW_CLI_CMD was being used for this and controlling the display of cli output.
New tests:
• test_group_rename - test that a group rename is only mirrored to the remote after a mirror group
snapshot command. Also test that a group rename is not inadvertantly mirrored or undone
(test commented out as it is failing)
• test_enable_mirroring_when_duplicate_group_exists - various scenarios that check an empty group
and approaches to fixing the duplicate names on either site.
(test is commented out as it is not yet finished)
• test_enable_mirroring_when_duplicate_group_and_images_exists - builds on the previous test
but has duplicate named images too (test is commented out as it is failing)
• test_image_snapshots_with_group - test regular image snapshots along with mirror group snapshots
Enabled tests:
- test_force_promote scenarios 1,2,3 and 5 pass
rbd-mirror: group-replayer check for remote demote state
I'm seeing a possibility for 3 situations here for resync flagging and
rbd-mirror daemon working on it:
1. No Demotion on Primary while/just-before resync is play'ed
there is no demote snap along side resync, we can cancel syncing other
snaps, and start resync as soon as resync is flagged, because there is
no point syncing snaps that we are anyway going to delete the whole
group and resync fresh.
2. first Demote + immediately Resync
demote came first, this mean before proceeding with resync, we should
always see if the last remote snap is PRIMARY (validate if the remote
is still primary, which is on point) and only proceed
3. first Resync + immediately Demote
resync Came first, so we head straight to resync.
projects / ceph.git / log