John Agombar [Tue, 4 Mar 2025 14:24:43 +0000 (14:24 +0000)]
qa/workunits/rbd: update to mirror group snapshot tests
New tests:
- force promote test with daemon running on both clusters
- test_enable_mirroring_when_duplicate_group_exists
- test_odf_failover_failback test
- test_resync_marker test
- test_force_promote_before_initial_sync test
- scenarios in test_create_group_with_images_then_mirror_with_regular_snapshots
Renamed tests:
- test_multiple_user_snapshot_time to test_multiple_mirror_group_snapshot_unlink_time
- test_multiple_user_snapshot_whilst_stopped to test_multiple_mirror_group_snapshot_whilst_stopped
* moved the internal functions scope to private
* use m_on_start_finish to save the init time Context and use later,
instead of passing it in various functions
N Balachandran [Tue, 11 Mar 2025 10:19:16 +0000 (15:49 +0530)]
rbd-mirror: reuse the ImageReplayers in the GroupReplayer
This fix will start image replayers even if the group replayer
is primary so as to have the correctmirror pool status.
The group replayer will also attempt to reuse the image replayers where
possible on restart.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
N Balachandran [Fri, 7 Mar 2025 07:26:35 +0000 (12:56 +0530)]
rbd-mirror: fix image map notifications for groups
The Group replayer Bootstrap now sends mirroring notifications
when creating or deleting the local group. The ImageMap will only
send the acquire_group notifications once for each group.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
Ramana Raja [Wed, 5 Mar 2025 19:29:16 +0000 (14:29 -0500)]
librbd/mirror: cleanup redundant parameters in CreatePrimaryRequest and
... CreateNonPrimaryRequest constructors. The objects can figure out
the image's group ID and group pool ID from the group_spec stored in
their image_ctx data member. No need to pass in group ID and
group pool ID into the constructors.
Ramana Raja [Tue, 4 Mar 2025 17:43:19 +0000 (12:43 -0500)]
librbd/mirror: change naming format of member image snap
... of primary and non-primary mirror group snaps.
Set the naming format of member image snap of a mirror group snap to be,
mirror.primary.<global_image_id>.<global_group_id>.<group_pool_id>_<group_id>_<group_snap_id>,
or
mirror.non_primary.<global_image_id>.<global_group_id>.<group_pool_id>_<group_id>_<group_snap_id>
Ramana Raja [Fri, 28 Feb 2025 21:49:27 +0000 (16:49 -0500)]
librbd/api: set `image_snap_name` as empty string for mirror gp snap
The member image snapshots of a mirror group snap do not share a
common name unlike those of a user group snap. So set the
`image_snap_name` to an empty string.
The following steps leaves stale group on seondary left undeleted,
1. Create and mirror enable a group with 2 images.
2. Let it sync to the secondary
3. Demote on the primary and promote on the secondary
4. Wait until it starts replaying on the original primary
5. Delete the group on the new primary
Credits to Nithya Balachandran for highlighting the issue with detailed steps.
Following issues are fixed:
* Allow to flag resync, but if remote is not primary do not resync or
even delete the local group.
- Wait for remote to turn primary, if it turns primary, then continue
to resync.
- Just in case if the same site is made primary right after issuing
resync, then clear that flag immediatly.
* Revert some old code in PoolWatcher, unintentional edits/changes.
* Do not send MIRROR_GROUP_STATE_DISABLED notification from group_resync
API, this will lead to release_group(). Credits to Nithya Balachandran
for pointing about this notification deatils.
rbd_mirror: avoid rescans in busy loop to detect new snapshots
Instead move the state to STATE_IDLE once the snapshot limits cannot be met
and move back to STATE_REPLAYING on a call from group_replayer to
set_remote_snap_id_end_limit()
Issue I:
As part of the remove_local_mirror_group if local mirror group global_group_id
doesn't match with GroupReplayer instance (m_global_group_id), then
remove_local_group()
In a case where the daemon is down then group is disabled then removed/added
images then re-enabled groups and then brought the daemon back to life,
the Groupreplayer instances belonging to same group name will mess
leading to path of create_local_mirror_group(), which is wrong.
Issue II:
Also, in cases where local mirror group global_group_id doesn't match with
GroupReplayer instance, if there are ENOENT errors in the bootstrapping
then retry the bootstrapping.
With out this fix, this will lead to group_replayer destroy of a valid instance.
Ramana Raja [Thu, 20 Feb 2025 00:09:32 +0000 (19:09 -0500)]
librbd/api: disallow mirror image operations on a group's member image
Disallow the following mirror image APIs when called directly on an
image that is member of a group:
- mirror image demote
- mirror image disable
- mirror image enable
- mirror image promote
- mirror image resync
- mirror image snapshot
Only allow mirror operations on a group's member image via the mirror
group APIs.
John Agombar [Thu, 13 Feb 2025 17:20:01 +0000 (17:20 +0000)]
qa/workunits/rbd: add new tests and improve existing
Change admin socket mirror group status checks to query status on normal CLI too
Disable test_stopped_daemon test whichhas intermittent failures.
Remove sleep 5 which is no longer needed in RBD mirror group tests
Fix test_image_replay_state() helper function to work without SHOW_CLI_CMD env variable set
rbd_mirror_group_group.sh:
+ testlog "TEST: add a large image to group and test replay"
Also this fix replace the set_finished(), which was removed in the previous
commit, which will cause a regression in the GroupReplayer destroy code path.
N Balachandran [Fri, 21 Feb 2025 06:17:19 +0000 (11:47 +0530)]
rbd-mirror: fixes multiple issues in the group replayer
The commit includes the following:
- Fixed crashes in the start/stop in GroupReplayer
- Fixed crashes in the shut_down sequence in group_replayer::Replayer
- ImageMap will now send release_group notifications for non-empty
groups.
- InstanceReplayer no longer checks if the GroupReplayer needs to be
restarted. The GroupReplayer will stop itself if it determines that it
needs to be restarted.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
Ramana Raja [Tue, 4 Feb 2025 00:55:08 +0000 (19:55 -0500)]
librbd: remove mirror APIs that change mirror group membership
Remove mirror APIs, group_image_add() and group_image_remove() that
are never called as we don't allow adding/removing images to/from a
mirrored group.
rbd-mirror: do not move the images to trash while the disabling is in progress
Images cannot be moved to trash if the state is disabling because its a
transient state where some of the images might have got the oportunity to
disable and some of them part of the group might still be enabled
waiting for the oportunity while a group disable is in progress.
So we wait until the state DISABLING moves to next state, and see if there are
any stale image to move into a trash queue later.
rbd-mirror: bootstrap wait for previous disabling group to cleanup
Was seeing a case where the following operations are done:
1. daemon is stopped on secondary
2. then mirroring on the group is disabled
3. added/removed image[s] to/from the group
4. enabled group back for mirroring
5. Mirroring daemon is brought back to life
From the handling:
1. Two GroupReplayer's are started by the InstanceReplayer, one for old group
and one for new group (not surprisingly both deal with the same pool images)
2. The GroupReplayer for old group instance enters into
group_replayer::BootstrapRequest, notices remote_group_id is not found, and
starts cleaning-up the group, """tries to remove local group and all the
images. Finally returns to GroupReplayer, stop the GroupReplayer setting
the state as stopped with description group removed and finally unregister
admin socket hook."""
3. On the other hand the GroupReplayer for new group instance runs in concurrent
to the old one, figures out local group_id by name exists and """tries to
remove local group and all the images. Finally returns to GroupReplayer,
stop the GroupReplayer setting the state as stopped with description group
removed and finally unregister admin socket hook."""
You can see 2 and 3 are ending up in the same situation because of the
concurrent behaviour. i.e one has to add the group with a name and create
images in the pool. Where as the other has to remove the group with same name
from the same pool.
Thanks to Ilya for the suggestion here, according to the suggestion the
fix is simple. The way this is handled for standalone images is that the
second replayer (i.e. (3)) sees that the image is in MIRROR_IMAGE_STATE_DISABLING
state and backs off (i.e.second group waits and retries later).
If the second replayer backs off with ERESTART, the first replayer should
eventually clean up the old group which would allow the second replayer to
proceed with creating a new group.
rbd_mirror: cleanup group status keys in the rbd_mirroring object
Keys & Values for "gremote_status_global_*" and "gstatus_global_*" are
getting readded in the rbd_mirroring object after they were removed at
group disable time as part of group_status_set(), as it doesn't defend
for disabled groups today.
Also librbd::cls_client::mirror_group_status_remove_down() was added in
the code but not levearaged, hence added code for unhappy path cleanup
as part of MirrorStatusWatcher::init() like it is calls
librbd::cls_client::mirror_image_status_remove_down() today.
rbd_mirror: update local {group_pool, group_pool} in ImageSnapshotNamespaceGroup
* Fix user group snapshot not moving to complete when the pool_id differ on
remote and locally.
* The Image snapshot namespace ImageSnapshotNamespaceGroup is copied
from remote src directly to dst locally and {group_pool, group_pool}
still hold remote details. This fix updates the namespace in the
image snapshot.
N Balachandran [Mon, 10 Feb 2025 15:10:07 +0000 (20:40 +0530)]
librbd: fix group snapshot unlink
Changes to the group mirror snapshot unlink:
- Fixes the group mirror snapshot unlink to behave like the
image mirror unlink.
- Renames UnlinkGroupPeerRequest to GroupUnlinkPeerRequest
and moves it into librbd/mirror/snapshot.
- Modifies prepare_group_images() to return the mirror_peer_uuids
which are then passed as an argument to GroupUnlinkPeerRequest.
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
John Agombar [Thu, 6 Feb 2025 12:03:36 +0000 (12:03 +0000)]
qa/workunits/rbd: improvements to smoke tests
- Remove dynamic group behaviour in rbd_mirror_group.sh tests
- Add test for group enable/disable after force promote
- Test new fields in group info cmd
rbd-mirror: add peer_uuids for non-primary demoted group snapshot
GroupReplayer should add peer uuid for group snapshot if it is a
non-primary demoted snapshot, other wise this snapshot will be
unconditionally unlinked further, as doesn't have a peer uuid leading to
split-brain scenarios.
Credits to N Balachandran <nithya.balachandran@ibm.com> for the find.
John Agombar [Thu, 30 Jan 2025 13:04:13 +0000 (13:04 +0000)]
qa/workunits/rbd: updates to mirror group bash scripts
- support cli parameters to specify the test to run
- support cli parameter to specify the number of times to repeat the test
- new tests
- added RBD_MIRROR_NEW_IMPLICIT_BEHAVIOUR env variable in preparation for
changes to group snapshot behaviour
1. If group snapshot is syncing do not remove mirror peer uuid of last complete
snapshot [ i.e. currently incomlpete - 1] (on remote)
2. If group snapshot is synced do not remove mirror peer uuid of its
respective on remote yet.
Creating a global, pool or namespace level mirror snapshot
schedule shouldn't schedule mirror snapshots of images that
are part of a group and reside in the pool or namespace.
Also disallow directly scheduling mirror image snapshots on
images that are part of a group.
Ramana Raja [Tue, 5 Nov 2024 16:12:54 +0000 (11:12 -0500)]
qa/workunits/rbd: add basic tests for mirror group snapshot scheduler
Add tests to check the basic functionality of the
mirror_group_snapshot_schedule module. Check that
- `rbd mirror group snapshot schedule add/rm/status/ls` commands work
- the module can recover from blocklisting of its client and continue
to process requests
rbd-mirror: more improvements in the group replayer
- Arresting the interruptions after shutdown
- Restart GroupReplayer if bootstrap returns remote group id as empty
- Fix a test failure in "regular group snapshots test"
rbd-mirror: create group snapshots only if remote group snapshot is complete
Also do not create a regular group snapshot until the next mirror group
snapshot is created. Otherwise we simply have an incomplete regular group
snapshot until the next mirror group snapshot is created.
* fix braces in the imageMap update_images_added & update_images_removed
* do not allow image add from non-primary
* `down+unknown` status shown on querying individual images which are part
of group enabled for mirroring
* `mirror pool status` shows down+unknown status
* fix imageMap being overwritten when multiple images are enabled for mirroring
* fix misleading error msg when getting status of a non-mirror enabled group
$ rbd --cluster site-a group snap ls pool1/test_group --debug-rbd=0
NAME STATUS
group_snap1 ok
group_snap2 ok
.mirror.2_10416b8b4567_10536b8b4567 ok
$ rbd --cluster site-b group snap ls pool1/test_group --debug-rbd=0
NAME STATUS
.mirror.2_10376b8b4567_1037327b23c6 ok
group_snap1 ok
group_snap2 ok
.mirror.2_10416b8b4567_10536b8b4567 ok
rbd-mirror: add undo code, exclusive locking and quiescing
* add essential logic to undo partially succeeded API's like, group promote,
group demote, group enable, group disable, group image add and
group image remove
* add exclusive locking and quiescing with-in all the required group API's
* adress code duplication and optimization with in the group API's
Make sure group snapshots doesn't get copied to secondary if the group snapshot
is incomplete on primary. On creation time of a group snapshots on primary,
make sure to delete the previous snapshot in case it is incomplete.
projects / ceph.git / log