From: Venky Shankar Date: Wed, 3 Feb 2021 05:11:59 +0000 (-0500) Subject: doc: cleanup cephfs-mirroring document X-Git-Tag: v17.1.0~3067^2 X-Git-Url: http://git.apps.os.sepia.ceph.com/?a=commitdiff_plain;h=bf19651d6ad0a11e939c5d9f4bb744e90ad8a9d7;p=ceph.git doc: cleanup cephfs-mirroring document Remove references to in-progress feature, remove internal interfaces section and minor cleanups. Signed-off-by: Venky Shankar --- diff --git a/doc/dev/cephfs-mirroring.rst b/doc/dev/cephfs-mirroring.rst index 75f3bb564a37a..80d1338bba642 100644 --- a/doc/dev/cephfs-mirroring.rst +++ b/doc/dev/cephfs-mirroring.rst @@ -2,10 +2,7 @@ CephFS Mirroring ================ -.. note:: CephFS mirroring feature is currently under development. Interfaces detailed in - this document might change during the development cycle. - -CephFS will support asynchronous replication of snapshots to a remote CephFS file system via +CephFS supports asynchronous replication of snapshots to a remote CephFS file system via `cephfs-mirror` tool. Snapshots are synchronized by mirroring snapshot data followed by creating a snapshot with the same name (for a given directory on the remote file system) as the snapshot being synchronized. @@ -13,16 +10,18 @@ the snapshot being synchronized. Requirements ------------ -The primary (local) and secondary (remote) Ceph clusters version should be Pacific or above. +The primary (local) and secondary (remote) Ceph clusters version should be Pacific or later. Key Idea -------- -For a given snapshot pair in a directory, `cephfs-mirror` daemon uses readdir diff to -identify changes in a directory tree. The diffs are applied to directory in the remote +For a given snapshot pair in a directory, `cephfs-mirror` daemon will rely on readdir diff +to identify changes in a directory tree. The diffs are applied to directory in the remote file system thereby only synchronizing files that have changed between two snapshots. -This feature is tracked here: https://tracker.ceph.com/issues/47034 +This feature is tracked here: https://tracker.ceph.com/issues/47034. + +Currently, snapshot data is synchronized by bulk copying to the remote filesystem. .. note:: Synchronizing hardlinks is not supported -- hardlinked files get synchronized as separate files. @@ -59,60 +58,16 @@ Mirror daemon should be spawned using `systemctl(1)` unit files:: Mirroring Design ---------------- -CephFS will support asynchronous replication of snapshots to a remote CephFS file system +CephFS supports asynchronous replication of snapshots to a remote CephFS file system via `cephfs-mirror` tool. For a given directory, snapshots are synchronized by transferring snapshot data to the remote file system and creating a snapshot with the same name as the snapshot being synchronized. -Change Detection ----------------- - -The first snapshot to get synchronized involves bulk transfer of data to the remotee -file system. Subsequent snapshot synchronization transfers changed/modified files. - -Identifying changed files between two snapshots (one of which is already synchronized), -involves (efficiently) walking two (snapshot) trees. Since one of the snapshots is -already synchronized, scanning the two snapshots locally would suffice. For the most -part, identifying changed files between two snapshot does not involve querying the -remote file system. However, there are cases where it is essential to query the remote -file system: - -Failed Synchronization ----------------------- - -Failure to transfer snapshot data (which could be due to a variety of reasons) leaves -the remote file system with incomplete data. Although retrying synchronization most -likely overcomes transient issues, non-recoverable errors would need to be handled. -When a snapshot synchronization hits a non-recoverable error, the snapshot is noted -for failed synchronization and gets skipped. This is done either manually by the -operator or when the number of consecutive retries (post-failure) hits a configured -limit. - -Skipping a "failed" snapshot leaves incomplete data on the remote file system. When the -next snapshot is chosen for synchronization, local scanning of (two) snapshots is not -feasible, since certain files (which were part of the "failed" snapshot) may have been -transferred to the remote file system but do not exist in the "next" chosen snapshot. -In such a case, scanning the two snapshots "locally" for changes would ignore the -incomplete data on the remote file system left behind. In such a case identifying -changes would need to compare files between the snapshot and the remote file system. -Alternatively, the remote file system data could be purged followed by bulk transfer of -data from the next chosen snapshot. Subsequent snapshots can then rely on scanning local -snapshots for identifying changes to be transferred. - -Deleted Snapshots ------------------ - -Snapshots on the primary file system can be deleted even during synchronization. -This needs to be handled by the mirror daemon and poses the same issues as mention -above. Moreover, mirroring can be disabled in midst of snapshots being synchronized -and enabled after deleting snapshots (including the one under synchronization) and -poses the same set of issues. - Snapshot Synchronization Order ------------------------------ Although the order in which snapshots get chosen for synchronization does not matter, -it may be beneficial to pick snapshots based on creation order (using snap-id). +snapshots are picked based on creation order (using snap-id). Snapshot Incarnation -------------------- @@ -131,48 +86,7 @@ Interfaces `Mirroring` module (manager plugin) provides interfaces for managing directory snapshot mirroring. Manager interfaces are (mostly) wrappers around monitor commands for managing -file system mirroring and is the recommended control interface. See `Internal Interfaces` -(monitor commands) are detailed below. - -.. note:: `Mirroring` module is work in progress and the interface is detailed in the - `Mirroring Module and Interface` section. - -Internal Interfaces -------------------- - -Mirroring needs be enabled for a Ceph file system for `cephfs-mirror` daemons to start -mirroring directory snapshots:: - - $ ceph fs mirror enable - -Once mirroring is enabled, mirror peers can be added. `cephfs-mirror` daemon mirrors -directory snapshots to mirror peers. A mirror peer is represented as a client name -and cluster name tuple. To add a mirror peer:: - - $ ceph fs mirror peer_add - -Each peer is assigned a unique identifier which can be fetched via `ceph fs dump` or -`ceph fs get` commands as below:: - - $ ceph fs get - ... - ... - [peers={uuid=e3739ebf-dbce-460a-bf9c-c66b57697c9a, remote_cluster={client_name=client.site-a, cluster_name=site-a, fs_name=backup}}] - -To remove a mirror peer use the following:: - - $ ceph fs mirror peer_remove - -Mirroring can be disabled for a Ceph file system with:: - - $ ceph fs mirror disable - -Mirror status (enabled/disabled) and file system mirror peers are persisted in `FSMap`. -This enables any entity in a Ceph cluster to subscribe to `FSMap` updates and get -notified about changes in mirror status and/or peers. `cephfs-mirror` daemon subscribes -to `FSMap` and gets notified on mirror status and/or peer updates. Peer changes are -handled by starting or stopping mirroring to when a new peer is added or an existing peer -is removed. +file system mirroring and is the recommended control interface. Mirroring Module and Interface ------------------------------ @@ -380,8 +294,7 @@ information. To check which mirror daemon a directory has been mapped to use:: "state": "mapped" } -.. note:: `instance_id` is the RAODS instance-id associated with a mirror daemon. This - will include the RADOS instance address to help map to a mirror daemon. +.. note:: `instance_id` is the RAODS instance-id associated with a mirror daemon. Other information such as `state` and `last_shuffled` are interesting when running multiple mirror daemons. @@ -410,5 +323,4 @@ a peer to the same primary file system it was earlier synchronized from. Feature Status -------------- -`cephfs-mirror` daemon is built by default (follows `WITH_CEPHFS` CMake rule). However, the -feature is in development phase. +`cephfs-mirror` daemon is built by default (follows `WITH_CEPHFS` CMake rule).