CephFS Mirroring
================
-.. note:: CephFS mirroring feature is currently under development. Interfaces detailed in
- this document might change during the development cycle.
-
-CephFS will support asynchronous replication of snapshots to a remote CephFS file system via
+CephFS supports asynchronous replication of snapshots to a remote CephFS file system via
`cephfs-mirror` tool. Snapshots are synchronized by mirroring snapshot data followed by
creating a snapshot with the same name (for a given directory on the remote file system) as
the snapshot being synchronized.
Requirements
------------
-The primary (local) and secondary (remote) Ceph clusters version should be Pacific or above.
+The primary (local) and secondary (remote) Ceph clusters version should be Pacific or later.
Key Idea
--------
-For a given snapshot pair in a directory, `cephfs-mirror` daemon uses readdir diff to
-identify changes in a directory tree. The diffs are applied to directory in the remote
+For a given snapshot pair in a directory, `cephfs-mirror` daemon will rely on readdir diff
+to identify changes in a directory tree. The diffs are applied to directory in the remote
file system thereby only synchronizing files that have changed between two snapshots.
-This feature is tracked here: https://tracker.ceph.com/issues/47034
+This feature is tracked here: https://tracker.ceph.com/issues/47034.
+
+Currently, snapshot data is synchronized by bulk copying to the remote filesystem.
.. note:: Synchronizing hardlinks is not supported -- hardlinked files get synchronized
as separate files.
Mirroring Design
----------------
-CephFS will support asynchronous replication of snapshots to a remote CephFS file system
+CephFS supports asynchronous replication of snapshots to a remote CephFS file system
via `cephfs-mirror` tool. For a given directory, snapshots are synchronized by transferring
snapshot data to the remote file system and creating a snapshot with the same name as the
snapshot being synchronized.
-Change Detection
-----------------
-
-The first snapshot to get synchronized involves bulk transfer of data to the remotee
-file system. Subsequent snapshot synchronization transfers changed/modified files.
-
-Identifying changed files between two snapshots (one of which is already synchronized),
-involves (efficiently) walking two (snapshot) trees. Since one of the snapshots is
-already synchronized, scanning the two snapshots locally would suffice. For the most
-part, identifying changed files between two snapshot does not involve querying the
-remote file system. However, there are cases where it is essential to query the remote
-file system:
-
-Failed Synchronization
-----------------------
-
-Failure to transfer snapshot data (which could be due to a variety of reasons) leaves
-the remote file system with incomplete data. Although retrying synchronization most
-likely overcomes transient issues, non-recoverable errors would need to be handled.
-When a snapshot synchronization hits a non-recoverable error, the snapshot is noted
-for failed synchronization and gets skipped. This is done either manually by the
-operator or when the number of consecutive retries (post-failure) hits a configured
-limit.
-
-Skipping a "failed" snapshot leaves incomplete data on the remote file system. When the
-next snapshot is chosen for synchronization, local scanning of (two) snapshots is not
-feasible, since certain files (which were part of the "failed" snapshot) may have been
-transferred to the remote file system but do not exist in the "next" chosen snapshot.
-In such a case, scanning the two snapshots "locally" for changes would ignore the
-incomplete data on the remote file system left behind. In such a case identifying
-changes would need to compare files between the snapshot and the remote file system.
-Alternatively, the remote file system data could be purged followed by bulk transfer of
-data from the next chosen snapshot. Subsequent snapshots can then rely on scanning local
-snapshots for identifying changes to be transferred.
-
-Deleted Snapshots
------------------
-
-Snapshots on the primary file system can be deleted even during synchronization.
-This needs to be handled by the mirror daemon and poses the same issues as mention
-above. Moreover, mirroring can be disabled in midst of snapshots being synchronized
-and enabled after deleting snapshots (including the one under synchronization) and
-poses the same set of issues.
-
Snapshot Synchronization Order
------------------------------
Although the order in which snapshots get chosen for synchronization does not matter,
-it may be beneficial to pick snapshots based on creation order (using snap-id).
+snapshots are picked based on creation order (using snap-id).
Snapshot Incarnation
--------------------
`Mirroring` module (manager plugin) provides interfaces for managing directory snapshot
mirroring. Manager interfaces are (mostly) wrappers around monitor commands for managing
-file system mirroring and is the recommended control interface. See `Internal Interfaces`
-(monitor commands) are detailed below.
-
-.. note:: `Mirroring` module is work in progress and the interface is detailed in the
- `Mirroring Module and Interface` section.
-
-Internal Interfaces
--------------------
-
-Mirroring needs be enabled for a Ceph file system for `cephfs-mirror` daemons to start
-mirroring directory snapshots::
-
- $ ceph fs mirror enable <fsname>
-
-Once mirroring is enabled, mirror peers can be added. `cephfs-mirror` daemon mirrors
-directory snapshots to mirror peers. A mirror peer is represented as a client name
-and cluster name tuple. To add a mirror peer::
-
- $ ceph fs mirror peer_add <fsname> <client@cluster> <remote_fsname>
-
-Each peer is assigned a unique identifier which can be fetched via `ceph fs dump` or
-`ceph fs get` commands as below::
-
- $ ceph fs get <fsname>
- ...
- ...
- [peers={uuid=e3739ebf-dbce-460a-bf9c-c66b57697c9a, remote_cluster={client_name=client.site-a, cluster_name=site-a, fs_name=backup}}]
-
-To remove a mirror peer use the following::
-
- $ ceph fs mirror peer_remove <uuid>
-
-Mirroring can be disabled for a Ceph file system with::
-
- $ ceph fs mirror disable <fsname>
-
-Mirror status (enabled/disabled) and file system mirror peers are persisted in `FSMap`.
-This enables any entity in a Ceph cluster to subscribe to `FSMap` updates and get
-notified about changes in mirror status and/or peers. `cephfs-mirror` daemon subscribes
-to `FSMap` and gets notified on mirror status and/or peer updates. Peer changes are
-handled by starting or stopping mirroring to when a new peer is added or an existing peer
-is removed.
+file system mirroring and is the recommended control interface.
Mirroring Module and Interface
------------------------------
"state": "mapped"
}
-.. note:: `instance_id` is the RAODS instance-id associated with a mirror daemon. This
- will include the RADOS instance address to help map to a mirror daemon.
+.. note:: `instance_id` is the RAODS instance-id associated with a mirror daemon.
Other information such as `state` and `last_shuffled` are interesting when running
multiple mirror daemons.
Feature Status
--------------
-`cephfs-mirror` daemon is built by default (follows `WITH_CEPHFS` CMake rule). However, the
-feature is in development phase.
+`cephfs-mirror` daemon is built by default (follows `WITH_CEPHFS` CMake rule).