From 0376ce721a1854a66b4ac6547cc3ac2b655dfa9f Mon Sep 17 00:00:00 2001 From: Patrick Donnelly Date: Tue, 8 Dec 2020 18:57:44 -0800 Subject: [PATCH] doc: add documentation for new multifs feature Signed-off-by: Patrick Donnelly --- PendingReleaseNotes | 5 ++ doc/cephfs/client-auth.rst | 2 + doc/cephfs/experimental-features.rst | 68 ---------------------------- doc/cephfs/index.rst | 3 +- doc/cephfs/multifs.rst | 54 ++++++++++++++++++++++ 5 files changed, 63 insertions(+), 69 deletions(-) create mode 100644 doc/cephfs/multifs.rst diff --git a/PendingReleaseNotes b/PendingReleaseNotes index 3db77906e1fa..af1c10f1843f 100644 --- a/PendingReleaseNotes +++ b/PendingReleaseNotes @@ -26,6 +26,11 @@ ``osd_scrub_end_week_day`` are 0 - 6. The use of 7 is now illegal. Specifying ``0`` for both values causes every day of the week to be allowed. +* Multiple file systems in a single Ceph cluster is now stable. New Ceph clusters + enable support for multiple file systems by default. Existing clusters + must still set the "enable_multiple" flag on the fs. Please see the CephFS + documentation for more information. + * A new health check, DAEMON_OLD_VERSION, will warn if different versions of Ceph are running on daemons. It will generate a health error if multiple versions are detected. This condition must exist for over mon_warn_older_version_delay (set to 1 week by default) in order for the diff --git a/doc/cephfs/client-auth.rst b/doc/cephfs/client-auth.rst index 3e8a791d3f58..fd0faa83963a 100644 --- a/doc/cephfs/client-auth.rst +++ b/doc/cephfs/client-auth.rst @@ -149,6 +149,8 @@ prefix length in CIDR notation (e.g., ``10.3.0.0/16``). If present, the use of this capability is restricted to clients connecting from this network. +.. _fs-authorize-multifs: + File system Information Restriction =================================== diff --git a/doc/cephfs/experimental-features.rst b/doc/cephfs/experimental-features.rst index 74aa6917d2ed..ba60d12c79ab 100644 --- a/doc/cephfs/experimental-features.rst +++ b/doc/cephfs/experimental-features.rst @@ -35,76 +35,8 @@ protect the mechanisms for balancing load (migration, replication, fragmentation) but stub out the balancing policies using Lua. For details, see :doc:`/cephfs/mantle`. -Snapshots ---------- -Like multiple active MDSes, CephFS is designed from the ground up to support -snapshotting of arbitrary directories. There are no known bugs at the time of -writing, but there is insufficient testing to provide stability guarantees and -every expansion of testing has generally revealed new issues. If you do enable -snapshots and experience failure, manual intervention will be needed. - -Snapshots are known not to work properly with multiple file systems (below) in -some cases. Specifically, if you share a pool for multiple FSes and delete -a snapshot in one FS, expect to lose snapshotted file data in any other FS using -snapshots. See the :doc:`/dev/cephfs-snapshots` page for more information. - -For somewhat obscure implementation reasons, the kernel client only supports up -to 400 snapshots (http://tracker.ceph.com/issues/21420). - -Snapshotting was blocked off with the ``allow_new_snaps`` flag prior to Mimic. - -Multiple File Systems within a Ceph Cluster -------------------------------------------- -Code was merged prior to the Jewel release which enables administrators -to create multiple independent CephFS file systems within a single Ceph cluster. -These independent file systems have their own set of active MDSes, cluster maps, -and data. But the feature required extensive changes to data structures which -are not yet fully qualified, and has security implications which are not all -apparent nor resolved. - -There are no known bugs, but any failures which do result from having multiple -active file systems in your cluster will require manual intervention and, so -far, will not have been experienced by anybody else -- knowledgeable help will -be extremely limited. You also probably do not have the security or isolation -guarantees you want or think you have upon doing so. - -Note that snapshots and multiple file systems are *not* tested in combination -and may not work together; see above. - -Multiple file systems were available starting in the Jewel release candidates -but must be turned on via the ``enable_multiple`` flag until declared stable. - LazyIO ------ LazyIO relaxes POSIX semantics. Buffered reads/writes are allowed even when a file is opened by multiple applications on multiple clients. Applications are responsible for managing cache coherency themselves. - -Previously experimental features -================================ - -Directory Fragmentation ------------------------ - -Directory fragmentation was considered experimental prior to the *Luminous* -(12.2.x). It is now enabled by default on new file systems. To enable -directory fragmentation on file systems created with older versions of Ceph, -set the ``allow_dirfrags`` flag on the file system:: - - ceph fs set allow_dirfrags 1 - -Multiple active metadata servers --------------------------------- - -Prior to the *Luminous* (12.2.x) release, running multiple active metadata -servers within a single file system was considered experimental. Creating -multiple active metadata servers is now permitted by default on new -file systems. - -File Systems created with older versions of Ceph still require explicitly -enabling multiple active metadata servers as follows:: - - ceph fs set allow_multimds 1 - -Note that the default size of the active mds cluster (``max_mds``) is -still set to 1 initially. diff --git a/doc/cephfs/index.rst b/doc/cephfs/index.rst index 7784b1f2ee8e..6064321b7ce1 100644 --- a/doc/cephfs/index.rst +++ b/doc/cephfs/index.rst @@ -77,7 +77,8 @@ Administration Create a CephFS file system Administrative commands - Provision/Add/Remove MDS(s) + Creating Multiple File Systems + Provision/Add/Remove MDS(s) MDS failover and standby configuration MDS Cache Configuration MDS Configuration Settings diff --git a/doc/cephfs/multifs.rst b/doc/cephfs/multifs.rst new file mode 100644 index 000000000000..2dcba7ae006a --- /dev/null +++ b/doc/cephfs/multifs.rst @@ -0,0 +1,54 @@ +.. _cephfs-multifs: + +Multiple Ceph File Systems +========================== + + +Beginning with the Pacific release, multiple file system support is stable +and ready to use. This functionality allows configuring separate file systems +with full data separation on separate pools. + +Existing clusters must set a flag to enable multiple file systems:: + + ceph fs flag set enable_multiple true + +New Ceph clusters automatically set this. + + +Creating a new Ceph File System +------------------------------- + +The new ``volumes`` plugin interface (see: :doc:`/cephfs/fs-volumes`) automates +most of the work of configuring a new file system. The "volume" concept is +simply a new file system. This can be done via:: + + ceph fs volume create + +Ceph will create the new pools and automate the deployment of new MDS to +support the new file system. The deployment technology used, e.g. cephadm, will +also configure the MDS affinity (see: :ref:`mds-join-fs`) of new MDS daemons to +operate the new file system. + + +Securing access +--------------- + +The ``fs authorize`` command allows configuring the client's access to a +particular file system. See also in :ref:`fs-authorize-multifs`. The client will +only have visibility of authorized file systems and the Monitors/MDS will +reject access to clients without authorization. + + +Other Notes +----------- + +Multiple file systems do not share pools. This is particularly important for +snapshots but also because no measures are in place to prevent duplicate +inodes. The Ceph commands prevent this dangerous configuration. + +Each file system has its own set of MDS ranks. Consequently, each new file +system requires more MDS daemons to operate and increases operational costs. +This can be useful for increasing metadata throughput by application or user +base but also adds cost to the creation of a file system. Generally, a single +file system with subtree pinning is a better choice for isolating load between +applications. -- 2.47.3