From: Patrick Donnelly Date: Fri, 13 Apr 2018 23:48:36 +0000 (-0700) Subject: doc: update upgrade procedure and release notes X-Git-Tag: v13.1.0~200^2~4 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=4e9ffffb95156f392926abc63bbff637d3bcba70;p=ceph.git doc: update upgrade procedure and release notes Signed-off-by: Patrick Donnelly --- diff --git a/PendingReleaseNotes b/PendingReleaseNotes index 7bbeff7f810e..f6505ca6d2ba 100644 --- a/PendingReleaseNotes +++ b/PendingReleaseNotes @@ -27,8 +27,8 @@ - mds stop -> mds deactivate - mds set_max_mds -> fs set max_mds - mds set -> fs set - - mds cluster_down -> fs set cluster_down true - - mds cluster_up -> fs set cluster_down false + - mds cluster_down -> fs set joinable false + - mds cluster_up -> fs set joinable true - mds add_data_pool -> fs add_data_pool - mds remove_data_pool -> fs rm_data_pool - mds rm_data_pool -> fs rm_data_pool @@ -46,12 +46,12 @@ is now redundant. * Taking a CephFS cluster down is now done by setting the down flag which - deactivates all MDS. + deactivates all MDS. For example: `ceph fs set cephfs down true`. - * Preventing standbys from joining as new actives (formerly the cluster_down - flag) on a file system is now accomplished by setting the joinable flag. - This is useful mostly for testing so that a file system may be quickly - brought down and deleted. + * Preventing standbys from joining as new actives (formerly the now + deprecated cluster_down flag) on a file system is now accomplished by + setting the joinable flag. This is useful mostly for testing so that a + file system may be quickly brought down and deleted. * New CephFS file system attributes session_timeout and session_autoclose are configurable via `ceph fs set`. The MDS config options diff --git a/doc/cephfs/administration.rst b/doc/cephfs/administration.rst index 76861d804b9b..3c4806876bd4 100644 --- a/doc/cephfs/administration.rst +++ b/doc/cephfs/administration.rst @@ -76,25 +76,60 @@ to enumerate the objects during operations like stats or deletes. Taking the cluster down ----------------------- -Taking a CephFS cluster down is done by setting the cluster_down flag: + +Taking a CephFS cluster down is done by setting the down flag: :: - mds set cluster_down true + mds set down true To bring the cluster back online: :: - mds set cluster_down false + mds set down false + +This will also restore the previous value of max_mds. MDS daemons are brought +down in a way such that journals are flushed to the metadata pool and all +client I/O is stopped. + + +Taking the cluster down rapidly for deletion or disaster recovery +----------------------------------------------------------------- + +To allow rapidly deleting a file system (for testing) or to quickly bring MDS +daemons down, the operator may also set a flag to prevent standbys from +activating on the file system. This is done using the ``joinable`` flag: + +:: + + fs set joinable false + +Then the operator can fail all of the ranks which causes the MDS daemons to +respawn as standbys. The file system will be left in a degraded state. + +:: -This will restore the previous value of max_mds. + # For all ranks, 0-N: + mds fail : + +Once all ranks are inactive, the file system may also be deleted or left in +this state for other purposes (perhaps disaster recovery). Daemons ------- -These commands act on specific mds daemons or ranks. +Most commands manipulating MDSs take a ```` argument which can take one +of three forms: + +:: + + : + : + + +Comamnds to manipulate MDS daemons: :: @@ -109,29 +144,13 @@ If the MDS daemon was in reality still running, then using ``mds fail`` will cause the daemon to restart. If it was active and a standby was available, then the "failed" daemon will return as a standby. -:: - - mds deactivate - -Deactivate an MDS, causing it to flush its entire journal to -backing RADOS objects and close all open client sessions. Deactivating an MDS -is primarily intended for bringing down a rank after reducing the number of -active MDS (max_mds). Once the rank is deactivated, the MDS daemon will rejoin the -cluster as a standby. -```` can take one of three forms: :: - : - : - - -Use ``mds deactivate`` in conjunction with adjustments to ``max_mds`` to -shrink an MDS cluster. See :doc:`/cephfs/multimds` - -:: + tell mds. command ... - tell mds. +Send a command to the MDS daemon(s). Use ``mds.*`` to send a command to all +daemons. Use ``ceph tell mds.* help`` to learn available commands. :: @@ -209,5 +228,5 @@ These legacy commands are obsolete and no longer usable post-Luminous. mds remove_data_pool # replaced by "fs rm_data_pool" mds set # replaced by "fs set" mds set_max_mds # replaced by "fs set max_mds" - mds stop # replaced by "mds deactivate" + mds stop # obsolete diff --git a/doc/cephfs/multimds.rst b/doc/cephfs/multimds.rst index 55ef152356cb..c0b81e0c14fd 100644 --- a/doc/cephfs/multimds.rst +++ b/doc/cephfs/multimds.rst @@ -27,17 +27,13 @@ are those with many clients, perhaps working on many separate directories. Increasing the MDS active cluster size ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Each CephFS filesystem has a *max_mds* setting, which controls -how many ranks will be created. The actual number of ranks -in the filesystem will only be increased if a spare daemon is -available to take on the new rank. For example, if there is only one MDS daemon running, and max_mds is set to two, no second rank will be created. - -Before ``max_mds`` can be increased, the ``allow_multimds`` flag must be set. -The following command sets this flag for a filesystem instance. - -:: - - # ceph fs set allow_multimds true --yes-i-really-mean-it +Each CephFS filesystem has a *max_mds* setting, which controls how many ranks +will be created. The actual number of ranks in the filesystem will only be +increased if a spare daemon is available to take on the new rank. For example, +if there is only one MDS daemon running, and max_mds is set to two, no second +rank will be created. (Note that such a configuration is not Highly Available +(HA) because no standby is available to take over for a failed rank. The +cluster will complain via health warnings when configured this way.) Set ``max_mds`` to the desired number of ranks. In the following examples the "fsmap" line of "ceph status" is shown to illustrate the expected @@ -63,7 +59,7 @@ requires standby daemons** to take over if any of the servers running an active daemon fail. Consequently, the practical maximum of ``max_mds`` for highly available systems -is one less than the total number of MDS servers in your system. +is at most one less than the total number of MDS servers in your system. To remain available in the event of multiple server failures, increase the number of standby daemons in the system to match the number of server failures @@ -72,49 +68,35 @@ you wish to withstand. Decreasing the number of ranks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -All ranks, including the rank(s) to be removed must first be active. This -means that you must have at least max_mds MDS daemons available. - -First, set max_mds to a lower number, for example we might go back to -having just a single active MDS: +Reducing the number of ranks is as simple as reducing ``max_mds``: :: # fsmap e9: 2/2/2 up {0=a=up:active,1=c=up:active}, 1 up:standby ceph fs set max_mds 1 - # fsmap e10: 2/2/1 up {0=a=up:active,1=c=up:active}, 1 up:standby - -Note that we still have two active MDSs: the ranks still exist even though -we have decreased max_mds, because max_mds only restricts creation -of new ranks. - -Next, use the ``ceph mds deactivate `` command to remove the -unneeded rank: - -:: - - ceph mds deactivate cephfs_a:1 - telling mds.1:1 172.21.9.34:6806/837679928 to deactivate + # fsmap e10: 2/2/1 up {0=a=up:active,1=c=up:stopping}, 1 up:standby + # fsmap e10: 2/2/1 up {0=a=up:active,1=c=up:stopping}, 1 up:standby + ... + # fsmap e10: 1/1/1 up {0=a=up:active}, 2 up:standby - # fsmap e11: 2/2/1 up {0=a=up:active,1=c=up:stopping}, 1 up:standby - # fsmap e12: 1/1/1 up {0=a=up:active}, 1 up:standby - # fsmap e13: 1/1/1 up {0=a=up:active}, 2 up:standby +The cluster will automatically deactivate extra ranks incrementally until +``max_mds`` is reached. See :doc:`/cephfs/administration` for more details which forms ```` can take. -The deactivated rank will first enter the stopping state for a period -of time while it hands off its share of the metadata to the remaining -active daemons. This phase can take from seconds to minutes. If the -MDS appears to be stuck in the stopping state then that should be investigated -as a possible bug. +Note: deactivated ranks will first enter the stopping state for a period of +time while it hands off its share of the metadata to the remaining active +daemons. This phase can take from seconds to minutes. If the MDS appears to +be stuck in the stopping state then that should be investigated as a possible +bug. -If an MDS daemon crashes or is killed while in the 'stopping' state, a -standby will take over and the rank will go back to 'active'. You can -try to deactivate it again once it has come back up. +If an MDS daemon crashes or is killed while in the ``up:stopping`` state, a +standby will take over and the cluster monitors will against try to deactivate +the daemon. -When a daemon finishes stopping, it will respawn itself and go -back to being a standby. +When a daemon finishes stopping, it will respawn itself and go back to being a +standby. Manually pinning directory trees to a particular rank diff --git a/doc/cephfs/upgrading.rst b/doc/cephfs/upgrading.rst index c29644b60ba8..bfc29090814d 100644 --- a/doc/cephfs/upgrading.rst +++ b/doc/cephfs/upgrading.rst @@ -18,11 +18,10 @@ The proper sequence for upgrading the MDS cluster is: ceph fs set max_mds 1 -2. Deactivate all non-zero ranks, from the highest rank to the lowest, while waiting for each MDS to finish stopping: +2. Wait for cluster to deactivate non-zero ranks where only rank 0 is active and the rest are standbys. :: - ceph mds deactivate : ceph status # wait for MDS to finish stopping 3. Take all standbys offline, e.g. using systemctl: @@ -30,17 +29,28 @@ The proper sequence for upgrading the MDS cluster is: :: systemctl stop ceph-mds.target - ceph status # confirm only one MDS is online and is active -4. Upgrade the single active MDS, e.g. using systemctl: +4. Confirm only one MDS is online and is rank 0 for your FS: :: + ceph status + +5. Upgrade the single active MDS, e.g. using systemctl: + +:: + + # use package manager to update cluster systemctl restart ceph-mds.target -5. Upgrade/start the standby daemons. +6. Upgrade/start the standby daemons. + +:: + + # use package manager to update cluster + systemctl restart ceph-mds.target -6. Restore the previous max_mds for your cluster: +7. Restore the previous max_mds for your cluster: ::