doc/rados/operations: document the kernel-client min-compat holdout

A kernel CephFS or RBD mount can keep ``set-require-min-compat-client
reef`` from succeeding: the kernel client does not advertise the
pg-upmap-primary feature, so ``ceph features`` reports it as a luminous
client even when the daemons and every userspace client are newer.
ceph-fuse, libcephfs, and librbd advertise the full feature set and are
not affected.

Document this, and add a section on finding the holdout with
``ceph features`` and switching it to a userspace client.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

diff --git a/doc/rados/operations/require-min-compat-client.rst b/doc/rados/operations/require-min-compat-client.rst
index dbfa6204f5d342a26e1c4ec032f5d0a2dbbaa1f4..ad5cb938d058eebf4d0f6152867c146fc29204dc 100644 (file)
--- a/doc/rados/operations/require-min-compat-client.rst
+++ b/doc/rados/operations/require-min-compat-client.rst
@@ -112,7 +112,10 @@ The Linux kernel Ceph client (``krbd`` for RBD block devices, ``kcephfs``
 for CephFS) ships with the kernel, not with Ceph packages, so its version
 is independent of the cluster release. New OSD map features reach the
 kernel client only after they are implemented and merged upstream, which
-can lag the Ceph daemon release by months or years.
+can lag the Ceph daemon release by months or years. The userspace clients
+(``ceph-fuse`` and ``libcephfs`` for CephFS, ``librbd`` for RBD) link the
+Ceph release they were built from and advertise its full feature set, so
+they are not subject to this lag.
 
 The *Min kernel* column shows the minimum Linux kernel version required on
 kernel client hosts before the corresponding commands are used. If no
@@ -164,6 +167,21 @@ kernel clients are present in the cluster, that column can be ignored.
 
    See :ref:`read_balancer` for removal commands.
 
+When a kernel mount blocks the flag
+===================================
+
+A single kernel mount can hold the whole cluster below the release you
+want, even when every daemon and userspace client is already newer.
+``ceph features`` reports such a mount as ``luminous`` because the kernel
+client does not advertise the ``pg-upmap-primary`` (reef) feature. To
+confirm which entry it adds, run ``ceph features`` with the mount
+unmounted, then again with it mounted, and compare the client counts.
+
+Since the userspace clients advertise the full feature set, switching the
+holdout to its userspace equivalent (``ceph-fuse`` or ``libcephfs`` for
+CephFS, ``librbd`` for RBD) clears the blocker, after which the flag can
+be raised and the read balancer enabled.
+
 See :ref:`upmap` and :ref:`read_balancer` for the operational procedures
 that depend on raising the flag. CephFS clients must satisfy both this
 flag and any per-filesystem :ref:`cephfs_required_client_features` on