doc/rados: explain what a device class breaks in stretch mode

The warning said device classes are "not supported" and that such a rule
"will not work", without saying what actually happens. Ceph accepts the
rule, honors the class, and places the data correctly. Peering is what
fails. The rule takes from a shadow tree, so an OSD's datacenter resolves
to "zone1~ssd", while the Monitors record the surviving zone as "zone1".
The two never match. Nothing looks wrong while both data centers are up,
but once one is lost, the PGs left behind cannot peer. Bringing the data
center back does not help either, because the cluster leaves degraded
stretch mode only once no PG is inactive.

Describe that, and point at the class-free rule under "Entering Stretch
Mode". Also fix the example, which wrote "type replicated class ssd". The
CRUSH grammar takes a device class on the take step.

Related-to: https://tracker.ceph.com/issues/67414
Signed-off-by: Kefu Chai <k.chai@proxmox.com>

diff --git a/doc/rados/operations/stretch-mode.rst b/doc/rados/operations/stretch-mode.rst
index f07b46d1e8ef4d068f9116adc01e6cf513e3a21f..3a48b5fcd92e6a7e7aef066e67850f7ea2b82091 100644 (file)
--- a/doc/rados/operations/stretch-mode.rst
+++ b/doc/rados/operations/stretch-mode.rst
@@ -152,6 +152,8 @@ loss during zone failures because if a PG were allowed to go ``active`` with rep
 writes could be acknowledged despite a lack of redundancy. In the event of a zone failure, all data in the
 affected PG would be lost.
 
+.. _entering_stretch_mode:
+
 Entering Stretch Mode
 ---------------------
 
@@ -353,18 +355,50 @@ SSDs. Hybrid HDD+SSD or HDD-only OSDs are not recommended
 due to the long time it takes for them to recover after connectivity between
 data centers has been restored. This reduces the potential for data loss.
 
-.. warning:: CRUSH rules that specify a device class are not supported in stretch mode.
-   For example, the following rule specifying the ``ssd`` device class will not work::
+.. warning:: Do not specify a device class in the CRUSH rule used for stretch
+   mode. Ceph accepts such a rule and reports healthy stretch mode, but the PGs
+   will fail to peer if a data center is later lost.
+
+   A device class makes CRUSH select OSDs from a shadow tree, in which
+   ``zone1`` is a separate bucket named ``zone1~ssd``::
 
       rule stretch_replicated_rule {
                  id 1
-                 type replicated class ssd
-                 step take default
+                 type replicated
+                 step take default class ssd
                  step choose firstn 0 type datacenter
                  step chooseleaf firstn 2 type host
                  step emit
       }
 
+   The class is honored and the placement is correct: OSDs of that class are
+   chosen from both data centers, as asked. The problem is in the peering
+   check. Stretch mode resolves each OSD to its ``datacenter`` through the
+   pool's CRUSH rule, so with the rule above it sees ``zone1~ssd`` rather than
+   ``zone1``. You will not notice this while both data centers are up, because
+   peering only counts how many distinct buckets the acting set spans, and two
+   shadow buckets count as two.
+
+   The mismatch matters once a data center is lost. When the cluster enters
+   degraded stretch mode, the Monitors record the surviving data center by its
+   real bucket, ``zone1``, and require every PG to include it. That never
+   matches the ``zone1~ssd`` the rule reports, so every PG of every pool using
+   that rule goes inactive and its data is unavailable.
+
+   Restoring the lost data center does not clear this. The cluster returns to
+   healthy stretch mode only once no PG is left degraded, inactive, or unknown,
+   and these PGs stay inactive for as long as that requirement stands, so
+   recovery never finishes on its own. Breaking the loop takes ``ceph osd
+   force_healthy_stretch_mode --yes-i-really-mean-it``, which drops the
+   requirement and lets the PGs peer.
+
+   Use a rule without a device class from the start, such as the one under
+   :ref:`entering_stretch_mode`. If a cluster is already in stretch mode, check
+   the rule that each of its pools uses and correct it while both data centers
+   are up. Switching a pool to another rule moves most of its data, because the
+   shadow tree and the real tree place data differently even when they hold the
+   same OSDs.
+
 In the future, stretch mode could support erasure-coded pools,
 enable deployments across more than two data centers,
 and accommodate multiple CRUSH device classes.