From: Kefu Chai Date: Mon, 27 Jul 2026 14:36:21 +0000 (+0800) Subject: doc/rados: explain what a device class breaks in stretch mode X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=b542c2feb348926bc7a9219251a1bb384fedd41d;p=ceph.git doc/rados: explain what a device class breaks in stretch mode The warning said device classes are "not supported" and that such a rule "will not work", without saying what actually happens. Ceph accepts the rule, honors the class, and places the data correctly. Peering is what fails. The rule takes from a shadow tree, so an OSD's datacenter resolves to "zone1~ssd", while the Monitors record the surviving zone as "zone1". The two never match. Nothing looks wrong while both data centers are up, but once one is lost, the PGs left behind cannot peer. Bringing the data center back does not help either, because the cluster leaves degraded stretch mode only once no PG is inactive. Describe that, and point at the class-free rule under "Entering Stretch Mode". Also fix the example, which wrote "type replicated class ssd". The CRUSH grammar takes a device class on the take step. Related-to: https://tracker.ceph.com/issues/67414 Signed-off-by: Kefu Chai --- diff --git a/doc/rados/operations/stretch-mode.rst b/doc/rados/operations/stretch-mode.rst index f07b46d1e8ef..3a48b5fcd92e 100644 --- a/doc/rados/operations/stretch-mode.rst +++ b/doc/rados/operations/stretch-mode.rst @@ -152,6 +152,8 @@ loss during zone failures because if a PG were allowed to go ``active`` with rep writes could be acknowledged despite a lack of redundancy. In the event of a zone failure, all data in the affected PG would be lost. +.. _entering_stretch_mode: + Entering Stretch Mode --------------------- @@ -353,18 +355,50 @@ SSDs. Hybrid HDD+SSD or HDD-only OSDs are not recommended due to the long time it takes for them to recover after connectivity between data centers has been restored. This reduces the potential for data loss. -.. warning:: CRUSH rules that specify a device class are not supported in stretch mode. - For example, the following rule specifying the ``ssd`` device class will not work:: +.. warning:: Do not specify a device class in the CRUSH rule used for stretch + mode. Ceph accepts such a rule and reports healthy stretch mode, but the PGs + will fail to peer if a data center is later lost. + + A device class makes CRUSH select OSDs from a shadow tree, in which + ``zone1`` is a separate bucket named ``zone1~ssd``:: rule stretch_replicated_rule { id 1 - type replicated class ssd - step take default + type replicated + step take default class ssd step choose firstn 0 type datacenter step chooseleaf firstn 2 type host step emit } + The class is honored and the placement is correct: OSDs of that class are + chosen from both data centers, as asked. The problem is in the peering + check. Stretch mode resolves each OSD to its ``datacenter`` through the + pool's CRUSH rule, so with the rule above it sees ``zone1~ssd`` rather than + ``zone1``. You will not notice this while both data centers are up, because + peering only counts how many distinct buckets the acting set spans, and two + shadow buckets count as two. + + The mismatch matters once a data center is lost. When the cluster enters + degraded stretch mode, the Monitors record the surviving data center by its + real bucket, ``zone1``, and require every PG to include it. That never + matches the ``zone1~ssd`` the rule reports, so every PG of every pool using + that rule goes inactive and its data is unavailable. + + Restoring the lost data center does not clear this. The cluster returns to + healthy stretch mode only once no PG is left degraded, inactive, or unknown, + and these PGs stay inactive for as long as that requirement stands, so + recovery never finishes on its own. Breaking the loop takes ``ceph osd + force_healthy_stretch_mode --yes-i-really-mean-it``, which drops the + requirement and lets the PGs peer. + + Use a rule without a device class from the start, such as the one under + :ref:`entering_stretch_mode`. If a cluster is already in stretch mode, check + the rule that each of its pools uses and correct it while both data centers + are up. Switching a pool to another rule moves most of its data, because the + shadow tree and the real tree place data differently even when they hold the + same OSDs. + In the future, stretch mode could support erasure-coded pools, enable deployments across more than two data centers, and accommodate multiple CRUSH device classes.