From 8ff129c89ffcd4831dcd9d8b8f0d49687cc57183 Mon Sep 17 00:00:00 2001 From: Matan Breizman Date: Wed, 13 Aug 2025 08:11:30 +0000 Subject: [PATCH] doc/dev/crimson: Update docs * CPU allocation missed some information and was confusing. * Drop alienized term when possible * introdice release/debug builds Signed-off-by: Matan Breizman --- doc/dev/crimson/crimson.rst | 205 ++++++++++++++++-------------------- doc/dev/seastore.rst | 2 + 2 files changed, 93 insertions(+), 114 deletions(-) diff --git a/doc/dev/crimson/crimson.rst b/doc/dev/crimson/crimson.rst index 21e6ab3443390..8316dbd2879ca 100644 --- a/doc/dev/crimson/crimson.rst +++ b/doc/dev/crimson/crimson.rst @@ -2,11 +2,17 @@ crimson ======= -Crimson is the code name of ``crimson-osd``, which is the next -generation ``ceph-osd``. It improves performance when using fast network -and storage devices, employing state-of-the-art technologies including -DPDK and SPDK. BlueStore continues to support HDDs and slower SSDs. -Crimson aims to be backward compatible with the classic ``ceph-osd``. +Crimson is the code name of ``crimson-osd``, which is the next generation ``ceph-osd``. +It is designed to deliver enhanced performance on fast network and storage devices by leveraging modern technologies such as DPDK and SPDK. + +Crimson is intended to be a drop-in replacement for the classic Object Storage Daemon (OSD), +aiming to allow seamless migration from existing ``ceph-osd`` deployments (through backfilling). + +The second phase of the project introduces :ref:`seastore`, a complete redesign of the object storage backend built around Crimson's native architecture. +Seastore is optimized for high-performance storage devices like NVMe and may not be suitable for traditional HDDs. +Crimson will continue to support BlueStore ensuring compatibility with HDDs and slower SSDs. + +See `ceph.io/en/news/crimson `_ .. highlight:: console @@ -23,115 +29,74 @@ cloned using ``git``. .. _ASan: https://github.com/google/sanitizers/wiki/AddressSanitizer -Testing crimson with cephadm -=============================== +Deploying Crimson with cephadm +============================== + +.. note:: + Cephadm SeaStore support is in `early stages `_. -The Ceph CI/CD pipeline builds containers with -``crimson-osd`` substituted for ``ceph-osd``. +The Ceph CI/CD pipeline builds containers with ``crimson-osd`` replacing the standard ``ceph-osd``. Once a branch at commit has been built and is available in -``shaman``, you can deploy it using the cephadm instructions outlined +Shaman / Quay, you can deploy it using the cephadm instructions outlined in :ref:`cephadm` with the following adaptations. -First, while performing the initial bootstrap, use the ``--image`` flag to -use a Crimson build: - -.. prompt:: bash # +The latest `main` branch is built `daily `_ +and the images are available in `quay `_ +(filter ``crimson-release``). +We recommend using one of the latest available builds, as Crimson evolves rapidly. - cephadm --image quay.ceph.io/ceph-ci/ceph:-crimson --allow-mismatched-release bootstrap ... +Use the ``--image`` flag to specify a Crimson build: -You'll likely need to supply the ``--allow-mismatched-release`` flag to -use a non-release branch. +.. prompt:: bash # -Configure Crimson with Bluestore -================================ + cephadm --image quay.ceph.io/ceph-ci/ceph:-crimson-release --allow-mismatched-release bootstrap ... -As Bluestore is not a Crimson native `object store backend`_, -deploying Crimson with Bluestore as the back end requires setting -one of the two following configuration options: .. note:: + Crimson builds are available in two variants: ``crimson-debug`` and ``crimson-release``. + For testing purposes the `release` variant should be used. + The `debug` variant is intended primarily for development. - #. These two options, along with ``crimson_alien_op_num_threads``, - can't be changed after deployment. - #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag. - - -1) ``crimson_seastar_num_threads`` - - In order to allow easier cluster deployments, this option can be used - instead of setting the CPU mask manually for each OSD. - - It's recommended to let the **number of OSDs on each host** multiplied by - ``crimson_seastar_num_threads`` to be less than the node's number of CPU - cores (``nproc``). - - For example, for deploying two nodes with eight CPU cores and two OSDs each: - - .. code-block:: yaml +You'll likely need to include the ``--allow-mismatched-release`` flag to use a non-release branch. - conf: - # Global to all OSDs - osd: - crimson seastar num threads: 3 +Crimson CPU allocation +====================== - .. note:: - - #. For optimal performance ``crimson_seastar_cpu_cores`` should be set instead. - -2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``. - - Explicitly set the CPU core allocation for each ``crimson-osd`` - and for the BlueStore back end. It's recommended for each set to be mutually exclusive. +.. note:: - For example, for deploying two nodes with eight CPU cores and two OSDs each: + #. Allocation options **cannot** be changed after deployment. + #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag. - .. code-block:: yaml +The ``crimson_seastar_num_threads`` parameter defines the number of threads used to serve Seastar reactors. +Each thread is expected to run on a dedicated CPU core. - conf: - # Both nodes - osd: - crimson alien thread cpu cores: 6-7 +This parameter **does not have a default value**. +Users must configure it at the OSD level based on system resources and cluster requirements **before** deploying the OSDs. - # First node - osd.0: - crimson seastar cpu cores: 0-2 - osd.1: - crimson seastar cpu cores: 3-5 +We recommend setting a value for ``crimson_seastar_num_threads`` that is less than the host's +number of CPU cores (``nproc``) divided by the **number of OSDs on that host**. - # Second node - osd.2: - crimson seastar cpu cores: 0-2 - osd.3: - crimson seastar cpu cores: 3-5 +For example, for deploying a node with eight CPU cores per OSD: - For a single node with eight node and three OSDs: +.. code-block:: bash # - .. code-block:: yaml + ceph config set osd crimson_seastar_num_threads 8 - conf: - osd: - crimson alien thread cpu cores: 6-7 - osd.0: - crimson seastar cpu cores: 0-1 - osd.1: - crimson seastar cpu cores: 2-3 - osd.2: - crimson seastar cpu cores: 4-5 +Note that ``crimson_seastar_num_threads`` does **not** pin threads to specific CPU cores. +To explicitly assign CPU cores to Crimson OSDs, use the ``crimson_seastar_cpu_cores`` parameter. +This enables CPU pinning, which *may* improve performance. +However, using this option requires manually setting the CPU set for each OSD, +and is generally less recommended due to its complexity. Running Crimson =============== .. note:: - Crimson is in a tech preview stage. - As you might expect, Crimson does not yet have as extensive a feature set as does ceph-osd. - Malfunctions including crashes and data loss are to be expected. - -Enabling Crimson -================ + Crimson is in a tech preview stage and is **not suitable for production use**. -After building Crimson and starting your cluster, but prior to deploying OSDs, you'll need to -`Configure Crimson with Bluestore`_ and enable Crimson to +After starting your cluster, prior to deploying OSDs, you'll need to configure the +`Crimson CPU allocation`_ and enable Crimson to direct the default pools to be created as Crimson pools. You can proceed by running the following after you have a running cluster: .. note:: @@ -152,6 +117,46 @@ The last causes pools to be created by default with the ``crimson`` flag. Crimson pools are restricted to operations supported by Crimson. ``Crimson-osd`` won't instantiate PGs from non-Crimson pools. +Object Store Backends +===================== + +``crimson-osd`` supports two categories of object store backends: **native** and **non-native**. + +Native Backends +--------------- + +Native backends perform I/O operations using the **Seastar reactor**. These are tightly integrated with the Seastar framework and follow its design principles: + +.. describe:: seastore + + SeaStore is the primary native object store for Crimson OSD. It is built with the Seastar framework and adheres to its asynchronous, shard-based architecture. + +.. describe:: cyanstore + + CyanStore is inspired by ``memstore`` from the classic OSD, offering a lightweight, in-memory object store model. + CyanStore **does not store data** and should be used only for measuring OSD overhead, without the cost of actually storing data. + +Non-Native Backends +------------------ + +Non-native backends operate through a **thread pool proxy**, which interfaces with object stores running in **alien threads**—worker threads not managed by Seastar. +These backends allow Crimson to interact with legacy or external object store implementations: + +.. describe:: bluestore + + The default object store used by the classic ``ceph-osd``. It provides robust, production-grade storage capabilities. + + The ``crimson_alien_op_num_threads`` option needs to be set according to the cpu set available. + This defines the number of threads dedicated to serving the BlueStore ObjectStore on each OSD. + + If ``crimson_seastar_num_threads`` is used from `Crimson CPU allocation`_, + The counterpart ``crimson_alien_thread_cpu_cores`` should also be used accordingly to + allow the two sets to be mutually exclusive. + +.. describe:: memstore + + An in-memory object store backend, primarily used for testing and development purposes. + vstart.sh ========= @@ -226,34 +231,6 @@ Stop this ``vstart`` cluster by running:: $ ../src/stop.sh --crimson -Object Store Backend -==================== - -At the moment, ``crimson-osd`` offers both native and alienized object store -backends. The native object store backends perform IO using the SeaStar reactor. -They are: - -.. describe:: cyanstore - - CyanStore is modeled after memstore in the classic OSD. - -.. describe:: seastore - - Seastore is still under active development. - -The alienized object store backends are backed by a thread pool, which -is a proxy of the alienstore adaptor running in Seastar. The proxy issues -requests to object stores running in alien threads, i.e., worker threads not -managed by the Seastar framework. They are: - -.. describe:: memstore - - The memory backend object store - -.. describe:: bluestore - - The object store used by the classic ``ceph-osd`` - daemonize --------- diff --git a/doc/dev/seastore.rst b/doc/dev/seastore.rst index dd080092c035f..6a3e8069937f6 100644 --- a/doc/dev/seastore.rst +++ b/doc/dev/seastore.rst @@ -1,3 +1,5 @@ +.. _seastore: + ========== SeaStore ========== -- 2.39.5