From: gitkenan Date: Sun, 21 Jul 2024 22:02:47 +0000 (+0100) Subject: doc/dev/crimson: clarify and rearrange for userability X-Git-Tag: v19.1.1~64^2 X-Git-Url: http://git.apps.os.sepia.ceph.com/?a=commitdiff_plain;h=refs%2Fpull%2F58743%2Fhead;p=ceph.git doc/dev/crimson: clarify and rearrange for userability Signed-off-by: gitkenan (cherry picked from commit 92fea64c1f28a010336b3696863d3f003cd684ef) --- diff --git a/doc/dev/crimson/crimson.rst b/doc/dev/crimson/crimson.rst index c06ece6f5a33c..f6d59a057ff85 100644 --- a/doc/dev/crimson/crimson.rst +++ b/doc/dev/crimson/crimson.rst @@ -16,8 +16,7 @@ Building Crimson Crimson is not enabled by default. Enable it at build time by running:: $ WITH_SEASTAR=true ./install-deps.sh - $ mkdir build && cd build - $ cmake -DWITH_SEASTAR=ON .. + $ ./do_cmake.sh -DWITH_SEASTAR=ON Please note, `ASan`_ is enabled by default if Crimson is built from a source cloned using ``git``. @@ -28,7 +27,7 @@ Testing crimson with cephadm =============================== The Ceph CI/CD pipeline builds containers with -``crimson-osd`` subsitituted for ``ceph-osd``. +``crimson-osd`` substituted for ``ceph-osd``. Once a branch at commit has been built and is available in ``shaman``, you can deploy it using the cephadm instructions outlined @@ -44,27 +43,6 @@ use a Crimson build: You'll likely need to supply the ``--allow-mismatched-release`` flag to use a non-release branch. -Additionally, prior to deploying OSDs, you'll need to -`Configure Crimson with Bluestore`_ and enable Crimson to -direct the default pools to be created as Crimson pools. From the cephadm shell run: - -.. prompt:: bash # - - ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson - ceph osd set-allow-crimson --yes-i-really-mean-it - ceph config set mon osd_pool_default_crimson true - -The first command enables the ``crimson`` experimental feature. Crimson -is highly experimental, and malfunctions including crashes -and data loss are to be expected. - -The second enables the ``allow_crimson`` OSDMap flag. The monitor will -not allow ``crimson-osd`` to boot without that flag. - -The last causes pools to be created by default with the ``crimson`` flag. -Crimson pools are restricted to operations supported by Crimson. -``Crimson-osd`` won't instantiate PGs from non-Crimson pools. - Configure Crimson with Bluestore ================================ @@ -76,7 +54,7 @@ one of the two following configuration options: #. These two options, along with ``crimson_alien_op_num_threads``, can't be changed after deployment. - #. `vstart.sh`_ sets this options using the ``--crimson-smp`` flag. + #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag. 1) ``crimson_seastar_num_threads`` @@ -103,7 +81,7 @@ one of the two following configuration options: 2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``. - Explictily set the CPU core allocation for each ``crimson-osd`` + Explicitly set the CPU core allocation for each ``crimson-osd`` and for the BlueStore back end. It's recommended for each set to be mutually exclusive. For example, for deploying two nodes with eight CPU cores and two OSDs each: @@ -144,86 +122,38 @@ one of the two following configuration options: Running Crimson =============== -As you might expect, Crimson does not yet have as extensive a feature set as does ``ceph-osd``. - -object store backend --------------------- - -At the moment, ``crimson-osd`` offers both native and alienized object store -backends. The native object store backends perform IO using the SeaStar reactor. -They are: - -.. describe:: cyanstore - - CyanStore is modeled after memstore in the classic OSD. - -.. describe:: seastore - - Seastore is still under active development. - -The alienized object store backends are backed by a thread pool, which -is a proxy of the alienstore adaptor running in Seastar. The proxy issues -requests to object stores running in alien threads, i.e., worker threads not -managed by the Seastar framework. They are: - -.. describe:: memstore - - The memory backed object store - -.. describe:: bluestore - - The object store used by the classic ``ceph-osd`` - -daemonize ---------- +.. note:: + Crimson is in a tech preview stage. + As you might expect, Crimson does not yet have as extensive a feature set as does ceph-osd. + Malfunctions including crashes and data loss are to be expected. -Unlike ``ceph-osd``, ``crimson-osd`` does not daemonize itself even if the -``daemonize`` option is enabled. In order to read this option, ``crimson-osd`` -needs to ready its config sharded service, but this sharded service lives -in the Seastar reactor. If we fork a child process and exit the parent after -starting the Seastar engine, that will leave us with a single thread which is -a replica of the thread that called `fork()`_. Tackling this problem in Crimson -would unnecessarily complicate the code. +Enabling Crimson +================ -Since supported GNU/Linux distributions use ``systemd``, which is able to -daemonize the application, there is no need to daemonize ourselves. -Those using sysvinit can use ``start-stop-daemon`` to daemonize ``crimson-osd``. -If this is does not work out, a helper utility may be devised. +After building Crimson and starting your cluster, but prior to deploying OSDs, you'll need to +`Configure Crimson with Bluestore`_ and enable Crimson to +direct the default pools to be created as Crimson pools. You can proceed by running the following after you have a running cluster: -.. _fork(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html +.. note:: + `vstart.sh`_ enables crimson automatically when `--crimson` is used. -logging -------- +.. prompt:: bash # -``Crimson-osd`` currently uses the logging utility offered by Seastar. See -``src/common/dout.h`` for the mapping between Ceph logging levels to -the severity levels in Seastar. For instance, messages sent to ``derr`` -will be issued using ``logger::error()``, and the messages with a debug level -greater than ``20`` will be issued using ``logger::trace()``. + ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson + ceph osd set-allow-crimson --yes-i-really-mean-it + ceph config set mon osd_pool_default_crimson true -+---------+---------+ -| ceph | seastar | -+---------+---------+ -| < 0 | error | -+---------+---------+ -| 0 | warn | -+---------+---------+ -| [1, 6) | info | -+---------+---------+ -| [6, 20] | debug | -+---------+---------+ -| > 20 | trace | -+---------+---------+ +The first command enables the ``crimson`` experimental feature. -Note that ``crimson-osd`` -does not send log messages directly to a specified ``log_file``. It writes -the logging messages to stdout and/or syslog. This behavior can be -changed using ``--log-to-stdout`` and ``--log-to-syslog`` command line -options. By default, ``log-to-stdout`` is enabled, and ``--log-to-syslog`` is disabled. +The second enables the ``allow_crimson`` OSDMap flag. The monitor will +not allow ``crimson-osd`` to boot without that flag. +The last causes pools to be created by default with the ``crimson`` flag. +Crimson pools are restricted to operations supported by Crimson. +``Crimson-osd`` won't instantiate PGs from non-Crimson pools. vstart.sh ---------- +========= The following options can be used with ``vstart.sh``. @@ -246,17 +176,17 @@ The following options can be used with ``vstart.sh``. for additional Seastar-specific command line options. -``--cyanstore`` - Use CyanStore as the object store backend. - -``--bluestore`` - Use the alienized BlueStore as the object store backend. This is the default. - ``--crimson-smp`` The number of cores to use for each OSD. If BlueStore is used, the balance of available cores (as determined by `nproc`) will be assigned to the object store. +``--bluestore`` + Use the alienized BlueStore as the object store backend. This is the default (see below section on the `object store backend`_ for more details) + +``--cyanstore`` + Use CyanStore as the object store backend. + ``--memstore`` Use the alienized MemStore as the object store backend. @@ -279,12 +209,9 @@ The following options can be used with ``vstart.sh``. To start a cluster with a single Crimson node, run:: - $ MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x \ - --without-dashboard --cyanstore \ - --crimson --redirect-output \ - --osd-args "--memory 4G" - -Here we assign 4 GiB memory and a single thread running on core-0 to ``crimson-osd``. + $ MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh \ + --without-dashboard --bluestore --crimson \ + --redirect-output Another SeaStore example:: @@ -299,19 +226,93 @@ Stop this ``vstart`` cluster by running:: $ ../src/stop.sh --crimson +Object Store Backend +==================== + +At the moment, ``crimson-osd`` offers both native and alienized object store +backends. The native object store backends perform IO using the SeaStar reactor. +They are: + +.. describe:: cyanstore + + CyanStore is modeled after memstore in the classic OSD. + +.. describe:: seastore + + Seastore is still under active development. + +The alienized object store backends are backed by a thread pool, which +is a proxy of the alienstore adaptor running in Seastar. The proxy issues +requests to object stores running in alien threads, i.e., worker threads not +managed by the Seastar framework. They are: + +.. describe:: memstore + + The memory backend object store + +.. describe:: bluestore + + The object store used by the classic ``ceph-osd`` + +daemonize +--------- + +Unlike ``ceph-osd``, ``crimson-osd`` does not daemonize itself even if the +``daemonize`` option is enabled. In order to read this option, ``crimson-osd`` +needs to ready its config sharded service, but this sharded service lives +in the Seastar reactor. If we fork a child process and exit the parent after +starting the Seastar engine, that will leave us with a single thread which is +a replica of the thread that called `fork()`_. Tackling this problem in Crimson +would unnecessarily complicate the code. + +Since supported GNU/Linux distributions use ``systemd``, which is able to +daemonize processes, there is no need to daemonize ourselves. +Those using sysvinit can use ``start-stop-daemon`` to daemonize ``crimson-osd``. +If this is does not work out, a helper utility may be devised. + +.. _fork(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html + +logging +------- + +``Crimson-osd`` currently uses the logging utility offered by Seastar. See +``src/common/dout.h`` for the mapping between Ceph logging levels to +the severity levels in Seastar. For instance, messages sent to ``derr`` +will be issued using ``logger::error()``, and the messages with a debug level +greater than ``20`` will be issued using ``logger::trace()``. + ++---------+---------+ +| ceph | seastar | ++---------+---------+ +| < 0 | error | ++---------+---------+ +| 0 | warn | ++---------+---------+ +| [1, 6) | info | ++---------+---------+ +| [6, 20] | debug | ++---------+---------+ +| > 20 | trace | ++---------+---------+ + +Note that ``crimson-osd`` +does not send log messages directly to a specified ``log_file``. It writes +the logging messages to stdout and/or syslog. This behavior can be +changed using ``--log-to-stdout`` and ``--log-to-syslog`` command line +options. By default, ``log-to-stdout`` is enabled, and ``--log-to-syslog`` is disabled. Metrics and Tracing =================== Crimson offers three ways to report stats and metrics. -pg stats reported to mgr +PG stats reported to mgr ------------------------ Crimson collects the per-pg, per-pool, and per-osd stats in a `MPGStats` message which is sent to the Ceph Managers. Manager modules can query them using the `MgrModule.get()` method. -asock command +Asock command ------------- An admin socket command is offered for dumping metrics:: @@ -334,7 +335,7 @@ see `Prometheus`_ for more details. Profiling Crimson ================= -fio +Fio --- ``crimson-store-nbd`` exposes configurable ``FuturizedStore`` internals as an @@ -506,7 +507,7 @@ When a Seastar application crashes, it leaves us with a backtrace of addresses, The ``seastar-addr2line`` utility provided by Seastar can be used to map these addresses to functions. The script expects input on ``stdin``, so we need to copy and paste the above addresses, then send EOF by inputting -``control-D`` in the terminal. One might use ``echo`` or ``cat`` instead`:: +``control-D`` in the terminal. One might use ``echo`` or ``cat`` instead:: $ ../src/seastar/scripts/seastar-addr2line -e bin/crimson-osd