Crimson is not enabled by default. Enable it at build time by running::
$ WITH_SEASTAR=true ./install-deps.sh
- $ mkdir build && cd build
- $ cmake -DWITH_SEASTAR=ON ..
+ $ ./do_cmake.sh -DWITH_SEASTAR=ON
Please note, `ASan`_ is enabled by default if Crimson is built from a source
cloned using ``git``.
===============================
The Ceph CI/CD pipeline builds containers with
-``crimson-osd`` subsitituted for ``ceph-osd``.
+``crimson-osd`` substituted for ``ceph-osd``.
Once a branch at commit <sha1> has been built and is available in
``shaman``, you can deploy it using the cephadm instructions outlined
You'll likely need to supply the ``--allow-mismatched-release`` flag to
use a non-release branch.
-Additionally, prior to deploying OSDs, you'll need to
-`Configure Crimson with Bluestore`_ and enable Crimson to
-direct the default pools to be created as Crimson pools. From the cephadm shell run:
-
-.. prompt:: bash #
-
- ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson
- ceph osd set-allow-crimson --yes-i-really-mean-it
- ceph config set mon osd_pool_default_crimson true
-
-The first command enables the ``crimson`` experimental feature. Crimson
-is highly experimental, and malfunctions including crashes
-and data loss are to be expected.
-
-The second enables the ``allow_crimson`` OSDMap flag. The monitor will
-not allow ``crimson-osd`` to boot without that flag.
-
-The last causes pools to be created by default with the ``crimson`` flag.
-Crimson pools are restricted to operations supported by Crimson.
-``Crimson-osd`` won't instantiate PGs from non-Crimson pools.
-
Configure Crimson with Bluestore
================================
#. These two options, along with ``crimson_alien_op_num_threads``,
can't be changed after deployment.
- #. `vstart.sh`_ sets this options using the ``--crimson-smp`` flag.
+ #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag.
1) ``crimson_seastar_num_threads``
2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``.
- Explictily set the CPU core allocation for each ``crimson-osd``
+ Explicitly set the CPU core allocation for each ``crimson-osd``
and for the BlueStore back end. It's recommended for each set to be mutually exclusive.
For example, for deploying two nodes with eight CPU cores and two OSDs each:
Running Crimson
===============
-As you might expect, Crimson does not yet have as extensive a feature set as does ``ceph-osd``.
-
-object store backend
---------------------
-
-At the moment, ``crimson-osd`` offers both native and alienized object store
-backends. The native object store backends perform IO using the SeaStar reactor.
-They are:
-
-.. describe:: cyanstore
-
- CyanStore is modeled after memstore in the classic OSD.
-
-.. describe:: seastore
-
- Seastore is still under active development.
-
-The alienized object store backends are backed by a thread pool, which
-is a proxy of the alienstore adaptor running in Seastar. The proxy issues
-requests to object stores running in alien threads, i.e., worker threads not
-managed by the Seastar framework. They are:
-
-.. describe:: memstore
-
- The memory backed object store
-
-.. describe:: bluestore
-
- The object store used by the classic ``ceph-osd``
-
-daemonize
----------
+.. note::
+ Crimson is in a tech preview stage.
+ As you might expect, Crimson does not yet have as extensive a feature set as does ceph-osd.
+ Malfunctions including crashes and data loss are to be expected.
-Unlike ``ceph-osd``, ``crimson-osd`` does not daemonize itself even if the
-``daemonize`` option is enabled. In order to read this option, ``crimson-osd``
-needs to ready its config sharded service, but this sharded service lives
-in the Seastar reactor. If we fork a child process and exit the parent after
-starting the Seastar engine, that will leave us with a single thread which is
-a replica of the thread that called `fork()`_. Tackling this problem in Crimson
-would unnecessarily complicate the code.
+Enabling Crimson
+================
-Since supported GNU/Linux distributions use ``systemd``, which is able to
-daemonize the application, there is no need to daemonize ourselves.
-Those using sysvinit can use ``start-stop-daemon`` to daemonize ``crimson-osd``.
-If this is does not work out, a helper utility may be devised.
+After building Crimson and starting your cluster, but prior to deploying OSDs, you'll need to
+`Configure Crimson with Bluestore`_ and enable Crimson to
+direct the default pools to be created as Crimson pools. You can proceed by running the following after you have a running cluster:
-.. _fork(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html
+.. note::
+ `vstart.sh`_ enables crimson automatically when `--crimson` is used.
-logging
--------
+.. prompt:: bash #
-``Crimson-osd`` currently uses the logging utility offered by Seastar. See
-``src/common/dout.h`` for the mapping between Ceph logging levels to
-the severity levels in Seastar. For instance, messages sent to ``derr``
-will be issued using ``logger::error()``, and the messages with a debug level
-greater than ``20`` will be issued using ``logger::trace()``.
+ ceph config set global 'enable_experimental_unrecoverable_data_corrupting_features' crimson
+ ceph osd set-allow-crimson --yes-i-really-mean-it
+ ceph config set mon osd_pool_default_crimson true
-+---------+---------+
-| ceph | seastar |
-+---------+---------+
-| < 0 | error |
-+---------+---------+
-| 0 | warn |
-+---------+---------+
-| [1, 6) | info |
-+---------+---------+
-| [6, 20] | debug |
-+---------+---------+
-| > 20 | trace |
-+---------+---------+
+The first command enables the ``crimson`` experimental feature.
-Note that ``crimson-osd``
-does not send log messages directly to a specified ``log_file``. It writes
-the logging messages to stdout and/or syslog. This behavior can be
-changed using ``--log-to-stdout`` and ``--log-to-syslog`` command line
-options. By default, ``log-to-stdout`` is enabled, and ``--log-to-syslog`` is disabled.
+The second enables the ``allow_crimson`` OSDMap flag. The monitor will
+not allow ``crimson-osd`` to boot without that flag.
+The last causes pools to be created by default with the ``crimson`` flag.
+Crimson pools are restricted to operations supported by Crimson.
+``Crimson-osd`` won't instantiate PGs from non-Crimson pools.
vstart.sh
----------
+=========
The following options can be used with ``vstart.sh``.
for additional Seastar-specific command line options.
-``--cyanstore``
- Use CyanStore as the object store backend.
-
-``--bluestore``
- Use the alienized BlueStore as the object store backend. This is the default.
-
``--crimson-smp``
The number of cores to use for each OSD.
If BlueStore is used, the balance of available cores
(as determined by `nproc`) will be assigned to the object store.
+``--bluestore``
+ Use the alienized BlueStore as the object store backend. This is the default (see below section on the `object store backend`_ for more details)
+
+``--cyanstore``
+ Use CyanStore as the object store backend.
+
``--memstore``
Use the alienized MemStore as the object store backend.
To start a cluster with a single Crimson node, run::
- $ MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x \
- --without-dashboard --cyanstore \
- --crimson --redirect-output \
- --osd-args "--memory 4G"
-
-Here we assign 4 GiB memory and a single thread running on core-0 to ``crimson-osd``.
+ $ MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh \
+ --without-dashboard --bluestore --crimson \
+ --redirect-output
Another SeaStore example::
$ ../src/stop.sh --crimson
+Object Store Backend
+====================
+
+At the moment, ``crimson-osd`` offers both native and alienized object store
+backends. The native object store backends perform IO using the SeaStar reactor.
+They are:
+
+.. describe:: cyanstore
+
+ CyanStore is modeled after memstore in the classic OSD.
+
+.. describe:: seastore
+
+ Seastore is still under active development.
+
+The alienized object store backends are backed by a thread pool, which
+is a proxy of the alienstore adaptor running in Seastar. The proxy issues
+requests to object stores running in alien threads, i.e., worker threads not
+managed by the Seastar framework. They are:
+
+.. describe:: memstore
+
+ The memory backend object store
+
+.. describe:: bluestore
+
+ The object store used by the classic ``ceph-osd``
+
+daemonize
+---------
+
+Unlike ``ceph-osd``, ``crimson-osd`` does not daemonize itself even if the
+``daemonize`` option is enabled. In order to read this option, ``crimson-osd``
+needs to ready its config sharded service, but this sharded service lives
+in the Seastar reactor. If we fork a child process and exit the parent after
+starting the Seastar engine, that will leave us with a single thread which is
+a replica of the thread that called `fork()`_. Tackling this problem in Crimson
+would unnecessarily complicate the code.
+
+Since supported GNU/Linux distributions use ``systemd``, which is able to
+daemonize processes, there is no need to daemonize ourselves.
+Those using sysvinit can use ``start-stop-daemon`` to daemonize ``crimson-osd``.
+If this is does not work out, a helper utility may be devised.
+
+.. _fork(): http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html
+
+logging
+-------
+
+``Crimson-osd`` currently uses the logging utility offered by Seastar. See
+``src/common/dout.h`` for the mapping between Ceph logging levels to
+the severity levels in Seastar. For instance, messages sent to ``derr``
+will be issued using ``logger::error()``, and the messages with a debug level
+greater than ``20`` will be issued using ``logger::trace()``.
+
++---------+---------+
+| ceph | seastar |
++---------+---------+
+| < 0 | error |
++---------+---------+
+| 0 | warn |
++---------+---------+
+| [1, 6) | info |
++---------+---------+
+| [6, 20] | debug |
++---------+---------+
+| > 20 | trace |
++---------+---------+
+
+Note that ``crimson-osd``
+does not send log messages directly to a specified ``log_file``. It writes
+the logging messages to stdout and/or syslog. This behavior can be
+changed using ``--log-to-stdout`` and ``--log-to-syslog`` command line
+options. By default, ``log-to-stdout`` is enabled, and ``--log-to-syslog`` is disabled.
Metrics and Tracing
===================
Crimson offers three ways to report stats and metrics.
-pg stats reported to mgr
+PG stats reported to mgr
------------------------
Crimson collects the per-pg, per-pool, and per-osd stats in a `MPGStats`
message which is sent to the Ceph Managers. Manager modules can query
them using the `MgrModule.get()` method.
-asock command
+Asock command
-------------
An admin socket command is offered for dumping metrics::
Profiling Crimson
=================
-fio
+Fio
---
``crimson-store-nbd`` exposes configurable ``FuturizedStore`` internals as an
The ``seastar-addr2line`` utility provided by Seastar can be used to map these
addresses to functions. The script expects input on ``stdin``,
so we need to copy and paste the above addresses, then send EOF by inputting
-``control-D`` in the terminal. One might use ``echo`` or ``cat`` instead`::
+``control-D`` in the terminal. One might use ``echo`` or ``cat`` instead::
$ ../src/seastar/scripts/seastar-addr2line -e bin/crimson-osd
projects / ceph.git / commitdiff