Get a CRUSH Map
---------------
-To get the CRUSH map for your cluster, execute the following::
+To get the CRUSH map for your cluster, execute the following:
+
+.. prompt:: bash $
ceph osd getcrushmap -o {compiled-crushmap-filename}
Decompile a CRUSH Map
---------------------
-To decompile a CRUSH map, execute the following::
+To decompile a CRUSH map, execute the following:
+
+.. prompt:: bash $
crushtool -d {compiled-crushmap-filename} -o {decompiled-crushmap-filename}
Recompile a CRUSH Map
---------------------
-To compile a CRUSH map, execute the following::
+To compile a CRUSH map, execute the following:
+
+.. prompt:: bash $
crushtool -c {decompiled-crushmap-filename} -o {compiled-crushmap-filename}
Set the CRUSH Map
-----------------
-To set the CRUSH map for your cluster, execute the following::
+To set the CRUSH map for your cluster, execute the following:
+
+.. prompt:: bash $
ceph osd setcrushmap -i {compiled-crushmap-filename}
``hdd`` or ``ssd``), allowing them to be conveniently targeted by a
crush rule.
+.. prompt:: bash #
+
+ devices
+
::
- # devices
device {num} {osd.name} [class {class}]
-For example::
+For example:
+
+.. prompt:: bash #
+
+ devices
+
+::
- # devices
device 0 osd.0 class ssd
device 1 osd.1 class hdd
device 2 osd.2
one for data and one for a journal or metadata), or in some cases a
small RAID device.
-
-
-
-
CRUSH Map Bucket Types
----------------------
By convention, there is one leaf bucket and it is ``type 0``; however, you may
give it any name you like (e.g., osd, disk, drive, storage, etc.)::
- #types
+ # types
type {num} {bucket-name}
For example::
single bucket. For example, in the previous example, we want the
``ssd`` bucket to be mapped to the ``default`` bucket.
-The final command to convert the map comprised of the above fragments would be something like::
+The final command to convert the map comprised of the above fragments would be something like:
+
+.. prompt:: bash $
- $ ceph osd getcrushmap -o original
- $ crushtool -i original --reclassify \
- --set-subtree-class default hdd \
- --reclassify-root default hdd \
- --reclassify-bucket %-ssd ssd default \
- --reclassify-bucket ssd ssd default \
- -o adjusted
+ ceph osd getcrushmap -o original
+ crushtool -i original --reclassify \
+ --set-subtree-class default hdd \
+ --reclassify-root default hdd \
+ --reclassify-bucket %-ssd ssd default \
+ --reclassify-bucket ssd ssd default \
+ -o adjusted
-In order to ensure that the conversion is correct, there is a ``--compare`` command that will test a large sample of inputs to the CRUSH map and ensure that the same result comes back out. These inputs are controlled by the same options that apply to the ``--test`` command. For the above example,::
+In order to ensure that the conversion is correct, there is a ``--compare`` command that will test a large sample of inputs to the CRUSH map and ensure that the same result comes back out. These inputs are controlled by the same options that apply to the ``--test`` command. For the above example,:
+
+.. prompt:: bash $
+
+ crushtool -i original --compare adjusted
+
+::
- $ crushtool -i original --compare adjusted
rule 0 had 0/10240 mismatched mappings (0)
rule 1 had 0/10240 mismatched mappings (0)
maps appear equivalent
If there were difference, you'd see what ratio of inputs are remapped
in the parentheses.
-If you are satisfied with the adjusted map, you can apply it to the cluster with something like::
+If you are satisfied with the adjusted map, you can apply it to the cluster with something like:
+
+.. prompt:: bash $
- ceph osd setcrushmap -i adjusted
+ ceph osd setcrushmap -i adjusted
Tuning CRUSH, the hard way
--------------------------
adjust the tunables by extracting the CRUSH map, modifying the values,
and reinjecting it into the cluster.
-* Extract the latest CRUSH map::
+* Extract the latest CRUSH map:
+
+ .. prompt:: bash $
ceph osd getcrushmap -o /tmp/crush
for both large and small clusters we tested with. You will need to
additionally specify the ``--enable-unsafe-tunables`` argument to
``crushtool`` for this to work. Please use this option with
- extreme care.::
+ extreme care.:
- crushtool -i /tmp/crush --set-choose-local-tries 0 --set-choose-local-fallback-tries 0 --set-choose-total-tries 50 -o /tmp/crush.new
+ .. prompt:: bash $
-* Reinject modified map::
+ crushtool -i /tmp/crush --set-choose-local-tries 0 --set-choose-local-fallback-tries 0 --set-choose-total-tries 50 -o /tmp/crush.new
- ceph osd setcrushmap -i /tmp/crush.new
+* Reinject modified map:
+
+ .. prompt:: bash $
+
+ ceph osd setcrushmap -i /tmp/crush.new
Legacy values
-------------
For reference, the legacy values for the CRUSH tunables can be set
-with::
+with:
+
+.. prompt:: bash $
crushtool -i /tmp/crush --set-choose-local-tries 2 --set-choose-local-fallback-tries 5 --set-choose-total-tries 19 --set-chooseleaf-descend-once 0 --set-chooseleaf-vary-r 0 -o /tmp/crush.legacy