From 899da82ae2e857f2a769c08d6623e6da7678d16b Mon Sep 17 00:00:00 2001 From: Sridhar Seshasayee Date: Tue, 15 Apr 2025 18:31:50 +0530 Subject: [PATCH] doc/rados: Update mClock doc on steps to override OSD IOPS capacity config Describe the steps involved to - Specify a global value for osd_mclock_max_capacity_iops_{ssd,hdd}, and - Override existing individually scoped values for OSDs determined during start-up for osd_mclock_max_capacity_iops_{ssd,hdd}. The above is to help with the following: - Steps to override existing setting with a global value. - reduce the number of entries in the mon store and instead use a single global specification for all OSDs in the cluster in case the underlying hardware is the same for all OSDs. Signed-off-by: Sridhar Seshasayee Fixes: https://tracker.ceph.com/issues/70774 --- doc/rados/configuration/mclock-config-ref.rst | 89 ++++++++++++++++--- 1 file changed, 79 insertions(+), 10 deletions(-) diff --git a/doc/rados/configuration/mclock-config-ref.rst b/doc/rados/configuration/mclock-config-ref.rst index c1287a21f1b53..9081028325ca3 100644 --- a/doc/rados/configuration/mclock-config-ref.rst +++ b/doc/rados/configuration/mclock-config-ref.rst @@ -714,29 +714,98 @@ general for HDDs, the bluestore throttle values are expected to be higher when compared to SSDs. -Specifying Max OSD Capacity ----------------------------- +Set or Override Max IOPS Capacity of an OSD +------------------------------------------- -The steps in this section may be performed only if you want to override the -max osd capacity automatically set during OSD initialization. The option -``osd_mclock_max_capacity_iops_[hdd, ssd]`` for an OSD can be set by running the -following command: +The steps in this section may be performed to set or override the max IOPS +capacity of an OSD. The ``osd_mclock_max_capacity_iops_[hdd, ssd]`` option for +an OSD can be overridden by running a command of the following form: - .. prompt:: bash # +.. prompt:: bash # - ceph config set osd.N osd_mclock_max_capacity_iops_[hdd,ssd] + ceph config set osd.N osd_mclock_max_capacity_iops_[hdd,ssd] For example, the following command sets the max capacity for a specific OSD (say "osd.0") whose underlying device type is HDD to 350 IOPS: - .. prompt:: bash # +.. prompt:: bash # - ceph config set osd.0 osd_mclock_max_capacity_iops_hdd 350 + ceph config set osd.0 osd_mclock_max_capacity_iops_hdd 350 Alternatively, you may specify the max capacity for OSDs within the Ceph configuration file under the respective [osd.N] section. See :ref:`ceph-conf-settings` for more details. +Global Override of Max IOPS Capacity for multiple OSDs +------------------------------------------------------ + +The max IOPS capacity of multiple OSDs may be overridden by a global config +specification. This section shows the steps to globally override the +individually scoped values in the mon store. + + .. note:: The examples use :confval:`osd_mclock_max_capacity_iops_hdd` and + the steps are also applicable for SSD based OSDs in which case + the option to use is :confval:`osd_mclock_max_capacity_iops_ssd`. + +Below are steps to override the IOPS capacities of individual OSDs. Note that +the individual value is taken by the OSD after it runs the usual startup +benchmark. + +#. Run the following command to verify the individual values set for the OSDs in + the central config database: + + .. prompt:: bash # + + ceph config dump | grep osd_mclock_max_capacity_iops + + :: + + WHO MASK LEVEL OPTION VALUE RO + osd.0 basic osd_mclock_max_capacity_iops_hdd 379.197568 + osd.1 basic osd_mclock_max_capacity_iops_hdd 400.903575 + osd.2 basic osd_mclock_max_capacity_iops_hdd 398.303428 + osd.3 basic osd_mclock_max_capacity_iops_hdd 419.035854 + +#. If there are no individual values reported, skip to the next step. Otherwise, + remove all the individual values reported in the previous step with a command + of the following form (where 'x' is the OSD id): + + .. prompt:: bash # + + ceph config rm osd.x osd_mclock_max_capacity_iops_hdd + +#. Confirm that the `ceph config dump` command from step 1 does not show any + individual values. + +#. Set the global value of `osd_mclock_max_capacity_iops_hdd` with a command of + the following form: + + .. prompt:: bash # + + ceph config set global osd_mclock_max_capacity_iops_hdd 111 + +#. Confirm that the global option is set by running: + + .. prompt:: bash # + + ceph config dump | grep osd_mclock_max_capacity_iops + + :: + + global basic osd_mclock_max_capacity_iops_hdd 111.000000 + + +#. Confirm that the global setting is now in effect for any OSD that no longer + has a specific per-OSD central config setting: + + .. prompt:: bash # + + ceph config show osd.0 | grep osd_mclock_max_capacity_iops_hdd + + :: + + osd_mclock_max_capacity_iops_hdd 111.000000 mon + .. index:: mclock; config settings -- 2.39.5