From 15892599f8cb05c6170f05210f4d8fe8f2dc188f Mon Sep 17 00:00:00 2001 From: Igor Golikov Date: Mon, 8 Dec 2025 11:41:27 +0000 Subject: [PATCH] docs: update docs with new metrics description docs for subvolume utilization and MDS perf metrics Signed-off-by: Igor Golikov Fixes: https://tracker.ceph.com/issues/74135 Fixes: https://tracker.ceph.com/issues/73700 --- doc/cephfs/metrics.rst | 84 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 78 insertions(+), 6 deletions(-) diff --git a/doc/cephfs/metrics.rst b/doc/cephfs/metrics.rst index 17de0854eed..8ec0be76fe6 100644 --- a/doc/cephfs/metrics.rst +++ b/doc/cephfs/metrics.rst @@ -66,13 +66,77 @@ CephFS exports client metrics as :ref:`Labeled Perf Counters`, which could be us - Gauge - Number of bytes written in input/output operations generated by all processes +MDS Rank Metrics +---------------- + +Per-MDS-rank metrics are also exported as :ref:`Labeled Perf Counters` with +a ``rank`` label. These describe the MDS daemon process itself, not any +particular client or subvolume. + +.. list-table:: MDS Rank Metrics + :widths: 25 25 75 + :header-rows: 1 + + * - Name + - Type + - Description + * - ``cpu_usage`` + - Gauge + - Sum of per-core CPU utilisation for the MDS process (``100`` == one + fully saturated core; values can exceed 100 on multi-core systems). + * - ``open_requests`` + - Gauge + - Number of metadata requests currently in flight for this MDS rank. + Subvolume Metrics ----------------- -CephFS exports subvolume metrics as :ref:`Labeled Perf Counters`, which could be used to monitor the subvolume performance. CephFS exports the below subvolume metrics. -Subvolume metrics are aggregated within sliding window of 30 seconds (default value, configurable via the ``subv_metrics_window_interval`` parameter, see :ref:`MDS config reference`). -In large Ceph clusters with tens of thousands of subvolumes, this parameter also helps clean up stale metrics. -When a subvolume’s sliding window becomes empty, it's metrics are removed and not reported as “zero” values, reducing memory usage and computational overhead. +CephFS exports subvolume metrics as :ref:`Labeled Perf Counters`, which can +be used to monitor subvolume performance and utilization. + +**I/O Performance Metrics** + +I/O performance metrics (IOPS, throughput, latency) are aggregated within a +sliding window of 30 seconds by default. This interval is configurable via +the ``subv_metrics_window_interval`` parameter (see :ref:`MDS config reference`). +In large clusters with tens of thousands of subvolumes, this parameter also +controls when stale metrics are evicted: once the sliding window becomes empty +(no I/O activity), the metrics entry is removed rather than reporting zeros, +reducing memory usage and computational overhead. + +.. important:: + + **Metadata operations do NOT trigger metric updates.** Only actual data + I/O (reads and writes to file contents) updates the sliding window and + keeps the subvolume metrics entry active. Metadata-only operations such + as ``mkdir``, ``rmdir``, ``unlink``, ``rename``, ``chmod``, ``chown``, + ``setxattr``, ``stat``, and ``ls`` do **not** generate I/O metrics. + + This means: + + - If a subvolume has only metadata activity (e.g., creating/deleting + files without writing data), its I/O metrics will show zeros or the + entry may be evicted after the window expires. + - After deleting files, the ``used_bytes`` value will not immediately + reflect the freed space until either new data I/O occurs or the MDS + broadcasts updated quota information. + +**Utilization Metrics** + +In addition to I/O performance, subvolume metrics include utilization +counters: + +- ``quota_bytes``: The configured quota limit for the subvolume + (0 if unlimited). +- ``used_bytes``: Current space usage based on the inode's recursive + statistics (``rstat.rbytes``). + +These values are updated when the MDS broadcasts quota information to +clients. The ``used_bytes`` reflects the recursive byte count of the +subvolume root inode, which is maintained by the MDS as files are created, +modified, or deleted. However, since metric reporting depends on I/O +activity to keep entries alive, the utilization values are only reported +while the subvolume has active I/O within the sliding window. .. list-table:: Subvolume Metrics :widths: 25 25 75 @@ -99,6 +163,12 @@ When a subvolume’s sliding window becomes empty, it's metrics are removed and * - ``avg_write_lat_msec`` - Gauge - Average write latency in milliseconds. + * - ``quota_bytes`` + - Gauge + - Configured quota limit in bytes (0 if no quota/unlimited). + * - ``used_bytes`` + - Gauge + - Current space usage in bytes (recursive byte count of subvolume root). Getting Metrics =============== @@ -165,7 +235,7 @@ The metrics could be scraped from the MDS admin socket as well as using the tell } ] -The subvolume metrics are dumped as a part of the same command. The ``mds_subvolume_metrics`` section in the output of ``counter dump`` command displays the metrics for each client as shown below:: +The subvolume metrics are dumped as a part of the same command. The ``mds_subvolume_metrics`` section in the output of ``counter dump`` command displays the metrics for each subvolume as shown below:: "mds_subvolume_metrics": [ { @@ -179,6 +249,8 @@ The subvolume metrics are dumped as a part of the same command. The ``mds_subvol "avg_read_lat_msec": 0, "avg_write_iops": 1564, "avg_write_tp_Bps": 6408316, - "avg_write_lat_msec": 338 + "avg_write_lat_msec": 338, + "quota_bytes": 10737418240, + "used_bytes": 1073741824 } } \ No newline at end of file -- 2.47.3