From 1e37037496c38aa26900af149d534d17af6ad591 Mon Sep 17 00:00:00 2001 From: Zac Dover Date: Fri, 30 Aug 2024 21:16:57 +1000 Subject: [PATCH] doc/ceph-volume: add spillover fix procedure Add a procedure that explains how, after an upgrade, to move bytes that have spilled over to a relatively slow device back to the faster device. This procedure was developed by Chris Dunlop on the [ceph-users] mailing list, here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/POPUFSZGXR3P2RPYPJ4WJ4HGHZ3QESF6/ Eugen Block requested the addition of this procedure to the documentation on 30 Aug 2024. Co-authored-by: Anthony D'Atri Signed-off-by: Zac Dover (cherry picked from commit 98618aaa1c8b786c7d240a210b62cc737fdb048d) --- doc/ceph-volume/lvm/newdb.rst | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/doc/ceph-volume/lvm/newdb.rst b/doc/ceph-volume/lvm/newdb.rst index dcc87fc8a740f..a8136c9886bb9 100644 --- a/doc/ceph-volume/lvm/newdb.rst +++ b/doc/ceph-volume/lvm/newdb.rst @@ -9,3 +9,48 @@ Logical volume name format is vg/lv. Fails if OSD has already got attached DB. Attach vgname/lvname as a DB volume to OSD 1:: ceph-volume lvm new-db --osd-id 1 --osd-fsid 55BD4219-16A7-4037-BC20-0F158EFCC83D --target vgname/new_db + +Reversing BlueFS Spillover to Slow Devices +------------------------------------------ + +Under certain circumstances, OSD RocksDB databases spill onto slow storage and +the Ceph cluster returns specifics regarding BlueFS spillover warnings. ``ceph +health detail`` returns these spillover warnings. Here is an example of a +spillover warning:: + + osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device + +To move this DB metadata from the slower device to the faster device, take the +following steps: + +#. Expand the database's logical volume (LV): + + .. prompt:: bash # + + lvextend -l ${size} ${lv}/${db} ${ssd_dev} + +#. Stop the OSD: + + .. prompt:: bash # + + cephadm unit --fsid $cid --name osd.${osd} stop + +#. Run the ``bluefs-bdev-expand`` command: + + .. prompt:: bash # + + cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-${osd} + +#. Run the ``bluefs-bdev-migrate`` command: + + .. prompt:: bash # + + cephadm shell --fsid $cid --name osd.${osd} -- ceph-bluestore-tool bluefs-bdev-migrate --path /var/lib/ceph/osd/ceph-${osd} --devs-source /var/lib/ceph/osd/ceph-${osd}/block --dev-target /var/lib/ceph/osd/ceph-${osd}/block.db + +#. Restart the OSD: + + .. prompt:: bash # + + cephadm unit --fsid $cid --name osd.${osd} start + +.. note:: *The above procedure was developed by Chris Dunlop on the [ceph-users] mailing list, and can be seen in its original context here:* `[ceph-users] Re: Fixing BlueFS spillover (pacific 16.2.14) `_ -- 2.39.5