--- /dev/null
+=======================
+ EXTBLKDEV, FCM plugin
+=======================
+
+.. index:: bluestore; extblkdev
+
+ExtBlkDev
+=========
+
+With Pacific release Ceph was extended to handle thinly-provisioned and compressing drives.
+The extension is made in form of loadable plugins that are part of Ceph codebase: ``src/extblkdev``.
+
+At startup OSD checks if specific plugins are enabled, as controlled by:
+
+.. confval:: osd_extblkdev_plugins
+
+To let OSD load and check for presence of FCM devices set:
+
+.. code-block::
+
+ osd_extblkdev_plugins = fcm
+
+
+Mode of operation
+-----------------
+
+Thin-provisioned and compressing drives have different capacity for data than what is advertised
+as addressable logical space. It means that a compressing drive might advertise its size as 10TB,
+but in reality only operate on 2TB of physical NAND.
+ExtBlkDev plugins provide an additional information channel between devices and BlueStore,
+giving Ceph report current disk logical / physical, used / available states.
+
+This information is critical for OSDs: they might otherwise try to write more data to the device
+than it can store, which will critically fail when it is really REALLY full.
+Without current and accurate drive usage the configurables are useless:
+
+.. confval:: mon_osd_full_ratio
+
+.. confval:: mon_osd_nearfull_ratio
+
+The ExtBlkDev extension is implemented on ``BlockDevice`` level, common for ``Main``, ``DB``, and ``WAL`` devices.
+However, BlueStore only uses it in context of ``Main`` device.
+For the ``DB`` and ``WAL`` devices, the plugin is detected and loaded, but BlueStore does not interact with it.
+
+Plugin persistence
+------------------
+
+.. note::
+
+ The examples below are derived from an actual deployment of the FCM plugin.
+
+Having the proper plugin loaded is critical to OSD operation.
+
+.. prompt:: # bash
+
+ ceph osd df
+
+.. code-block::
+
+ ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
+ 0 ssd 5.07739 1.00000 31 TiB 4.3 TiB 4.2 TiB 11 KiB 12 GiB 26 TiB 13.92 0.77 129 up
+ 1 ssd 5.07739 1.00000 31 TiB 4.3 TiB 4.2 TiB 12 KiB 12 GiB 26 TiB 13.93 0.77 128 up
+ 2 ssd 5.07739 1.00000 31 TiB 4.3 TiB 4.2 TiB 12 KiB 12 GiB 26 TiB 13.93 0.77 129 up
+ 3 ssd 5.07739 1.00000 31 TiB 4.2 TiB 4.2 TiB 12 KiB 12 GiB 26 TiB 13.90 0.77 128 up
+ 4 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 4.2 TiB 12 KiB 12 GiB 2.9 TiB 43.08 2.38 128 up
+ 5 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 4.3 TiB 12 KiB 12 GiB 2.9 TiB 43.25 2.39 128 up
+ 6 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 4.2 TiB 12 KiB 12 GiB 2.9 TiB 43.06 2.38 128 up
+ 7 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 4.3 TiB 14 KiB 12 GiB 2.9 TiB 43.44 2.40 129 up
+ TOTAL 143 TiB 26 TiB 34 TiB 101 KiB 94 GiB 117 TiB 18.09 1027
+ MIN/MAX VAR: 0.77/2.40 STDDEV: 18.00
+
+In the above case OSDs 0-3 run without the plugin; OSDs 4-7 have the plugin loaded and active.
+All eight OSDs participate in a single 6+2 erasure coded pool and therefore carry equivalent utilization.
+With EC6+2 this is moot, but for other pool types the balancer will prioritize OSDs 0-3 which seem to be mostly empty
+trying to offload seemingly overburdened OSDs 4-7.
+
+Tentacle provides a new configurable that signals BlueStore to expect that an ExtBlkDev plugin will be in use.
+
+.. confval:: bluestore_use_ebd
+
+The ExtBlkDev infrastructure adds new category of health warnings:
+
+.. code-block::
+
+ HEALTH_WARN 8 OSD(s) reporting problems with ExtBlkDev plugin;
+ [WRN] EXTBLKDEV: 8 OSD(s) reporting problems with ExtBlkDev plugin
+
+
+With the ``bluestore_use_ebd`` set to ``true``, two EXTBLKDEV health warnings may appear:
+
+ 1) "plugin 'fcm' not loaded"
+
+ 2) "plugin 'fcm' used on mkfs, but now uses plugin 'disabled'"
+
+
+Having the configuration option set to ``true`` does not require that a plugin is used.
+If a plugin is not found during OSD creation, this is still a proper and acceptable state.
+However, once a plugin is detected during the OSD's creation, its usage becomes mandatory.
+Should plugin presence be required for specific OSD, one can check for it:
+
+.. prompt:: bash #
+
+ ceph osd metadata osd.2
+
+.. code-block::
+
+ ...
+ "bluestore_bdev_fcm": "true",
+ ...
+
+or
+
+.. prompt:: bash #
+
+ ceph-bluestore-tool --path dev/osd4/ show-label
+
+.. code-block::
+
+ ...
+ "extblkdev": "fcm",
+ ...
+
+The first check is somewhat weaker than the second.
+In a specific error case that the plugin was detected and memorized at OSD creation,
+but is not loaded, one will not see the value in OSD metadata. Instead a health warning appears.
+The second check never fails. It simply stores the setting in BlueStore label metadata.
+
+.. _fix-use-ebd:
+
+A fix
+-----
+
+If ``bluestore_use_ebd=true`` was not set when OSDs were created on devices that should run with a plugin,
+it may be set after the fact with a command of the following form:
+
+.. prompt:: bash #
+
+ ceph-bluestore-tool --dev dev/osd4/block -k extblkdev -v fcm set-label-key
+
+.. _fcm-plugin:
+
+FCM plugin
+==========
+
+Ceph releases beginning with Tentacle provide a plugin to operate with FlashCoreModule devices (FCMs).
+Like all ExtBlkDev plugins, it reports true disk usage to BlueStore, effectively overriding
+BlueStore reported metrics of used and available device space.
+FCM reported metrics are in physical NAND size, not in logical space size.
+
+In addition the plugin reports fixed stats via OSD metadata:
+
+.. prompt:: bash #
+
+ ceph osd metadata osd.2
+
+.. code-block::
+
+ ...
+ "bluestore_bdev_fcm_device_logical_size": "33599931809792",
+ "bluestore_bdev_fcm_device_physical_size": "5582606303232",
+ "bluestore_bdev_fcm_partition_logical_size": "33599931809792",
+ "bluestore_bdev_fcm_partition_physical_size": "5582606303232",
+ ...
+
+And dynamic current state via performance counters:
+
+.. prompt:: bash #
+
+ ceph tell osd.6 perf dump extblkdev
+
+.. code-block::
+
+ {
+ "extblkdev": {
+ "fcm": 0, <- value is irrelevant, it is the name that counts
+ "dev_phy_size": 5582606303232,
+ "dev_log_size": 33599931809792,
+ "dev_phy_util": 2403758935846,
+ "dev_log_util": 4714543366144,
+ "part_phy_size": 5582606303232,
+ "part_log_size": 33599931809792,
+ "part_phy_avail": 3178847367386,
+ "part_log_avail": 28885388443648
+ }
+ }
+
+The FCM plugin provides three new health warnings in the EXTBLKDEV class.
+
+ 1) "failed accessing FCM utilization log" :ref:`fcm-permissions`
+
+ 2) "bdev_enable_discard not enabled - free space will leak" :ref:`fcm-discard`
+
+ 3) "multivolume fcm will not work properly" :ref:`fcm-multivolume`
+
+
+.. _fcm-permissions:
+
+FCM permissions
+---------------
+
+The FCM plugin talks to the NVMe controller within each storage device.
+In Linux, this is a privileged operation, tied to the `cap_sys_admin` privilege.
+To get the privilege do either:
+
+ a) Run Ceph OSD as root.
+
+ b) Grant ``setcap "cap_sys_admin=p" ceph-osd``
+ and run with ``ceph-osd --set-keepcaps=true`` .
+
+.. _fcm-discard:
+
+FCM discard
+-----------
+
+FCM devices implement compression; their physical NAND capacity is much smaller than their logical device size.
+Conventional SSDs storing data that is no longer used by BlueStore experience may experience suboptimal performance.
+With FCMs, orphaned data represents a direct loss of available capacity. To prevent this problem, one should always set:
+
+.. code-block::
+
+ bdev_enable_discard = true
+
+.. _fcm-multivolume:
+
+FCM multivolume
+---------------
+
+BlueStore works well when there is a one-to-one mapping between FCM devices to BlueStore block devices.
+
+Splitting an FCM device in half by partitioning does not create any boundary for the controller.
+Filling one partition will affect available space in the other partition.
+Two partitions together will have the total logical capacity exactly the same, but available physical NAND will be counted twice.
+This will most likely confuse both balancers and operators.
+
+ NOTE: The FCM plugin does not detect this condition.
+
+Combining multiple FCM devices using the device mapper, LVM, or Linux MD software RAID is not recommended. When doing so,
+BlueStore has no visibility of the topology of the backing block devices and cannot know to address storage by specific logical device offsets.
+For regular drives it does not matter; when the physical:logical ratio is 1:1 BlueStore will eventually run out of logical space on first drive
+and without even noticing start consuming the second drive.
+With FCMs, and any other compressing drives, it is no longer the case. When BlueStore is not running out of logical space on first drive,
+it will keep asking the drive to accomodate more data. The second drive has it, but first does not.
+BlueStore will eventually fail to allocate space and fail with a hard ``-ENOSPC`` error.
+
+Trying to run BlueStore with compound block device when one of them is an FCM will raise the warning.
+An OSD cannot operate long-term in this way. It must be redeployed.
+
+
+
+Redeploy and discard
+--------------------
+
+When one redeploys an FCM OSD without first formatting or issuing a whole-device discard operation,
+BlueStore will be logically empty, but the underlying device will still hold the burden of previously held data.
+
+.. code-block::
+
+ ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
+ 0 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 956 KiB 24 KiB 26 MiB 2.9 TiB 43.16 1.00 1 up
+ 1 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 376 KiB 20 KiB 26 MiB 2.9 TiB 43.14 1.00 0 up
+ 2 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 956 KiB 20 KiB 26 MiB 2.9 TiB 43.15 1.00 1 up
+ 3 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 376 KiB 16 KiB 26 MiB 2.9 TiB 43.05 1.00 0 up
+ 4 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 376 KiB 14 KiB 26 MiB 2.9 TiB 43.08 1.00 0 up
+ 5 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 376 KiB 11 KiB 26 MiB 2.9 TiB 43.24 1.00 0 up
+ 6 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 376 KiB 7 KiB 26 MiB 2.9 TiB 43.05 1.00 0 up
+ 7 ssd 5.07739 1.00000 5.1 TiB 2.2 TiB 956 KiB 6 KiB 26 MiB 2.9 TiB 43.44 1.01 1 up
+ TOTAL 41 TiB 18 TiB 4.6 MiB 121 KiB 210 MiB 23 TiB 43.17 3
+ MIN/MAX VAR: 1.00/1.01 STDDEV: 0.12
+
+Solution:
+
+ a) ``nvme format`` before deploying
+ b) Set ``bluestore_discard_on_mkfs=true`` (Umbrella+).
+
+
+CRUSH weight
+------------
+
+This section describes :ref:`fix-use-ebd` when using the FCM plugin :ref:`fcm-plugin`.
+
+When an OSD is properly deployed on an FCM, the WEIGHT and SIZE values reflect the physical NAND capacity:
+
+.. code-block::
+
+ ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
+ 0 ssd 5.07739 1.00000 5.1 TiB 12 MiB 944 KiB 21 KiB 26 MiB 5.1 TiB 0 1.00 1 up
+
+When an OSD is deployed without the active FCM plugin, the WEIGHT and SIZE reflect logical advertised values:
+
+.. code-block::
+
+ ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
+ 0 ssd 30.55899 1.00000 31 TiB 27 MiB 908 KiB 16 KiB 26 MiB 31 TiB 0 1.01 1 up
+
+When a plugin is active for the underlying device, the result is that the SIZE
+is the proper physical NAND capacity, but the CRUSH weight is still locked to original value at deployment:
+
+.. code-block::
+
+ ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
+ 0 ssd 30.55899 1.00000 5.1 TiB 28 KiB 1.0 MiB 7 KiB 27 MiB 5.1 TiB 0 1.42 1 up
+
+The CRUSH weight of such OSDs must be adjusted.
projects / ceph.git / commitdiff