Goals and Basics
================
-* Target NVMe devices. Not primarily concerned with pmem or HDD.
+* Target NVMe devices primarily, with interest in HDDs as well.
* make use of SPDK for user-space driven IO
* Use Seastar futures programming model to facilitate
run-to-completion and a sharded memory/processing model
-* Allow zero- (or minimal) data copying on read and write paths when
- combined with a seastar-based messenger using DPDK
-
-Motivation and background
--------------------------
-
-All flash devices are internally structured in terms of segments that
-can be written efficiently but must be erased in their entirety. The
-NVMe device generally has limited knowledge about what data in a
-segment is still "live" (hasn't been logically discarded), making the
-inevitable garbage collection within the device inefficient. We can
-design an on-disk layout that is friendly to GC at lower layers and
-drive garbage collection at higher layers.
-
-In principle a fine-grained discard could communicate our intent to
-the device, but in practice discard is poorly implemented in the
-device and intervening software layers.
-
-The basic idea is that all data will be stream out sequentially to
-large segments on the device. In the SSD hardware, segments are
-likely to be on the order of 100's of MB to tens of GB.
-
-SeaStore's logical segments would ideally be perfectly aligned with
-the hardware segments. In practice, it may be challenging to
-determine geometry and to sufficiently hint to the device that LBAs
-being written should be aligned to the underlying hardware. In the
-worst case, we can structure our logical segments to correspond to
-e.g. 5x the physical segment size so that we have about ~20% of our
-data misaligned.
-
-When we reach some utilization threshold, we mix cleaning work in with
-the ongoing write workload in order to evacuate live data from
-previously written segments. Once they are completely free we can
-discard the entire segment so that it can be erased and reclaimed by
-the device.
-
-The key is to mix a small bit of cleaning work with every write
-transaction to avoid spikes and variance in write latency.
-
+* Allow zero (or minimal) data copying on read and write paths when
+ combined with a Seastar-based messenger using DPDK
+
+
Data layout basics
------------------
can be hinted accordingly so that data from different shards is not
mixed on the underlying media.
-Persistent Memory
------------------
-
-As the initial sequential design above matures, we'll introduce
-persistent memory support for metadata and caching structures.
+Architecture
+============
+
+.. .. Mermaid source of seastore_arch.svg
+.. flowchart TB
+ %% Level 1
+ SeaStore[SeaStore]
+ %% Level 2
+ subgraph Logical
+ direction LR
+ OnodeManager[OnodeManager]
+ OmapManager[OmapManager]
+ ObjectDataHandler[ObjectDataHandler]
+ Ellipsis[...]
+ end
+ %% Level 3
+ TransactionManager[TransactionManager]
+ %% Level 4
+ subgraph Physical
+ direction LR
+ Journal[Journal]
+ LBAManager[LBAManager]
+ ExtentPlacementManager[ExtentPlacementManager]
+ end
+ %% Level 5
+ subgraph EPM
+ direction LR
+ Cache[Cache]
+ AsyncCleaner[AsyncCleaner]
+ Devices[Devices]
+ end
+ %% Order
+ SeaStore --> OmapManager
+ OmapManager --> TransactionManager
+ TransactionManager --> LBAManager
+ ExtentPlacementManager --> AsyncCleaner
+ ExtentPlacementManager --> Cache
+ ExtentPlacementManager --> Devices
+ ExtentPlacementManager --> EPM
+ %% Hide Lines
+ linkStyle 0 stroke:transparent
+ linkStyle 1 stroke:transparent
+ linkStyle 2 stroke:transparent
+ linkStyle 3 stroke:transparent
+ linkStyle 4 stroke:transparent
+ linkStyle 5 stroke:transparent
+
+.. image:: seastore_arch.svg
Design
======
-The design is based heavily on both f2fs and btrfs. Each reactor
-manages its own root. Prior to reusing a segment, we rewrite any live
+Each reactor manages its own root. Prior to reusing a segment, we rewrite any live
blocks to an open segment.
Because we are only writing sequentially to open segments, we must
“clean” one byte of an existing segment for every byte written at
steady state. Generally, we’ll need to reserve some portion of the
usable capacity in order to ensure that write amplification remains
-acceptably low (20% for 2x? -- TODO: find prior work). As a design
-choice, we want to avoid a background gc scheme as it tends to
-complicate estimating operation cost and tends to introduce
-non-deterministic latency behavior. Thus, we want a set of structures
-that permits us to relocate blocks from existing segments inline with
-ongoing client IO.
-
-To that end, at a high level, we’ll maintain 2 basic metadata trees.
-First, we need a tree mapping ghobject_t->onode_t (onode_by_hobject).
-Second, we need a way to find live blocks within a segment and a way
-to decouple internal references from physical locations (lba_tree).
-
-Each onode contains xattrs directly as well as the top of the omap and
-extent trees (optimization: we ought to be able to fit small enough
-objects into the onode).
-
-Segment Layout
---------------
-
-The backing storage is abstracted into a set of segments. Each
-segment can be in one of 3 states: empty, open, closed. The byte
-contents of a segment are a sequence of records. A record is prefixed
-by a header (including length and checksums) and contains a sequence
-of deltas and/or blocks. Each delta describes a logical mutation for
-some block. Each included block is an aligned extent addressable by
-<segment_id_t, segment_offset_t>. A transaction can be implemented by
-constructing a record combining deltas and updated blocks and writing
-it to an open segment.
-
-Note that segments will generally be large (something like >=256MB),
-so there will not typically be very many of them.
-
-record: [ header | delta | delta... | block | block ... ]
-segment: [ record ... ]
-
-See src/crimson/os/seastore/journal.h for Journal implementation
-See src/crimson/os/seastore/seastore_types.h for most seastore structures.
-
-Each shard will keep open N segments for writes
-
-- HDD: N is probably 1 on one shard
-- NVME/SSD: N is probably 2/shard, one for "journal" and one for
- finished data records as their lifetimes are different.
-
-I think the exact number to keep open and how to partition writes
-among them will be a tuning question -- gc/layout should be flexible.
-Where practical, the goal is probably to partition blocks by expected
-lifetime so that a segment either has long lived or short lived
-blocks.
-
-The backing physical layer is exposed via a segment based interface.
-See src/crimson/os/seastore/segment_manager.h
-
-Journal and Atomicity
----------------------
-
-One open segment is designated to be the journal. A transaction is
-represented by an atomically written record. A record will contain
-blocks written as part of the transaction as well as deltas which
-are logical mutations to existing physical extents. Transaction deltas
-are always written to the journal. If the transaction is associated
+acceptably low (20% for 2x? -- TODO: find prior work). We support both
+background garbage collection, which allows us to exploit periods of
+lower I/O demand, and inline relocation of blocks from existing
+segments as part of ongoing client I/O.
+
+
+.. _metadata-structures:
+
+Metadata Structures
+-------------------
+
+.. Mermaid source of seastore.svg
+.. flowchart TD
+ %% Root
+ Root((Root))
+ %% Physical
+ subgraph Physical["Physically Addressed"]
+ LBABtree["LBA Btree"]
+ BackrefTree["Backref Tree"]
+ end
+ %% Logical
+ subgraph Logical["Logically Addressed"]
+ OnodeTree["Onode Tree"]
+ OnodeN["Onode"]
+ %% Per-Onode structures
+ Omap1["OMAP
+ B-tree Root
+ (LBA)"]
+ Omap2["XATTR
+ B-tree Root
+ (LBA)"]
+ Extents["Data Extents
+ (LBA Range)"]
+ %% Mapping and containment
+ OnodeTree -- "map: ghobject_t → Onode" --> OnodeN
+ OnodeN --> Omap1
+ OnodeN --> Omap2
+ OnodeN --> Extents
+ end
+ %% Top-level links
+ Root --> OnodeTree
+ Root --> LBABtree
+ Root --> BackrefTree
+ %% Styling
+ classDef logical fill:#e0f7fa,stroke:#333,stroke-width:1px;
+ classDef physical fill:#f1f8e9,stroke:#333,stroke-width:1px;
+ class OnodeTree,OnodeN,Omap1,Omap2,Extents logical;
+ class LBABtree,BackrefTree physical;
+
+
+.. image:: seastore.svg
+
+
+
+
+**Extents**:
+
+ An extent is an allocation unit on disk. All extents are either **physically addressed** (``paddr_t``)
+ (direct physical location on disk) or **logically addressed** (``laddr_t``).
+ Logically addressed extents are referred to by logical rather than physical addresses, allowing them to
+ be relocated freely without affecting the data structures they belong to.
+ The vast majority of extents fall into this category.
+
+ There are multiple types of extents, such as ``ROOT``, ``OMAP_LEAF``, and ``OBJECT_DATA_BLOCK``
+ (see ``extent_types_t`` for the full list).
+ Each extent type can interpret **journal deltas** differently, see :ref:`journal` for details.
+
+ *Interface:* ``ObjectDataHandler``
+
+
+**Onode Tree**:
+ Maps ``ghobject_t`` to ``Onode``. The Onode layout (``onode_layout_t``) will consist the
+ Omap Btrees and the ``extent_types_t::OBJECT_DATA_BLOCK`` extents.
+
+ *Interface:* ``FLTreeOnodeManager``
+
+**Omap Btree**:
+ Stores ``omap_type_t`` entries for each object.
+
+ *Interface:* ``BtreeOMapManager``
+
+ The same ``BtreeOMapManager`` interface is used for handling multiple omap types:
+
+ ``omap_type_t::OMAP``, ``omap_type_t::XATTRS``, and ``omap_type_t::LOG``.
+
+ Each type is stored in its own tree, but the interface remains consistent to avoid reimplementation for each type.
+
+**LBA Btree**:
+ Mappings of a logically addressed extents to either a physical address or
+ another (indirect) logical address.
+
+ *Mapping:* ``laddr_t`` --> ``pladdr_t``
+
+ *Extents:* ``LADDR_INTERNAL`` and ``LADDR_LEAF``
+
+ *Interface:* ``LBAManager``
+
+.. note::
+ ``pladdr_t`` represents either ``paddr_t`` (direct mapping) or ``laddr_t`` (indirect mapping).
+ See :ref:`lbamanager`.
+
+
+**Backref Tree**:
+ Maps physically addressed extents to logical addresses.
+
+ *Mapping:* ``paddr_t`` --> ``laddr_t``
+
+ *Extents:* ``BACKREF_INTERNAL`` and ``BACKREF_LEAF``
+
+ *Interface:* ``BackrefManager``
+
+.. note::
+ The Backref Tree is used only for **garbage collection** on **Segmented Seastore device types**
+ (See SegmmentCleaner).
+
+
+.. _device-types:
+
+Device Types
+------------
+
+Configured via the ``seastore_main_device_type`` option, the device types are
+separated into **Segmented** and **RBM** backend types as follows:
+
+
+- **backend_type_t::SEGMENTED**:
+
+ **Segmented Backend Motivation:**:
+
+ All flash devices are internally structured in terms of erasure blocks that
+ can be written efficiently but must be erased in their entirety. The
+ NVMe device generally has limited knowledge about what data in a
+ segment is still "live" (hasn't been logically discarded). This limited
+ knowledge reduces garbage-collection efficiency when data with mixed lifetimes
+ shares the same segments. We can design an on-device layout that is friendly to GC at lower layers and
+ drive garbage collection at higher layers.
+
+ The basic idea is that all data will be stream out sequentially to
+ large segments on the device. In the SSD hardware, segments are
+ likely to be on the order of hundreds of MB to tens of GB.
+
+ SeaStore's logical segments would ideally be perfectly aligned with
+ the hardware segments. In practice, it may be challenging to
+ determine geometry and to sufficiently hint to the device that LBAs
+ being written should be aligned to the underlying hardware.
+ In the worst case, we can structure our logical segments to span multiple physical segments sizes,
+ to avoid the metadata cost of representing each erasure block independently.
+
+ When we reach some utilization threshold, we mix cleaning work in with
+ the ongoing write workload in order to evacuate live data from
+ previously written segments. Once they are completely free we can
+ discard the entire segment so that it can be erased and reclaimed by
+ the device.
+
+ The key is to mix a small bit of cleaning work with every write
+ transaction to avoid spikes and variance in write latency.
+
+
+ **Segment Layout:**:
+
+ The backing storage is abstracted into a set of segments.
+
+ The types of segements are:
+
+ * ``segment_type_t::JOURNAL`` - See Journal Types in :ref:`journal`
+ * ``segment_type_t::OOL`` - Data segments
+ * ``segment_type_t::NULL_SEG`` - Initial segment state
+
+ Each segment can be in one of Three states:
+
+ * ``segment_state_t::EMPTY``
+ * ``segment_state_t::OPEN``
+ * ``segment_state_t::CLOSED``.
+
+ Each segment is prefixed with ``segment_header_t`` in the first block and suffixed with a ``segment_tail_t``.
+
+ Note that segments will generally be large (something like >=256MB),
+ so there will not typically be very many of them.
+
+ Each shard will keep open N segments for writes
+
+ - HDD: N is probably 1 on one shard
+ - NVME/SSD: N is probably 2/shard, one for "journal" and one for
+ finished data records as their lifetimes are different.
+
+ I think the exact number to keep open and how to partition writes
+ among them will be a tuning question -- GC/layout should be flexible.
+ Where practical, the goal is to partition blocks by expected
+ lifetime so that a segment either has long lived or short lived
+ blocks.
+
+ Segment: [ ``segment_header_t`` | ``record_t`` | ``record_t`` | ``record_t`` ... ]
+
+ Used for:
+
+ * ``device_type_t::SSD`` (default)
+ * ``device_type_t::HDD``
+ * ``device_type_t::ZBD``
+
+ Preferred for sequential writes. Can't overwrite written extents (similar to ZNS).
+
+ *Interface:* ``SegmentManager``
+
+
+- **backend_type_t::RANDOM_BLOCK**:
+
+ Used for:
+
+ * ``device_type_t::RANDOM_BLOCK_SSD``
+
+ Preferred for fast NVMe devices where overwrites are efficient enough
+ that log structured updates aren't worth the overhead.
+
+ *Interface:* ``RBMDevice``
+
+
+.. _journal:
+
+Journal
+-------
+
+*Interface:* ``Journal``
+
+See ``src/crimson/os/seastore/journal.h``
+
+Responsible for atomically writing (``Journal::submit_record``) and replaying
+(``Journal::reply``) journal records. A journal record ``record_t`` is a struct containing deltas and extents.
+
+
+**Journal Records**:
+
+The byte contents of segments are a sequence of ``record_t`` records.
+A group of records is prefixed by a ``record_group_header_t`` header which includes the number of records,
+length of the data and checksum. A record group is a set of records that are written to disk in a single write operation.
+
+
+See ``RecordSubmitter::flush_current_batch()``.
+
+* ``record_type_t::OOL``:
+
+ Data records. These records are not padded and do not contain **any** metadata (See ``encode_records``).
+
+* ``record_type_t::JOURNAL``:
+
+ Used in the Journal segment only.
+ Each record is then prefixed by ``record_header_t`` which contain the number of deltas and extents sequence in this record.
+
+ Each delta (``delta_info_t``) describes a logical mutation for some block.
+ Each included block is an aligned extent addressable by ``<segment_id_t, segment_off_t>``.
+ A transaction can be implemented by constructing a record combining deltas and updated blocks and writing
+ it to an open segment.
+
+Record: [ ``record_header_t`` | ``record_type_t::delta`` | ``record_type_t::delta``... | ``record_type_t::extents`` | ``record_type_t::extents`` ... ]
+
+See ``src/crimson/os/seastore/seastore_types.h`` for most SeaStore structures.
+
+**Journal Trimming**:
+
+
+The journal trimming interface is ``JournalTrimmer`` and is implemented in ``JournalTrimmerImpl``.
+Periodically, we trim the journal (else, we’d have to replay
+journal deltas from the beginning of time). Journal entries can be trimmed once the
+extents they reference have been rewritten elsewhere (See: ``JournalTrimmerImpl::trim_dirty()``).
+
+**Journal Deltas**:
+
+Deltas are logical mutations to existing extents.
+Note, deltas are not always byte range modifications. Consider a btree
+node structured with keys to the left and values to the right (common
+trick for improving point query/key scan performance). Inserting a
+key/value into that node at the min would involve moving a bunch of
+bytes, which would be expensive (or verbose) to express purely as a
+sequence of byte operations. As such, each delta indicates the type
+as well as the location of the corresponding extent. Each block
+type can therefore implement ``CachedExtent::apply_delta`` as appropriate.
+For example, ``OMapInnerNode::apply_delta`` or ``ObjectDataBlock::apply_delta``.
+
+A transaction is represented by an atomically written record.
+Transaction deltas are always written to the journal. If the transaction is associated
with blocks written to other segments, final record with the deltas
should be written only once the other blocks are persisted. Crash
recovery is done by finding the segment containing the beginning of
the current journal, loading the root node, replaying the deltas, and
loading blocks into the cache as needed.
-See src/crimson/os/seastore/journal.h
+**Journal Types**:
+
+According to the :ref:`device-types`, the journal has two implementations:
+
+* **SegmentedJournal**: Used for Segmented devices.
+
+ * One open segment is designated as the journal (``segment_type_t::JOURNAL``).
+ * The size of the journal segment is equal to ``seastore_segment_size``.
+ * A record can contain:
+
+ * Blocks written as part of the transaction (inline)
+ * Journal deltas
+
+ * Small or short-lived extents (e.g., leaf nodes) can be written inline to the journal segment.
+ This is possible when the delta fits in the journal record and when the extent is expected
+ to be updated again soon. Longer-lived or large extents are written to out-of-line segments
+ (``segment_type_t::OOL``). Writing out-of-line extents avoids unnecessary
+ data movement during segment cleaning.
+ * ``record_type_t::OOL`` will never contain metadata. See ``record_size_t::get_raw_mdlength()`` for example.
+
+
+* **CircularBoundedJournal**: Used for RBM devices.
+
+ * Journal is fixed size (configured via ``seastore_cbjournal_size``)
+ * A record will contain **only** Journal deltas as all extents are written out-of-line, outside the journal.
+ * Trimming the journal does not require moving data since no data is written inline.
+
Block Cache
-----------
-Every block is in one of two states:
-- clean: may be in cache or not, reads may cause cache residence or
- not
-- dirty: the current version of the record requires overlaying deltas
- from the journal. Must be fully present in the cache.
+Every block is in one of the following states:
-Periodically, we need to trim the journal (else, we’d have to replay
-journal deltas from the beginning of time). To do this, we need to
-create a checkpoint by rewriting the root blocks and all currently
-dirty blocks. Note, we can do journal checkpoints relatively
-infrequently, and they needn’t block the write stream.
+- ``extent_state_t::INITIAL_WRITE_PENDING``:
-Note, deltas may not be byte range modifications. Consider a btree
-node structured with keys to the left and values to the right (common
-trick for improving point query/key scan performance). Inserting a
-key/value into that node at the min would involve moving a bunch of
-bytes, which would be expensive (or verbose) to express purely as a
-sequence of byte operations. As such, each delta indicates the type
-as well as the location of the corresponding extent. Each block
-type can therefore implement CachedExtent::apply_delta as appropriate.
+ In ``Transaction::write_set`` and ``fresh_block_list``; has ``prior_instance`` under rewrite.
+
+- ``extent_state_t::MUTATION_PENDING``:
+
+ In ``Transaction::write_set`` and ``mutated_block_list``; has ``prior_instance``.
+
+- ``extent_state_t::CLEAN``:
+
+ In ``Cache::extent_index`` and in ``Transaction::read_set`` during write.
+ Contents match disk; ``version == 0``.
+
+- ``extent_state_t::DIRTY``:
+
+ Must be fully present in the cache. Contents do not match disk; ``version > 0``.
+ The current version of the record requires overlaying deltas from the journal.
+
+- ``extent_state_t::EXIST_CLEAN``:
+
+ Similar to ``CLEAN``, but its metadata has not yet been persisted to disk.
+ Present in ``Transaction::write_set`` and ``existing_block_list``.
+ After the transaction commits, the state becomes ``CLEAN`` and the extent is added to the Cache.
+ Modifying such extents will transition the state to ``EXIST_MUTATION_PENDING``.
+
+- ``extent_state_t::EXIST_MUTATION_PENDING``:
+
+ Similar to ``MUTATION_PENDING``, but ``prior_instance`` is empty.
+ Present in ``Transaction::write_set``, ``existing_block_list``, and ``mutated_block_list``.
+ After the transaction commits, the state becomes ``DIRTY`` and the extent is added to the Cache.
+
+- ``extent_state_t::INVALID``:
+
+ No ``ExtentIndex`` set.
+
+
+SeaStore uses caching not only for performance but also for **correctness** purposes.
+
+The cache can represent the *projected* outcome of to-be-committed transactions.
+This projected state can be used during reads (``TransactionManager::read_pin``) to verify checksums.
+It ensures that persisted extent checksum matches the projected transaction outcome when handling
+a fully loaded extent during reads. For testing purposes, we can also force a full re-load on reads
+(see ``Cache::check_full_extent_integrity``).
+
+
+Caching also keeps ``extent_state_t::DIRTY`` extents, those whose contents differ
+from what is persisted, **pinned in memory**. Pinning is managed through the
+``ExtentPinboard`` interface, which supports two caching algorithms: **LRU** and
+**2Q** (configured via ``seastore_cachepin_type``).
+
+
+The caching layer additionally detects **transaction conflicts** that require
+retries (see ``Cache::mark_transaction_conflicted``). However, this mechanism is
+planned to be removed due to high retry rate write workloads.
+
+
+**Interface:** ``Cache``
+
+
+See: ``src/crimson/os/seastore/cache.h``
See src/os/crimson/seastore/cached_extent.h.
See src/os/crimson/seastore/cache.h.
determine whether we need to do cleaning work (could be simply a range
of live/used space ratios).
-TODO: there is not yet a GC implementation
Logical Layout
==============
-Using the above block and delta semantics, we build two root level trees:
-- onode tree: maps hobject_t to onode_t
-- lba_tree: maps lba_t to lba_range_t
+The above block and delta semantics are used to build the root level trees
+described here :ref:`metadata-structures`.
Each of the above structures is comprised of blocks with mutations
-encoded in deltas. Each node of the above trees maps onto a block.
-Each block is either physically addressed (root blocks and the
-lba_tree nodes) or is logically addressed (everything else).
-Physically addressed blocks are located by a paddr_t: <segment_id_t,
-segment_off_t> tuple and are marked as physically addressed in the
-record. Logical blocks are addressed by laddr_t and require a lookup in
+encoded in deltas. Each node of the above trees maps onto an extent.
+Each extent is either physically addressed (root blocks and the
+lba_tree nodes) or is logically-addressed (everything else).
+Physically addressed extents are located by a ``paddr_t: <segment_id_t,
+segment_off_t>`` tuple and are marked as physically addressed in the
+record. logically-addressed are addressed by ``laddr_t`` and require a lookup in
the lba_tree to address.
Because the cache/transaction machinery lives below the level of the
lba tree, we can represent atomic mutations of the lba tree and other
structures by simply including both in a transaction.
+.. _lbamanager:
+
LBAManager/BtreeLBAManager
--------------------------
Implementations of the LBAManager interface are responsible for managing
-the logical->physical mapping -- see crimson/os/seastore/lba_manager.h.
+the logical to physical mapping.
+
+See: crimson/os/seastore/lba_manager.h
The BtreeLBAManager implements this interface directly on top of
Journal and SegmentManager using a wandering btree approach.
they will be written to the original block location) and
record_relative otherwise (value will be written to delta).
+To support cloning, there are two kinds of LBA Mappings:
+
+* Direct LBA Mapping: the ``pladdr_t`` in the value of is the ``paddr_t`` of the corresponding extent.
+
+* Indirect LBA Mapping: the ``pladdr_t`` in the value is an ``laddr_t`` pointing
+ to the direct LBA mapping that's pointing to the actual paddr of the extent being searched.
+
+.. _transactionmanager:
+
TransactionManager
------------------
LBAManager. Users can allocate and mutate extents based on logical
addresses with segment cleaning handled in the background.
-See crimson/os/seastore/transaction_manager.h
+See ``crimson/os/seastore/transaction_manager.h``
+
+.. _extentplacementmanager:
+
+ExtentPlacementManager
+----------------------
+
+Manages extents across backing devices, allowing SeaStore to work with multiple,
+potentially heterogeneous devices of potentially different performance classes.
+
+
+See: ``crimson/os/seastore/extent_placement_manager.h``
+
+ExtentPlacementManager is responsible for:
+
+**Placing extents**:
+ The extent is placed on a segment based on several factors, such as:
+
+
+ * ``data_category_t``: either ``data_category_t::DATA`` or ``data_category_t::METADATA``.
+
+ * `placement_hint_t`: whether a mutation or retirement is expected, the options are
+ ``placement_hint_t::HOT`` and ``placement_hint_t::COLD``. ``placement_hint_t::REWRITE``
+ is used after ExtentPlacementManager (EPM) decisions for internal rewrites.
+
+
+**Background processes**:
+ See ``ExtentPlacementManager::BackgroundProcess::do_background_cycle()``.
+
+ *Interface:* ``AsyncCleaner`` which holds the ``background_callback``.
+
+
+ There are two implementations of the ``AsyncCleaner`` interface.
+ According to the :ref:`device-types`, these are **RBMCleaner** and **SegmentCleaner**.
+
+ * **SegmentCleaner**: For Segmented devices, performs garbage collection on the segemets
+ on that device (See `SegmentCleaner::clean_space()`). Logical extents are remapped within the :ref:`lbamanager`
+ and physical extents are updated accordingly. The SegmmentCleaner is also responisble for throttling GC work
+ in order to avoid abrupt pauses and maintain smooth IO latenices.
+
+**Tiering**:
+
+ .. note::
+ Tiering is supported only for Segmented devices, using a dedicated cold
+ ``SegmentCleaner``. All RBM extents belong to the **same** static generation
+ (``rewrite_gen_t::OOL``) and are never adjusted.
+
+
+ When multiple devices are present, the EPM considers how long ago an extent was
+ written. Extents that are rarely written or read may be demoted to a cold tier.
+ This logic is implemented by ``rewrite_gen_t`` which is designed to group the similar
+ aged extents in the same segment (See ``adjust_generation``).
+
+ The number of tiers is set based on the configured ``seastore_hot_tier_generations`` and ``seastore_cold_tier_generations``.
+ Each genreation maps its own segment and has its own dedicated ``ExtentOolWriter`` writer (See ``generation_to_writer``).
Next Steps
==========
-Journal
--------
-
-- Support for scanning a segment to find physically addressed blocks
-- Add support for trimming the journal and releasing segments.
Cache
-----
LBAManager
----------
-- Add support for pinning
- Add segment -> laddr for use in GC
- Support for locating remaining used blocks in segments
GC
---
-- Initial implementation
- Support in BtreeLBAManager for tracking used blocks in segments
- Heuristic for identifying segments to clean
------
- Add support for periodically generating a journal checkpoint.
-- Onode tree
-- Extent tree
-- Remaining ObjectStore integration
ObjectStore considerations
==========================
--- /dev/null
+<svg id="mermaid-1767698989935-fs9j64rly" width="100%" xmlns="http://www.w3.org/2000/svg" class="flowchart" style="max-width: 864.65625px;" viewBox="0 0 864.65625 636" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#mermaid-1767698989935-fs9j64rly{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-1767698989935-fs9j64rly .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-1767698989935-fs9j64rly .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-1767698989935-fs9j64rly .error-icon{fill:#552222;}#mermaid-1767698989935-fs9j64rly .error-text{fill:#552222;stroke:#552222;}#mermaid-1767698989935-fs9j64rly .edge-thickness-normal{stroke-width:1px;}#mermaid-1767698989935-fs9j64rly .edge-thickness-thick{stroke-width:3.5px;}#mermaid-1767698989935-fs9j64rly .edge-pattern-solid{stroke-dasharray:0;}#mermaid-1767698989935-fs9j64rly .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-1767698989935-fs9j64rly .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-1767698989935-fs9j64rly .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-1767698989935-fs9j64rly .marker{fill:#333333;stroke:#333333;}#mermaid-1767698989935-fs9j64rly .marker.cross{stroke:#333333;}#mermaid-1767698989935-fs9j64rly svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-1767698989935-fs9j64rly p{margin:0;}#mermaid-1767698989935-fs9j64rly .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-1767698989935-fs9j64rly .cluster-label text{fill:#333;}#mermaid-1767698989935-fs9j64rly .cluster-label span{color:#333;}#mermaid-1767698989935-fs9j64rly .cluster-label span p{background-color:transparent;}#mermaid-1767698989935-fs9j64rly .label text,#mermaid-1767698989935-fs9j64rly span{fill:#333;color:#333;}#mermaid-1767698989935-fs9j64rly .node rect,#mermaid-1767698989935-fs9j64rly .node circle,#mermaid-1767698989935-fs9j64rly .node ellipse,#mermaid-1767698989935-fs9j64rly .node polygon,#mermaid-1767698989935-fs9j64rly .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-1767698989935-fs9j64rly .rough-node .label text,#mermaid-1767698989935-fs9j64rly .node .label text,#mermaid-1767698989935-fs9j64rly .image-shape .label,#mermaid-1767698989935-fs9j64rly .icon-shape .label{text-anchor:middle;}#mermaid-1767698989935-fs9j64rly .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-1767698989935-fs9j64rly .rough-node .label,#mermaid-1767698989935-fs9j64rly .node .label,#mermaid-1767698989935-fs9j64rly .image-shape .label,#mermaid-1767698989935-fs9j64rly .icon-shape .label{text-align:center;}#mermaid-1767698989935-fs9j64rly .node.clickable{cursor:pointer;}#mermaid-1767698989935-fs9j64rly .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-1767698989935-fs9j64rly .arrowheadPath{fill:#333333;}#mermaid-1767698989935-fs9j64rly .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-1767698989935-fs9j64rly .flowchart-link{stroke:#333333;fill:none;}#mermaid-1767698989935-fs9j64rly .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-1767698989935-fs9j64rly .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-1767698989935-fs9j64rly .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-1767698989935-fs9j64rly .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-1767698989935-fs9j64rly .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-1767698989935-fs9j64rly .cluster text{fill:#333;}#mermaid-1767698989935-fs9j64rly .cluster span{color:#333;}#mermaid-1767698989935-fs9j64rly div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-1767698989935-fs9j64rly .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-1767698989935-fs9j64rly rect.text{fill:none;stroke-width:0;}#mermaid-1767698989935-fs9j64rly .icon-shape,#mermaid-1767698989935-fs9j64rly .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-1767698989935-fs9j64rly .icon-shape p,#mermaid-1767698989935-fs9j64rly .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-1767698989935-fs9j64rly .icon-shape rect,#mermaid-1767698989935-fs9j64rly .image-shape rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-1767698989935-fs9j64rly .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-1767698989935-fs9j64rly .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-1767698989935-fs9j64rly :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}</style><g><marker id="mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="mermaid-1767698989935-fs9j64rly_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="mermaid-1767698989935-fs9j64rly_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="mermaid-1767698989935-fs9j64rly_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="mermaid-1767698989935-fs9j64rly_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="mermaid-1767698989935-fs9j64rly_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><g class="root"><g class="clusters"><g class="cluster " id="EPM" data-look="classic"><rect style="" x="303.8984375" y="524" width="552.7578125" height="104"></rect><g class="cluster-label " transform="translate(562.94140625, 524)"><foreignObject width="34.671875" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>EPM</p></span></div></foreignObject></g></g><g class="cluster " id="Physical" data-look="classic"><rect style="" x="73.375" y="370" width="756.5546875" height="104"></rect><g class="cluster-label " transform="translate(421.86328125, 370)"><foreignObject width="59.578125" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>Physical</p></span></div></foreignObject></g></g><g class="cluster " id="Logical" data-look="classic"><rect style="" x="8" y="112" width="827.3125" height="104"></rect><g class="cluster-label " transform="translate(396.3046875, 112)"><foreignObject width="50.703125" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>Logical</p></span></div></foreignObject></g></g></g><g class="edgePaths"><path d="M347.547,62L347.547,66.167C347.547,70.333,347.547,78.667,347.547,87C347.547,95.333,347.547,103.667,347.547,111.333C347.547,119,347.547,126,347.547,129.5L347.547,133" id="L_SeaStore_OmapManager_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style="stroke:transparent;fill:none;" marker-end="url(#mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd_transparent)"></path><path d="M347.547,191L347.547,195.167C347.547,199.333,347.547,207.667,347.547,216C347.547,224.333,347.547,232.667,347.547,240.333C347.547,248,347.547,255,347.547,258.5L347.547,262" id="L_OmapManager_TransactionManager_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style="stroke:transparent;fill:none;" marker-end="url(#mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd_transparent)"></path><path d="M347.547,320L347.547,324.167C347.547,328.333,347.547,336.667,347.547,345C347.547,353.333,347.547,361.667,347.547,369.333C347.547,377,347.547,384,347.547,387.5L347.547,391" id="L_TransactionManager_LBAManager_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style="stroke:transparent;fill:none;" marker-end="url(#mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd_transparent)"></path><path d="M585.211,449L583.5,453.167C581.79,457.333,578.37,465.667,576.659,474C574.949,482.333,574.949,490.667,574.949,499C574.949,507.333,574.949,515.667,574.949,523.333C574.949,531,574.949,538,574.949,541.5L574.949,545" id="L_ExtentPlacementManager_AsyncCleaner_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style="stroke:transparent;fill:none;" marker-end="url(#mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd_transparent)"></path><path d="M477.736,449L459.44,453.167C441.144,457.333,404.553,465.667,386.257,474C367.961,482.333,367.961,490.667,367.961,499C367.961,507.333,367.961,515.667,369.609,523.395C371.257,531.123,374.553,538.247,376.202,541.808L377.85,545.37" id="L_ExtentPlacementManager_Cache_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style="stroke:transparent;fill:none;" marker-end="url(#mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd_transparent)"></path><path d="M682.958,449L696.332,453.167C709.706,457.333,736.455,465.667,749.829,474C763.203,482.333,763.203,490.667,763.203,499C763.203,507.333,763.203,515.667,763.203,523.333C763.203,531,763.203,538,763.203,541.5L763.203,545" id="L_ExtentPlacementManager_Devices_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style="stroke:transparent;fill:none;" marker-end="url(#mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd_transparent)"></path><path d="M532.528,449L522.688,453.167C512.847,457.333,493.167,465.667,483.327,474C473.486,482.333,473.486,490.667,473.486,498.333C473.486,506,473.486,513,473.486,516.5L473.486,520" id="L_ExtentPlacementManager_EPM_0" class=" edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style="" marker-end="url(#mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd)"></path></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" transform="translate(0, 0)"><foreignObject width="0" height="0"><div style="stroke: transparent; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml" class="labelBkg"><span style=";stroke:transparent;color:none" class="edgeLabel "></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" transform="translate(0, 0)"><foreignObject width="0" height="0"><div style="stroke: transparent; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml" class="labelBkg"><span style=";stroke:transparent;color:none" class="edgeLabel "></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" transform="translate(0, 0)"><foreignObject width="0" height="0"><div style="stroke: transparent; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml" class="labelBkg"><span style=";stroke:transparent;color:none" class="edgeLabel "></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" transform="translate(0, 0)"><foreignObject width="0" height="0"><div style="stroke: transparent; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml" class="labelBkg"><span style=";stroke:transparent;color:none" class="edgeLabel "></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" transform="translate(0, 0)"><foreignObject width="0" height="0"><div style="stroke: transparent; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml" class="labelBkg"><span style=";stroke:transparent;color:none" class="edgeLabel "></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" transform="translate(0, 0)"><foreignObject width="0" height="0"><div style="stroke: transparent; display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;" xmlns="http://www.w3.org/1999/xhtml" class="labelBkg"><span style=";stroke:transparent;color:none" class="edgeLabel "></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel "></span></div></foreignObject></g></g></g><g class="nodes"><g class="node default " id="flowchart-SeaStore-0" transform="translate(347.546875, 35)"><rect class="basic label-container" style="" x="-63.35546875" y="-27" width="126.7109375" height="54"></rect><g class="label" style="" transform="translate(-33.35546875, -12)"><rect></rect><foreignObject width="66.7109375" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>SeaStore</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-OnodeManager-1" transform="translate(128.59375, 164)"><rect class="basic label-container" style="" x="-85.59375" y="-27" width="171.1875" height="54"></rect><g class="label" style="" transform="translate(-55.59375, -12)"><rect></rect><foreignObject width="111.1875" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>OnodeManager</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-OmapManager-2" transform="translate(347.546875, 164)"><rect class="basic label-container" style="" x="-83.359375" y="-27" width="166.71875" height="54"></rect><g class="label" style="" transform="translate(-53.359375, -12)"><rect></rect><foreignObject width="106.71875" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>OmapManager</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-ObjectDataHandler-3" transform="translate(578.94140625, 164)"><rect class="basic label-container" style="" x="-98.03515625" y="-27" width="196.0703125" height="54"></rect><g class="label" style="" transform="translate(-68.03515625, -12)"><rect></rect><foreignObject width="136.0703125" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>ObjectDataHandler</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-Ellipsis-4" transform="translate(763.64453125, 164)"><rect class="basic label-container" style="" x="-36.66796875" y="-27" width="73.3359375" height="54"></rect><g class="label" style="" transform="translate(-6.66796875, -12)"><rect></rect><foreignObject width="13.3359375" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>...</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-TransactionManager-5" transform="translate(347.546875, 293)"><rect class="basic label-container" style="" x="-103.07421875" y="-27" width="206.1484375" height="54"></rect><g class="label" style="" transform="translate(-73.07421875, -12)"><rect></rect><foreignObject width="146.1484375" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>TransactionManager</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-Journal-6" transform="translate(164.61328125, 422)"><rect class="basic label-container" style="" x="-56.23828125" y="-27" width="112.4765625" height="54"></rect><g class="label" style="" transform="translate(-26.23828125, -12)"><rect></rect><foreignObject width="52.4765625" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>Journal</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-LBAManager-7" transform="translate(347.546875, 422)"><rect class="basic label-container" style="" x="-76.6953125" y="-27" width="153.390625" height="54"></rect><g class="label" style="" transform="translate(-46.6953125, -12)"><rect></rect><foreignObject width="93.390625" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>LBAManager</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-ExtentPlacementManager-8" transform="translate(596.29296875, 422)"><rect class="basic label-container" style="" x="-122.05078125" y="-27" width="244.1015625" height="54"></rect><g class="label" style="" transform="translate(-92.05078125, -12)"><rect></rect><foreignObject width="184.1015625" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>ExtentPlacementManager</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-Cache-9" transform="translate(392.0234375, 576)"><rect class="basic label-container" style="" x="-53.125" y="-27" width="106.25" height="54"></rect><g class="label" style="" transform="translate(-23.125, -12)"><rect></rect><foreignObject width="46.25" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>Cache</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-AsyncCleaner-10" transform="translate(574.94921875, 576)"><rect class="basic label-container" style="" x="-79.80078125" y="-27" width="159.6015625" height="54"></rect><g class="label" style="" transform="translate(-49.80078125, -12)"><rect></rect><foreignObject width="99.6015625" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>AsyncCleaner</p></span></div></foreignObject></g></g><g class="node default " id="flowchart-Devices-11" transform="translate(763.203125, 576)"><rect class="basic label-container" style="" x="-58.453125" y="-27" width="116.90625" height="54"></rect><g class="label" style="" transform="translate(-28.453125, -12)"><rect></rect><foreignObject width="56.90625" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel "><p>Devices</p></span></div></foreignObject></g></g></g></g><marker id="mermaid-1767698989935-fs9j64rly_flowchart-v2-pointEnd_transparent" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;" stroke="transparent" fill="transparent"></path></marker></g></svg>
\ No newline at end of file
projects / ceph.git / commitdiff
