--- /dev/null
+.. _seastore:
+
+==========
+ SeaStore
+==========
+
+Goals and Basics
+================
+
+* Target NVMe devices. Not primarily concerned with pmem or HDD.
+* make use of SPDK for user-space driven IO
+* Use Seastar futures programming model to facilitate
+ run-to-completion and a sharded memory/processing model
+* Allow zero- (or minimal) data copying on read and write paths when
+ combined with a seastar-based messenger using DPDK
+
+Motivation and background
+-------------------------
+
+All flash devices are internally structured in terms of segments that
+can be written efficiently but must be erased in their entirety. The
+NVMe device generally has limited knowledge about what data in a
+segment is still "live" (hasn't been logically discarded), making the
+inevitable garbage collection within the device inefficient. We can
+design an on-disk layout that is friendly to GC at lower layers and
+drive garbage collection at higher layers.
+
+In principle a fine-grained discard could communicate our intent to
+the device, but in practice discard is poorly implemented in the
+device and intervening software layers.
+
+The basic idea is that all data will be stream out sequentially to
+large segments on the device. In the SSD hardware, segments are
+likely to be on the order of 100's of MB to tens of GB.
+
+SeaStore's logical segments would ideally be perfectly aligned with
+the hardware segments. In practice, it may be challenging to
+determine geometry and to sufficiently hint to the device that LBAs
+being written should be aligned to the underlying hardware. In the
+worst case, we can structure our logical segments to correspond to
+e.g. 5x the physical segment size so that we have about ~20% of our
+data misaligned.
+
+When we reach some utilization threshold, we mix cleaning work in with
+the ongoing write workload in order to evacuate live data from
+previously written segments. Once they are completely free we can
+discard the entire segment so that it can be erased and reclaimed by
+the device.
+
+The key is to mix a small bit of cleaning work with every write
+transaction to avoid spikes and variance in write latency.
+
+Data layout basics
+------------------
+
+One or more cores/shards will be reading and writing to the device at
+once. Each shard will have its own independent data it is operating
+on and stream to its own open segments. Devices that support streams
+can be hinted accordingly so that data from different shards is not
+mixed on the underlying media.
+
+Persistent Memory
+-----------------
+
+As the initial sequential design above matures, we'll introduce
+persistent memory support for metadata and caching structures.
+
+Design
+======
+
+The design is based heavily on both f2fs and btrfs. Each reactor
+manages its own root. Prior to reusing a segment, we rewrite any live
+blocks to an open segment.
+
+Because we are only writing sequentially to open segments, we must
+“clean” one byte of an existing segment for every byte written at
+steady state. Generally, we’ll need to reserve some portion of the
+usable capacity in order to ensure that write amplification remains
+acceptably low (20% for 2x? -- TODO: find prior work). As a design
+choice, we want to avoid a background gc scheme as it tends to
+complicate estimating operation cost and tends to introduce
+non-deterministic latency behavior. Thus, we want a set of structures
+that permits us to relocate blocks from existing segments inline with
+ongoing client IO.
+
+To that end, at a high level, we’ll maintain 2 basic metadata trees.
+First, we need a tree mapping ghobject_t->onode_t (onode_by_hobject).
+Second, we need a way to find live blocks within a segment and a way
+to decouple internal references from physical locations (lba_tree).
+
+Each onode contains xattrs directly as well as the top of the omap and
+extent trees (optimization: we ought to be able to fit small enough
+objects into the onode).
+
+Segment Layout
+--------------
+
+The backing storage is abstracted into a set of segments. Each
+segment can be in one of 3 states: empty, open, closed. The byte
+contents of a segment are a sequence of records. A record is prefixed
+by a header (including length and checksums) and contains a sequence
+of deltas and/or blocks. Each delta describes a logical mutation for
+some block. Each included block is an aligned extent addressable by
+<segment_id_t, segment_offset_t>. A transaction can be implemented by
+constructing a record combining deltas and updated blocks and writing
+it to an open segment.
+
+Note that segments will generally be large (something like >=256MB),
+so there will not typically be very many of them.
+
+record: [ header | delta | delta... | block | block ... ]
+segment: [ record ... ]
+
+See src/crimson/os/seastore/journal.h for Journal implementation
+See src/crimson/os/seastore/seastore_types.h for most seastore structures.
+
+Each shard will keep open N segments for writes
+
+- HDD: N is probably 1 on one shard
+- NVME/SSD: N is probably 2/shard, one for "journal" and one for
+ finished data records as their lifetimes are different.
+
+I think the exact number to keep open and how to partition writes
+among them will be a tuning question -- gc/layout should be flexible.
+Where practical, the goal is probably to partition blocks by expected
+lifetime so that a segment either has long lived or short lived
+blocks.
+
+The backing physical layer is exposed via a segment based interface.
+See src/crimson/os/seastore/segment_manager.h
+
+Journal and Atomicity
+---------------------
+
+One open segment is designated to be the journal. A transaction is
+represented by an atomically written record. A record will contain
+blocks written as part of the transaction as well as deltas which
+are logical mutations to existing physical extents. Transaction deltas
+are always written to the journal. If the transaction is associated
+with blocks written to other segments, final record with the deltas
+should be written only once the other blocks are persisted. Crash
+recovery is done by finding the segment containing the beginning of
+the current journal, loading the root node, replaying the deltas, and
+loading blocks into the cache as needed.
+
+See src/crimson/os/seastore/journal.h
+
+Block Cache
+-----------
+
+Every block is in one of two states:
+
+- clean: may be in cache or not, reads may cause cache residence or
+ not
+- dirty: the current version of the record requires overlaying deltas
+ from the journal. Must be fully present in the cache.
+
+Periodically, we need to trim the journal (else, we’d have to replay
+journal deltas from the beginning of time). To do this, we need to
+create a checkpoint by rewriting the root blocks and all currently
+dirty blocks. Note, we can do journal checkpoints relatively
+infrequently, and they needn’t block the write stream.
+
+Note, deltas may not be byte range modifications. Consider a btree
+node structured with keys to the left and values to the right (common
+trick for improving point query/key scan performance). Inserting a
+key/value into that node at the min would involve moving a bunch of
+bytes, which would be expensive (or verbose) to express purely as a
+sequence of byte operations. As such, each delta indicates the type
+as well as the location of the corresponding extent. Each block
+type can therefore implement CachedExtent::apply_delta as appropriate.
+
+See src/os/crimson/seastore/cached_extent.h.
+See src/os/crimson/seastore/cache.h.
+
+GC
+---
+
+Prior to reusing a segment, we must relocate all live blocks. Because
+we only write sequentially to empty segments, for every byte we write
+to currently open segments, we need to clean a byte of an existing
+closed segment. As a design choice, we’d like to avoid background
+work as it complicates estimating operation cost and has a tendency to
+create non-deterministic latency spikes. Thus, under normal operation
+each seastore reactor will be inserting enough work to clean a segment
+at the same rate as incoming operations.
+
+In order to make this cheap for sparse segments, we need a way to
+positively identify dead blocks. Thus, for every block written, an
+entry will be added to the lba tree with a pointer to the previous lba
+in the segment. Any transaction that moves a block or modifies the
+reference set of an existing one will include deltas/blocks required
+to update the lba tree to update or remove the previous block
+allocation. The gc state thus simply needs to maintain an iterator
+(of a sort) into the lba tree segment linked list for segment
+currently being cleaned and a pointer to the next record to be
+examined -- records not present in the allocation tree may still
+contain roots (like allocation tree blocks) and so the record metadata
+must be checked for a flag indicating root blocks.
+
+For each transaction, we evaluate a heuristic function of the
+currently available space and currently live space in order to
+determine whether we need to do cleaning work (could be simply a range
+of live/used space ratios).
+
+TODO: there is not yet a GC implementation
+
+Logical Layout
+==============
+
+Using the above block and delta semantics, we build two root level trees:
+- onode tree: maps hobject_t to onode_t
+- lba_tree: maps lba_t to lba_range_t
+
+Each of the above structures is comprised of blocks with mutations
+encoded in deltas. Each node of the above trees maps onto a block.
+Each block is either physically addressed (root blocks and the
+lba_tree nodes) or is logically addressed (everything else).
+Physically addressed blocks are located by a paddr_t: <segment_id_t,
+segment_off_t> tuple and are marked as physically addressed in the
+record. Logical blocks are addressed by laddr_t and require a lookup in
+the lba_tree to address.
+
+Because the cache/transaction machinery lives below the level of the
+lba tree, we can represent atomic mutations of the lba tree and other
+structures by simply including both in a transaction.
+
+LBAManager/BtreeLBAManager
+--------------------------
+
+Implementations of the LBAManager interface are responsible for managing
+the logical->physical mapping -- see crimson/os/seastore/lba_manager.h.
+
+The BtreeLBAManager implements this interface directly on top of
+Journal and SegmentManager using a wandering btree approach.
+
+Because SegmentManager does not let us predict the location of a
+committed record (a property of both SMR and Zone devices), references
+to blocks created within the same transaction will necessarily be
+*relative* addresses. The BtreeLBAManager maintains an invariant by
+which the in-memory copy of any block will contain only absolute
+addresses when !is_pending() -- on_commit and complete_load fill in
+absolute addresses based on the actual block addr and on_delta_write
+does so based on the just committed record. When is_pending(), if
+is_initial_pending references in memory are block_relative (because
+they will be written to the original block location) and
+record_relative otherwise (value will be written to delta).
+
+TransactionManager
+------------------
+
+The TransactionManager is responsible for presenting a unified
+interface on top of the Journal, SegmentManager, Cache, and
+LBAManager. Users can allocate and mutate extents based on logical
+addresses with segment cleaning handled in the background.
+
+See crimson/os/seastore/transaction_manager.h
+
+Next Steps
+==========
+
+Journal
+-------
+
+- Support for scanning a segment to find physically addressed blocks
+- Add support for trimming the journal and releasing segments.
+
+Cache
+-----
+
+- Support for rewriting dirty blocks
+
+ - Need to add support to CachedExtent for finding/updating
+ dependent blocks
+ - Need to add support for adding dirty block writout to
+ try_construct_record
+
+LBAManager
+----------
+
+- Add support for pinning
+- Add segment -> laddr for use in GC
+- Support for locating remaining used blocks in segments
+
+GC
+---
+
+- Initial implementation
+- Support in BtreeLBAManager for tracking used blocks in segments
+- Heuristic for identifying segments to clean
+
+Other
+------
+
+- Add support for periodically generating a journal checkpoint.
+- Onode tree
+- Extent tree
+- Remaining ObjectStore integration
+
+ObjectStore considerations
+==========================
+
+Splits, merges, and sharding
+----------------------------
+
+One of the current ObjectStore requirements is to be able to split a
+collection (PG) in O(1) time. Starting in mimic, we also need to be
+able to merge two collections into one (i.e., exactly the reverse of a
+split).
+
+However, the PGs that we split into would hash to different shards of
+the OSD in the current sharding scheme. One can imagine replacing
+that sharding scheme with a temporary mapping directing the smaller
+child PG to the right shard since we generally then migrate that PG to
+another OSD anyway, but this wouldn't help us in the merge case where
+the constituent pieces may start out on different shards and
+ultimately need to be handled in the same collection (and be operated
+on via single transactions).
+
+This suggests that we likely need a way for data written via one shard
+to "switch ownership" and later be read and managed by a different
+shard.
+
+
+
+++ /dev/null
-.. _seastore:
-
-==========
- SeaStore
-==========
-
-Goals and Basics
-================
-
-* Target NVMe devices. Not primarily concerned with pmem or HDD.
-* make use of SPDK for user-space driven IO
-* Use Seastar futures programming model to facilitate
- run-to-completion and a sharded memory/processing model
-* Allow zero- (or minimal) data copying on read and write paths when
- combined with a seastar-based messenger using DPDK
-
-Motivation and background
--------------------------
-
-All flash devices are internally structured in terms of segments that
-can be written efficiently but must be erased in their entirety. The
-NVMe device generally has limited knowledge about what data in a
-segment is still "live" (hasn't been logically discarded), making the
-inevitable garbage collection within the device inefficient. We can
-design an on-disk layout that is friendly to GC at lower layers and
-drive garbage collection at higher layers.
-
-In principle a fine-grained discard could communicate our intent to
-the device, but in practice discard is poorly implemented in the
-device and intervening software layers.
-
-The basic idea is that all data will be stream out sequentially to
-large segments on the device. In the SSD hardware, segments are
-likely to be on the order of 100's of MB to tens of GB.
-
-SeaStore's logical segments would ideally be perfectly aligned with
-the hardware segments. In practice, it may be challenging to
-determine geometry and to sufficiently hint to the device that LBAs
-being written should be aligned to the underlying hardware. In the
-worst case, we can structure our logical segments to correspond to
-e.g. 5x the physical segment size so that we have about ~20% of our
-data misaligned.
-
-When we reach some utilization threshold, we mix cleaning work in with
-the ongoing write workload in order to evacuate live data from
-previously written segments. Once they are completely free we can
-discard the entire segment so that it can be erased and reclaimed by
-the device.
-
-The key is to mix a small bit of cleaning work with every write
-transaction to avoid spikes and variance in write latency.
-
-Data layout basics
-------------------
-
-One or more cores/shards will be reading and writing to the device at
-once. Each shard will have its own independent data it is operating
-on and stream to its own open segments. Devices that support streams
-can be hinted accordingly so that data from different shards is not
-mixed on the underlying media.
-
-Persistent Memory
------------------
-
-As the initial sequential design above matures, we'll introduce
-persistent memory support for metadata and caching structures.
-
-Design
-======
-
-The design is based heavily on both f2fs and btrfs. Each reactor
-manages its own root. Prior to reusing a segment, we rewrite any live
-blocks to an open segment.
-
-Because we are only writing sequentially to open segments, we must
-“clean” one byte of an existing segment for every byte written at
-steady state. Generally, we’ll need to reserve some portion of the
-usable capacity in order to ensure that write amplification remains
-acceptably low (20% for 2x? -- TODO: find prior work). As a design
-choice, we want to avoid a background gc scheme as it tends to
-complicate estimating operation cost and tends to introduce
-non-deterministic latency behavior. Thus, we want a set of structures
-that permits us to relocate blocks from existing segments inline with
-ongoing client IO.
-
-To that end, at a high level, we’ll maintain 2 basic metadata trees.
-First, we need a tree mapping ghobject_t->onode_t (onode_by_hobject).
-Second, we need a way to find live blocks within a segment and a way
-to decouple internal references from physical locations (lba_tree).
-
-Each onode contains xattrs directly as well as the top of the omap and
-extent trees (optimization: we ought to be able to fit small enough
-objects into the onode).
-
-Segment Layout
---------------
-
-The backing storage is abstracted into a set of segments. Each
-segment can be in one of 3 states: empty, open, closed. The byte
-contents of a segment are a sequence of records. A record is prefixed
-by a header (including length and checksums) and contains a sequence
-of deltas and/or blocks. Each delta describes a logical mutation for
-some block. Each included block is an aligned extent addressable by
-<segment_id_t, segment_offset_t>. A transaction can be implemented by
-constructing a record combining deltas and updated blocks and writing
-it to an open segment.
-
-Note that segments will generally be large (something like >=256MB),
-so there will not typically be very many of them.
-
-record: [ header | delta | delta... | block | block ... ]
-segment: [ record ... ]
-
-See src/crimson/os/seastore/journal.h for Journal implementation
-See src/crimson/os/seastore/seastore_types.h for most seastore structures.
-
-Each shard will keep open N segments for writes
-
-- HDD: N is probably 1 on one shard
-- NVME/SSD: N is probably 2/shard, one for "journal" and one for
- finished data records as their lifetimes are different.
-
-I think the exact number to keep open and how to partition writes
-among them will be a tuning question -- gc/layout should be flexible.
-Where practical, the goal is probably to partition blocks by expected
-lifetime so that a segment either has long lived or short lived
-blocks.
-
-The backing physical layer is exposed via a segment based interface.
-See src/crimson/os/seastore/segment_manager.h
-
-Journal and Atomicity
----------------------
-
-One open segment is designated to be the journal. A transaction is
-represented by an atomically written record. A record will contain
-blocks written as part of the transaction as well as deltas which
-are logical mutations to existing physical extents. Transaction deltas
-are always written to the journal. If the transaction is associated
-with blocks written to other segments, final record with the deltas
-should be written only once the other blocks are persisted. Crash
-recovery is done by finding the segment containing the beginning of
-the current journal, loading the root node, replaying the deltas, and
-loading blocks into the cache as needed.
-
-See src/crimson/os/seastore/journal.h
-
-Block Cache
------------
-
-Every block is in one of two states:
-
-- clean: may be in cache or not, reads may cause cache residence or
- not
-- dirty: the current version of the record requires overlaying deltas
- from the journal. Must be fully present in the cache.
-
-Periodically, we need to trim the journal (else, we’d have to replay
-journal deltas from the beginning of time). To do this, we need to
-create a checkpoint by rewriting the root blocks and all currently
-dirty blocks. Note, we can do journal checkpoints relatively
-infrequently, and they needn’t block the write stream.
-
-Note, deltas may not be byte range modifications. Consider a btree
-node structured with keys to the left and values to the right (common
-trick for improving point query/key scan performance). Inserting a
-key/value into that node at the min would involve moving a bunch of
-bytes, which would be expensive (or verbose) to express purely as a
-sequence of byte operations. As such, each delta indicates the type
-as well as the location of the corresponding extent. Each block
-type can therefore implement CachedExtent::apply_delta as appropriate.
-
-See src/os/crimson/seastore/cached_extent.h.
-See src/os/crimson/seastore/cache.h.
-
-GC
----
-
-Prior to reusing a segment, we must relocate all live blocks. Because
-we only write sequentially to empty segments, for every byte we write
-to currently open segments, we need to clean a byte of an existing
-closed segment. As a design choice, we’d like to avoid background
-work as it complicates estimating operation cost and has a tendency to
-create non-deterministic latency spikes. Thus, under normal operation
-each seastore reactor will be inserting enough work to clean a segment
-at the same rate as incoming operations.
-
-In order to make this cheap for sparse segments, we need a way to
-positively identify dead blocks. Thus, for every block written, an
-entry will be added to the lba tree with a pointer to the previous lba
-in the segment. Any transaction that moves a block or modifies the
-reference set of an existing one will include deltas/blocks required
-to update the lba tree to update or remove the previous block
-allocation. The gc state thus simply needs to maintain an iterator
-(of a sort) into the lba tree segment linked list for segment
-currently being cleaned and a pointer to the next record to be
-examined -- records not present in the allocation tree may still
-contain roots (like allocation tree blocks) and so the record metadata
-must be checked for a flag indicating root blocks.
-
-For each transaction, we evaluate a heuristic function of the
-currently available space and currently live space in order to
-determine whether we need to do cleaning work (could be simply a range
-of live/used space ratios).
-
-TODO: there is not yet a GC implementation
-
-Logical Layout
-==============
-
-Using the above block and delta semantics, we build two root level trees:
-- onode tree: maps hobject_t to onode_t
-- lba_tree: maps lba_t to lba_range_t
-
-Each of the above structures is comprised of blocks with mutations
-encoded in deltas. Each node of the above trees maps onto a block.
-Each block is either physically addressed (root blocks and the
-lba_tree nodes) or is logically addressed (everything else).
-Physically addressed blocks are located by a paddr_t: <segment_id_t,
-segment_off_t> tuple and are marked as physically addressed in the
-record. Logical blocks are addressed by laddr_t and require a lookup in
-the lba_tree to address.
-
-Because the cache/transaction machinery lives below the level of the
-lba tree, we can represent atomic mutations of the lba tree and other
-structures by simply including both in a transaction.
-
-LBAManager/BtreeLBAManager
---------------------------
-
-Implementations of the LBAManager interface are responsible for managing
-the logical->physical mapping -- see crimson/os/seastore/lba_manager.h.
-
-The BtreeLBAManager implements this interface directly on top of
-Journal and SegmentManager using a wandering btree approach.
-
-Because SegmentManager does not let us predict the location of a
-committed record (a property of both SMR and Zone devices), references
-to blocks created within the same transaction will necessarily be
-*relative* addresses. The BtreeLBAManager maintains an invariant by
-which the in-memory copy of any block will contain only absolute
-addresses when !is_pending() -- on_commit and complete_load fill in
-absolute addresses based on the actual block addr and on_delta_write
-does so based on the just committed record. When is_pending(), if
-is_initial_pending references in memory are block_relative (because
-they will be written to the original block location) and
-record_relative otherwise (value will be written to delta).
-
-TransactionManager
-------------------
-
-The TransactionManager is responsible for presenting a unified
-interface on top of the Journal, SegmentManager, Cache, and
-LBAManager. Users can allocate and mutate extents based on logical
-addresses with segment cleaning handled in the background.
-
-See crimson/os/seastore/transaction_manager.h
-
-Next Steps
-==========
-
-Journal
--------
-
-- Support for scanning a segment to find physically addressed blocks
-- Add support for trimming the journal and releasing segments.
-
-Cache
------
-
-- Support for rewriting dirty blocks
-
- - Need to add support to CachedExtent for finding/updating
- dependent blocks
- - Need to add support for adding dirty block writout to
- try_construct_record
-
-LBAManager
-----------
-
-- Add support for pinning
-- Add segment -> laddr for use in GC
-- Support for locating remaining used blocks in segments
-
-GC
----
-
-- Initial implementation
-- Support in BtreeLBAManager for tracking used blocks in segments
-- Heuristic for identifying segments to clean
-
-Other
-------
-
-- Add support for periodically generating a journal checkpoint.
-- Onode tree
-- Extent tree
-- Remaining ObjectStore integration
-
-ObjectStore considerations
-==========================
-
-Splits, merges, and sharding
-----------------------------
-
-One of the current ObjectStore requirements is to be able to split a
-collection (PG) in O(1) time. Starting in mimic, we also need to be
-able to merge two collections into one (i.e., exactly the reverse of a
-split).
-
-However, the PGs that we split into would hash to different shards of
-the OSD in the current sharding scheme. One can imagine replacing
-that sharding scheme with a temporary mapping directing the smaller
-child PG to the right shard since we generally then migrate that PG to
-another OSD anyway, but this wouldn't help us in the merge case where
-the constituent pieces may start out on different shards and
-ultimately need to be handled in the same collection (and be operated
-on via single transactions).
-
-This suggests that we likely need a way for data written via one shard
-to "switch ownership" and later be read and managed by a different
-shard.
-
-
-
projects / ceph.git / commitdiff