--- /dev/null
+===========================
+Erasure coding enhancements
+===========================
+
+Objectives
+==========
+
+Our objective is to improve the performance of erasure coding, in particular
+for small random accesses to make it more viable to use erasure coding pools
+for storing block and file data.
+
+We are looking to reduce the number of OSD read and write accesses per client
+I/O (sometimes referred to as I/O amplification), reduce the amount of network
+traffic between OSDs (network bandwidth) and reduce I/O latency (time to
+complete read and write I/O operations). We expect the changes will also
+provide modest reductions to CPU overheads.
+
+While the changes are focused on enhancing small random accesses, some
+enhancements will provide modest benefits for larger I/O accesses and for
+object storage.
+
+The following sections give a brief description of the improvements we are
+looking to make. Please see the later design sections for more details
+
+Current Read Implementation
+---------------------------
+
+For reference this is how erasure code reads currently work
+
+.. ditaa::
+
+ RADOS Client
+ * Current code reads all data chunks
+ ^ * Discards unneeded data
+ | * Returns requested data to client
+ +----+----+
+ | Discard | If data cannot be read then the coding parity
+ |unneeded | chunks are read as well and are used to reconstruct
+ | data | the data
+ +---------+
+ ^^^^
+ ||||
+ ||||
+ ||||
+ |||+----------------------------------------------+
+ ||+-------------------------------------+ |
+ |+----------------------------+ | |
+ | | | |
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+Note: All the diagrams illustrate a K=4 + M=2 configuration, however the
+concepts and techniques can be used for all K+M configurations.
+
+Partial Reads
+-------------
+
+If only a small amount of data is being read it is not necessary to read the
+whole stripe, for small I/Os ideally only a single OSD needs to be involved in
+reading the data. See also larger chunk size below.
+
+.. ditaa::
+
+ RADOS Client
+ * Optimize by only reading required chunks
+ ^ * For large chunk sizes and sub-chunk reads only
+ | read a sub-chunk
+ +----+----+
+ | Return | If data cannot be read then extra data and coding
+ | data | parity chunks are read as well and are used to
+ | | reconstruct the data
+ +---------+
+ ^
+ |
+ |
+ |
+ |
+ |
+ +----------------------------+
+ |
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+Pull Request https://github.com/ceph/ceph/pull/55196 is implementing most of
+this optimization, however it still issues full chunk reads.
+
+Current Overwrite Implementation
+--------------------------------
+
+For reference here is how erasure code overwrites currently work
+
+.. ditaa::
+
+ RADOS Client
+ | * Read all data chunks
+ | * Merges new data
+ +----v-----+ * Encodes new coding parities
+ | Read old | * Writes data and coding parities
+ |Merge new |
+ | Encode |-------------------------------------------------------------+
+ | Write |---------------------------------------------------+ |
+ +----------+ | |
+ ^|^|^|^| | |
+ |||||||+-------------------------------------------+ | |
+ ||||||+-------------------------------------------+| | |
+ |||||+-----------------------------------+ || | |
+ ||||+-----------------------------------+| || | |
+ |||+---------------------------+ || || | |
+ ||+---------------------------+| || || | |
+ |v |v |v |v v v
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+Partial Overwrites
+------------------
+
+Ideally we aim to be able to perform updates to erasure coded stripes by only
+updating a subset of the shards (those with modified data or coding
+parities). Avoiding performing unnecessary data updates on the other shards is
+easy, avoiding performing any metadata updates on the other shards is much
+harder (see design section on metadata updates).
+
+.. ditaa::
+
+ RADOS Client
+ | * Only read chunks that are not being overwritten
+ | * Merge new data
+ +----v-----+ * Encode new coding parities
+ | Read old | * Only write modified data and parity shards
+ |Merge new |
+ | Encode |-------------------------------------------------------------+
+ | Write |---------------------------------------------------+ |
+ +----------+ | |
+ ^ |^ ^ | |
+ | || | | |
+ | || +-------------------------------------------+ | |
+ | || | | |
+ | |+-----------------------------------+ | | |
+ | +---------------------------+ | | | |
+ | | | | | |
+ | v | | v v
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+This diagram is overly simplistic, only showing the data flows. The simplest
+implementation of this optimization retains a metadata write to every
+OSD. With more effort it is possible to reduce the number of metadata updates
+as well, see design below for more details.
+
+Parity-delta-write
+------------------
+
+A common technique used by block storage controllers implementing RAID5 and
+RAID6 is to implement what is sometimes called a parity delta write. When a
+small part of the stripe is being overwritten it is possible to perform the
+update by reading the old data, XORing this with the new data to create a
+delta and then read each coding parity, apply the delta and write the new
+parity. The advantage of this technique is that it can involve a lot less I/O,
+especially for K+M encodings with larger values of K. The technique is not
+specific to M=1 and M=2, it can be applied with any number of coding parities.
+
+.. ditaa::
+
+ Parity delta writes
+ * Read old data and XOR with new data to create a delta
+ RADOS Client * Read old encoding parities apply the delta and write
+ | the new encoding parities
+ |
+ | For K+M erasure codings where K is larger and M is small
+ | +-----+ +-----+ this is much more efficient
+ +->| XOR |-+->| GF |---------------------------------------------------+
+ +-+->| | | | |<------------------------------------------------+ |
+ | | +-----+ | +-----+ | |
+ | | | | |
+ | | | +-----+ | |
+ | | +->| XOR |-----------------------------------------+ | |
+ | | | |<--------------------------------------+ | | |
+ | | +-----+ | | | |
+ | | | | | |
+ | | | | | |
+ | +-------------------------------+ | | | |
+ +-------------------------------+ | | | | |
+ | | | | | |
+ | v | v | v
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+Direct Read I/O
+---------------
+
+We want clients to submit small I/Os directly to the OSD that stores the data
+rather than directing all I/O requests to the Primary OSD and have it issue
+requests to the secondary OSDs. By eliminating an intermediate hop this
+reduces network bandwidth and improves I/O latency
+
+.. ditaa::
+
+ RADOS Client
+ ^
+ |
+ +----+----+ Client sends small read requests directly to OSD
+ | Return | avoiding extra network hop via Primary
+ | data |
+ | |
+ +---------+
+ ^
+ |
+ |
+ |
+ |
+ |
+ |
+ |
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+
+.. ditaa::
+
+ RADOS Client
+ ^ ^
+ | |
+ +----+----+ +--+------+ Client breaks larger read
+ | Return | | Return | requests into separate
+ | data | | data | requests to multiple OSDs
+ | | | |
+ +---------+ +---------+ Note client loses atomicity
+ ^ ^ guarantees if this optimization
+ | | is used as an update could occur
+ | | between the two reads
+ | |
+ | |
+ | |
+ | |
+ | |
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+Distributed processing of writes
+--------------------------------
+
+The existing erasure code implementation processes write I/Os on the primary
+OSD, issuing both reads and writes to other OSDs to fetch and update data for
+other shards. This is perhaps the simplest implementation, but it uses a lot
+of network bandwidth. With parity-delta-writes it is possible to distribute
+the processing across OSDs to reduce network bandwidth.
+
+.. ditaa::
+
+ Performing the coding parity delta updates on the coding parity
+ OSD instead of the primary OSD reduces network bandwidth
+ RADOS Client
+ | Note: A naive implementation will increase latency by serializing
+ | the data and coding parity reads, for best performance these
+ | reads need to happen in parallel
+ | +-----+ +-----+
+ +->| XOR |-+------------------------------------------------------->| GF |
+ +-+->| | | | |
+ | | +-----+ | +----++
+ | | | +-----+ ^ |
+ | | +--------------------------------------------->| XOR | | |
+ | | | | | |
+ | | +---+-+ | |
+ | +-------------------------------+ ^ | | |
+ +-------------------------------+ | | | | |
+ | | | | | |
+ | | | | | |
+ | | | | | |
+ | | | | | |
+ | v | v | v
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+Direct Write I/O
+----------------
+
+.. ditaa::
+
+ RADOS Client
+ |
+ | Similarly Clients could direct small write I/Os
+ | to the OSD that needs updating
+ |
+ | +-----+ +-----+
+ +->| XOR |-+--------------------->| GF |
+ +-----+->| | | | |
+ | | +-----+ | +----++
+ | | | +-----+ ^ |
+ | | +----------->| XOR | | |
+ | | | | | |
+ | | +---+-+ | |
+ | | ^ | | |
+ | | | | | |
+ | | | | | |
+ | | | | | |
+ | | | | | |
+ | | | | | |
+ | v | v | v
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+This diagram is overly simplistic, only showing the data flows - direct writes
+are much harder to implement and will need control messages to the Primary to
+ensure writes to the same stripe are ordered correctly
+
+Larger chunk size
+-----------------
+
+The default chunk size is 4K, this is too small and means that small reads
+have to be split up and processed by many OSDs. It is more efficient if small
+I/Os can be serviced by a single OSD. Choosing a larger chunk size such as 64K
+or 256K and implementing partial reads and writes will fix this issue, but has
+the disadvantage that small sized RADOS objects get rounded up in size to a
+whole stripe of capacity.
+
+We would like the code to automatically choose what chunk size to use to
+optimize for both capacity and performance. Small objects should use a small
+chunk size like 4K, larger objects should use a larger chunk size.
+
+Code currently rounds up I/O sizes to multiples of the chunk size, which isn't
+an issue with a small chunk size. With a larger chunk size and partial
+reads/writes we should round up to the page size rather than the chunk size.
+
+Design
+======
+
+We will describe the changes we aim to make in three sections, the first
+section looks at the existing test tools for erasure coding and discusses the
+improvements we believe will be necessary to get good test coverage for the
+changes.
+
+The second section covers changes to the read and write I/O path.
+
+The third section discusses the changes to metadata to avoid the need to
+update metadata on all shards for each metadata update. While it is possible
+to implement many of the I/O path changes without reducing the number of
+metadata updates, there are bigger performance benefits if the number of
+metadata updates can be reduced as well.
+
+Test tools
+----------
+
+A survey of the existing test tools shows that there is insufficient coverage
+of erasure coding to be able to just make changes to the code and expect the
+existing CI pipelines to get sufficient coverage. Therefore one of the first
+steps will be to improve the test tools to be able to get better test
+coverage.
+
+Teuthology is the main test tool used to get test coverage and it relies
+heavily on the following tests for generating I/O:
+
+1. **rados** task - qa/tasks/rados.py. This uses ceph_test_rados
+ (src/test/osd/TestRados.cc) which can generate a wide mixture of different
+ rados operations. There is limited support for read and write I/Os,
+ typically using offset 0 although there is a chunked read command used by a
+ couple of tests.
+
+2. **radosbench** task - qa/tasks/radosbench.py. This uses the **rados bench**
+ (src/tools/rados/rados.cc and src/common/obj_bencher.cc). Can be used to
+ generate sequential and random I/O workloads, offset starts at 0 for
+ sequential I/O. I/O size can be set but is constant for whole test.
+
+3. **rbd_fio** task - qa/tasks/fio.py. This uses **fio** to generate
+ read/write I/O to an rbd image volume
+
+4. **cbt** task - qa/tasks/cbt.py. This uses the Ceph benchmark tool **cbt**
+ to run fio or radosbench to benchmark the performance of a cluster.
+
+5. **rbd bench**. Some of the standalone tests use rbd bench
+ (src/tools/rbd/action/Bench.cc) to generate small amounts of I/O
+ workload. It is also used by the **rbd_pwl_cache_recovery** task.
+
+It is hard to use these tools to get good coverage of I/Os to non-zero (and
+non-stripe aligned) offsets, or to generate a wide variety of offsets and
+lengths of I/O requests including all the boundary cases for chunks and
+stripes. There is scope to improve either rados, radosbench or rbd bench to
+generate much more interesting I/O patterns for testing erasure coding.
+
+For the optimizations described above it is essential that we have good tools
+for checking the consistency of either selected objects or all objects in an
+erasure coded pool by checking that the data and coding parities are
+coherent. There is a test tool **ceph-erasure-code-tool** which can use the
+plugins to encode and decode data provided in a set of files. However there
+does not seem to be any scripting in teuthology to perform consistency checks
+by using objectstore tool to read data and then using this tool to validate
+consistency. We will write some teuthology helpers that use
+ceph-objectstore-tool and ceph-erasure-code-tool to perform offline
+validation.
+
+We would also like an online way of performing full consistency checks, either
+for specific objects or for a whole pool. Inconveniently EC pools do not
+support class methods so it's not possible to use this as a way of
+implementing a full consistency check. We will investigate putting a flag on a
+read request, on the pool or implementing a new request type to perform a full
+consistency check on an object and look at making extensions to the rados CLI
+to be able to perform these tests. See also the discussion on deep scrub
+below.
+
+When there is more than one coding parity and there is an inconsistency
+between the data and the coding parities it is useful to try and analyze the
+cause of the inconsistency. Because the multiple coding parities are providing
+redundancy, there can be multiple ways of reconstructing each chunk and this
+can be used to detect the most like cause of the inconsistency. For example
+with a 4+2 erasure coding and a dropped write to 1st data OSD, the stripe (all
+6 OSDs) will be inconsistent, as will be any selection of 5 OSDs that includes
+the 1st data OSD, but data OSDs 2,3 and 4 and the two coding parity OSDs will
+be still be consistent. While there are many ways a stripe could get into this
+state, a tool could conclude that the most likely cause is a missed update to
+OSD 1. Ceph does not have a tool to perform this type of analysis, but it
+should be easy to extend ceph-erasure-code-tool.
+
+Teuthology seems to have adequate tools for taking OSDs offline and bringing
+them back online again. There are a few tools for injecting read I/O errors
+(without taking an OSD offline) but there is scope to improve these
+(e.g. ability to specify a particular offset in an object that will fail a
+read, more controls over setting and deleting error inject sites).
+
+The general philosophy of teuthology seems to be to randomly inject faults and
+simply through brute force get sufficient coverage of all the error
+paths. This is a good approach for CI testing, however when EC code paths
+become complex and require multiple errors to occur with precise timings to
+cause a particular code path to execute it becomes hard to get coverage
+without running the tests for a very long time. There are some standalone
+tests for EC which do test some of the multiple failure paths, but these tests
+perform very limited amounts of I/O and don't inject failures while there are
+I/Os in flight so miss some of the interesting scenarios.
+
+To deal with these more complex error paths we propose developing a new type
+of thrasher for erasure coding that injects a sequence of errors and makes use
+of debug hooks to capture and delay I/O requests at particular points to
+ensure an error inject hits a particular timing window. To do this we will
+extend the tell osd command to include extra interfaces to inject errors and
+capture and stall I/Os at specific points.
+
+Some parts of erasure coding such as the plugins are stand alone bits of code
+which can be tested with unit tests. There are already some unit tests and
+performance benchmark tools for erasure coding, we will look to extend these
+to get further coverage of code that can be run stand alone.
+
+I/O path changes
+----------------
+
+Avoid unnecessary reads and writes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current code reads too much data for read and overwrite I/Os. For
+overwrites it will also rewrite unmodified data. This occurs because reads and
+overwrites are rounded up to full-stripe operations. This isn’t a problem when
+data is mainly being accessed sequentially but is very wasteful for random I/O
+operations. The code can be changed to only read/write necessary shards. To
+allow the code to efficiently support larger chunk sizes I/Os should be
+rounded to page size I/Os instead of chunk sized I/Os.
+
+The first simple set of optimizations eliminates unnecessary reads and
+unnecessary writes of data, but retains writes of metadata on all shards. This
+avoids breaking the current design which depends on all shards receiving a
+metadata update for every transaction. When changes to the metadata handling
+are completed (see below) then it will be possible to make further
+optimizations to reduce the number of metadata updates for additional savings.
+
+Parity-delta-write
+^^^^^^^^^^^^^^^^^^
+
+The current code implements overwrites by performing a full-stripe read,
+merging the overwritten data, calculating new coding parities and performing a
+full-stripe write. Reading and writing every shard is expensive, there are a
+number of optimizations that can be applied to speed this up. For a K+M
+configuration where M is small, it is often less work to perform a
+parity-delta-write. This is implemented by reading the old data that is about
+to be overwritten and XORing it with the new data to create a delta. The
+coding parities can then be read, updated to apply the delta and
+re-written. With M=2 (RAID6) this can result in just 3 read and 3 writes to
+perform an overwrite of less than one chunk.
+
+Note that where a large fraction of the data in the stripe is being updated,
+this technique can result in more work than performing a partial overwrite,
+however if both update techniques are supported it is fairly easy to calculate
+for a given I/O offset and length which is the optimal technique to use.
+
+Write I/Os submitted to the Primary OSD will perform this calculation to
+decide whether to use a full-stripe update or a parity-delta-write. Note that
+if read failures are encountered while performing a parity-delta-write and it
+is necessary to reconstruct data or a coding parity then it will be more
+efficient to switch to performing a full-stripe read, merge and write.
+
+Not all erasure codings and erasure coding libraries support the capability of
+performing delta updates, however those implemented using XOR and/or GF
+arithmetic should. We have checked jerasure and isa-l and confirmed that they
+support this feature, although the necessary APIs are not currently exposed by
+the plugins. For some erasure codes such as clay and lrc it may be possible to
+apply delta updates, but the delta may need to be applied in so many places
+that this makes it a worthless optimization. This proposal suggests that
+parity-delta-write optimizations are initially implemented only for the most
+commonly used erasure codings. Erasure code plugins will provide a new flag
+indicating whether they support the new interfaces needed to perform delta
+updates.
+
+Direct reads
+^^^^^^^^^^^^
+
+Read I/Os are currently directed to the primary OSD which then issues reads to
+other shards. To reduce I/O latency and network bandwidth it would be better
+if clients could issue direct read requests to the OSD storing the data,
+rather than via the primary. There are a few error scenarios where the client
+may still need to fallback to submitting reads to the primary, a secondary OSD
+will have the option of failing a direct read with -EAGAIN to request the
+client retries the request to the primary OSD.
+
+Direct reads will always be for <= one chunk. For reads of more than one chunk
+the client can issue direct reads to multiple OSDs, however these will no
+longer guaranteed to be atomic because an update (write) may be applied in
+between the separate read requests. If a client needs atomicity guarantees
+they will need to continue to send the read to the primary.
+
+Direct reads will be failed with EAGAIN where a reconstruct and decode
+operation is required to return the data. This means only reads to primary OSD
+will need to handle the reconstruct code path. When an OSD is backfilling we
+don't want the client to have large quantities of I/O failed with EAGAIN,
+therefore we will make the client detect this situation and avoid issuing
+direct I/Os to a backfilling OSD.
+
+For backwards compatibility, for client requests that cannot cope with the
+reduced guarantees of a direct read, and for scenarios where the direct read
+would be to an OSD that is absent or backfilling, reads directed to the
+primary OSD will still be supported.
+
+Direct writes
+^^^^^^^^^^^^^
+
+Write I/Os are currently directed to the primary OSD which then updates the
+other shards. To reduce latency and network bandwidth it would be better if
+clients could direct small overwrites requests directly to the OSD storing the
+data, rather than via the primary. For larger write I/Os and for error
+scenarios and abnormal cases clients will continue to submit write I/Os to the
+primary OSD.
+
+Direct writes will always be for <= one chunk and will use the
+parity-delta-write technique to perform the update. For medium sized writes a
+client may issue direct writes to multiple OSDs, but such updates will no
+longer be guaranteed to be atomic. If a client requires atomicity for a larger
+write they will need to continue to send it to the primary.
+
+For backwards compatibility, and for scenarios where the direct write would be
+to an OSD that is absent, writes directed to the primary OSD will still be
+supported.
+
+I/O serialization, recovery/backfill and other error scenarios
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+Direct writes look fairly simple until you start considering all the abnormal
+scenarios. The current implementation of processing all writes on the Primary
+OSD means that there is one central point of control for the stripe that can
+manage things like the ordering of multiple inflight I/Os to the same stripe,
+ensuring that recovery/backfill for an object has been completed before it is
+accessed and assigning the object version number and modification time.
+
+With direct I/Os these become distributed problems. Our approach is to send a
+control path message to the Primary OSD and let it continue to be the central
+point of control. The Primary OSD will issue a reply when the OSD can start
+the direct write and will be informed with another message when the I/O has
+completed. See section below on metadata updates for more details.
+
+Stripe cache
+^^^^^^^^^^^^
+
+Erasure code pools maintain a stripe cache which stores shard data while
+updates are in progress. This is required to allow writes and reads to the
+same stripe to be processed in parallel. For small sequential write workloads
+and for extreme hot spots (e.g. where the same block is repeatedly re-written
+for some kind of crude checkpointing mechanism) there would be a benefit in
+keeping the stripe cache slightly longer than the duration of the I/O. In
+particularly the coding parities are typically read and written for every
+update to a stripe. There is obviously a balancing act to achieve between
+keeping the cache long enough that it reduces the overheads for future I/Os
+versus the memory overheads of storing this data. A small (MiB as opposed to
+GiB sized cache) should be sufficient for most workloads. The stripe cache can
+also help reduce latency for direct write I/Os by allowing prefetch I/Os to
+read old data and coding parities ready for later parts of the write operation
+without requiring more complex interlocks.
+
+The stripe cache is less important when the default chunk size is small
+(e.g. 4K), because even with small write I/O requests there will not be many
+sequential updates to fill a stripe. With a larger chunk size (e.g. 64K) the
+benefits of a good stripe cache become more significant because the stripe
+size will be 100’s KiB to small number of MiB’s and hence it becomes much more
+likely that a sequential workload will issue many I/Os to the same stripe.
+
+Automatically choose chunk size
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default chunk size of 4K is good for small objects because the data and
+coding parities are rounded up to whole chunks and because if an object has
+less than one data stripe of data then the capacity overheads for the coding
+parities are higher (e.g. a 4K object in a 10+2 erasure coded pool has 4K of
+data and 8K of coding parity, so there is a 200% overhead). However the
+optimizations above all provide much bigger savings if the typical random
+access I/O only reads or writes a single shard. This means that so long as
+objects are big enough that a larger chunk size such as 64K would be better.
+
+Whilst the user can try and predict what their typically object size will be
+and choose an appropriate chunk size, it would be better if the code could
+automatically select a small chunk size for small objects and a larger chunk
+size for larger objects. There will always be scenarios where an object grows
+(or is truncated) and the chosen chunk size becomes inappropriate, however
+reading and re-writing the object with a new chunk size when this happens
+won’t have that much performance impact. This also means that the chunk size
+can be deduced from the object size in object_info_t which is read before the
+objects data is read/modified. Clients already provide a hint as to the object
+size when creating the object so this could be used to select a chunk size to
+reduce the likelihood of having to re-stripe an object
+
+The thought is to support a new chunk size of auto/variable to enable this
+feature, it will only be applicable for newly created pools, there will be no
+way to migrate an existing pool.
+
+Deep scrub support
+^^^^^^^^^^^^^^^^^^
+
+EC Pools with overwrite do not check CRCs because it is too costly to update
+the CRC for the object on every overwrite, instead the code relies on
+Bluestore to maintain and check CRCs. When an EC pool is operating with
+overwrite disabled a CRC is kept for each shard, because it is possible to
+update CRCs as the object is appended to just by calculating a CRC for the new
+data being appended and then doing a simple (quick) calculation to combine the
+old and new CRC together.
+
+In dev/osd_internals/erasure_coding/proposals.rst it discusses the possibility
+of keeping CRCs at a finer granularity (for example per chunk), storing these
+either as an xattr or an omap (omap is more suitable as large objects could
+end up with a lot of CRC metadata) and updating these CRCs when data is
+overwritten (the update would need to perform a read-modify-write at the same
+granularity as the CRC). These finer granularity CRCs can then easily be
+combined to produce a CRC for the whole shard or even the whole erasure coded
+object.
+
+This proposal suggests going in the opposite direction - EC overwrite pools
+have survived without CRCs and relied on Bluestore up until now, so why is
+this feature needed? The current code doesn’t check CRCs if overwrite is
+enabled, but sadly still calculates and updates a CRC in the hinfo xattr, even
+if performing overwrites which mean that the calculated value will be
+garbage. This means we pay all the overheads of calculating the CRC and get no
+benefits.
+
+The code can easily be fixed so that CRCs are calculated and maintained when
+objects are written sequentially, but as soon as the first overwrite to an
+object occurs the hinfo xattr will be discarded and CRCs will no longer be
+calculated or checked. This will improve performance when objects are
+overwritten, and will improve data integrity in cases where they are not.
+
+While the thought is to abandon EC storing CRCs in objects being overwritten,
+there is an improvement that can be made to deep scrub. Currently deep scrub
+of an EC with overwrite pool just checks that every shard can read the object,
+there is no checking to verify that the copies on the shards are consistent. A
+full consistency check would require large data transfers between the shards
+so that the coding parities could be recalculated and compared with the stored
+versions, in most cases this would be unacceptably slow. However for many
+erasure codes (including the default ones used by Ceph) if the contents of a
+chunk are XOR’d together to produce a longitudinal summary value, then an
+encoding of the longitudinal summary values of each data shard should produce
+the same longitudinal summary values as are stored by the coding parity
+shards. This comparison is less expensive than the CRC checks performed by
+replication pools. There is a risk that by XORing the contents of a chunk
+together that a set of corruptions cancel each other out, but this level of
+check is better than no check and will be very successful at detecting a
+dropped write which will be the most common type of corruption.
+
+Metadata changes
+----------------
+
+What metadata do we need to consider?
+
+1. object_info_t. Every Ceph object has some metadata stored in the
+ object_info_t data structure. Some of these fields (e.g. object length) are
+ not updated frequently and we can simply avoid performing partial writes
+ optimizations when these fields need updating. The more problematic fields
+ are the version numbers and the last modification time which are updated on
+ every write. Version numbers of objects are compared to version numbers in
+ PG log entries for peering/recovery and with version numbers on other
+ shards for backfill. Version numbers and modification times can be read by
+ clients.
+
+2. PG log entries. The PG log is used to track inflight transactions and to
+ allow incomplete transactions to be rolled forward/backwards after an
+ outage/network glitch. The PG log is also used to detect and resolve
+ duplicate requests (e.g. resent due to network glitch) from
+ clients. Peering currently assumes that every shard has a copy of the log
+ and that this is updated for every transaction.
+
+3. PG stats entries and other PG metadata. There is other PG metadata (PG
+ stats is the simplest example) that gets updated on every
+ transaction. Currently all OSDs retain a cached and a persistent copy of
+ this metadata.
+
+How many copies of metadata are required?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The current implementation keeps K+M replicated copies of metadata, one copy
+on each shard. The minimum number of copies that need to be kept to support up
+to M failures is M+1. In theory metadata could be erasure encoded, however
+given that it is small it is probably not worth the effort. One advantage of
+keeping K+M replicated copies of the metadata is that any fully in sync shard
+can read the local copy of metadata, avoiding the need for inter-OSD messages
+and asynchronous code paths. Specifically this means that any OSD not
+performing backfill can become the primary and can access metadata such as
+object_info_t locally.
+
+M+1 arbitrarily distributed copies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A partial write to one data shard will always involve updates to the data
+shard and all M coding parity shards, therefore for optimal performance it
+would be ideal if the same M+1 shards are updated to track the associated
+metadata update. This means that for small random writes that a different M+1
+shards would get updated for each write. The drawback of this approach is that
+you might need to read K shards to find the most up to date version of the
+metadata.
+
+In this design no shard will have an up to date copy of the metadata for every
+object. This means that whatever shard is picked to be the acting primary that
+it may not have all the metadata available locally and may need to send
+messages to other OSDs to read it. This would add significant extra complexity
+to the PG code and cause divergence between Erasure coded pools and Replicated
+pools. For these reasons we discount this design option.
+
+M+1 copies on known shards
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The next best performance can be achieved by always applying metadata updates
+to the same M+1 shards, for example choosing the 1st data shard and all M
+coding parity shards. Coding parity shards will get updated by every partial
+write so this will result in zero or one extra shard being updated. With this
+approach only 1 shard needs to be read to find the most up to date version of
+the metadata.
+
+We can restrict the acting primary to be one of the M+1 shards, which means
+that once any incomplete updates in the log have been resolved that the
+primary will have an up to date local copy of all the metadata, this means
+that much more of the PG code can be kept unchanged.
+
+Partial Writes and the PG log
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Peering currently assumes that every shard has a copy of the log, however
+because of inflight updates and small term absences it is possible that some
+shards are missing some of the log entries. The job of peering is to combine
+the logs from the set of present shards to form a definitive log of
+transactions that have been committed by all the shards. Any discrepancies
+between a shards log and the definitive log are then resolved, typically by
+rolling backwards transactions (using information held in the log entry) so
+that all the shards are in a consistent state.
+
+To support partial writes the log entry needs to be modified to include the
+set of shards that are being updated. Peering needs to be modified to consider
+a log entry as missing from a shard only if a copy of the log entry on another
+shard indicates that this shard was meant to be updated.
+
+The logs are not infinite in size, and old log entries where it is known that
+the update has been successfully committed on all affected shards are
+trimmed. Log entries are first condensed to a pg_log_dup_t entry which can no
+longer assist in rollback of a transaction but can still be used to detect
+duplicated client requests, and then later completely discarded. Log trimming
+is performed at the same time as adding a new log entry, typically when a
+future write updates the log. With partial writes log trimming will only occur
+on shards that receive updates, which means that some shards may have stale
+log entries that should have been discarded.
+
+TBD: I think the code can already cope with discrepancies in log trimming
+between the shards. Clearly an in flight trim operation may not have completed
+on every shard so small discrepancies can be dealt with, but I think an absent
+OSD can cause larger discrepancies. I believe that this is resolved during
+Peering, with each OSD keeping a record of what the oldest log entry should be
+and this gets shared between OSDs so that they can work out stale log entries
+that were trimmed in absentia. Hopefully this means that only sending log
+trimming updates to shards that are creating new log entries will work without
+code changes.
+
+Backfill
+^^^^^^^^
+
+Backfill is used to correct inconsistencies between OSDs that occur when an
+OSD is absent for a longer period of time and the PG log entries have been
+trimmed. Backfill works by comparing object versions between shards. If some
+shards have out of date versions of an object then a reconstruct is performed
+by the backfill process to update the shard. If the version numbers on objects
+are not updated on all shards then this will break the backfill process and
+cause a huge amount of unnecessary reconstruct work. This is unacceptable, in
+particular for the scenario where an OSD is just absent for maintenance for a
+relatively short time with noout set. The requirement is to be able to
+minimize the amount of reconstruct work needed to complete a backfill.
+
+In dev/osd_internals/erasure_coding/proposals.rst it discusses the idea of
+each shard storing a vector of version numbers that records the most recent
+update that the pair <this shard, other shard> both should have participated
+in. By collecting this information from at least M shards it is possible to
+work out what the expected minimum version number should be for an object on a
+shard and hence deduce whether a backfill is required to update the
+object. The drawback of this approach is that backfill will need to scan M
+shards to collect this information, compared with the current implementation
+that only scans the primary and shard(s) being backfilled.
+
+With the additional constraint that a known M+1 shards will always be updated
+and that the (acting) primary will be one of these shards, it will be possible
+to determine whether a backfill is required just by examining the vector on
+the primary and the object version on the shard being backfilled. If the
+backfill target is one of the M+1 shards the existing version number
+comparison is sufficient, if it is another shard then the version in the
+vector on the primary needs to be compared with the version on the backfill
+target. This means that backfill does not have to scan any more shards than it
+currently does, however the scan of the primary does need to read the vector
+and if there are multiple backfill targets then it may need to store multiple
+entries of the vector per object increasing memory usage during the backfill.
+
+There is only a requirement to keep the vector on the M+1 shards, and the
+vector only needs K-1 entires because we only need to track version number
+differences between any of the M+1 shards (which should have the same version)
+and each of the K-1 shards (which can have a stale version number). This will
+slightly reduce the amount of extra metadata required. The vector of version
+numbers could be stored in the object_info_t structure or stored as a separate
+attribute.
+
+Our preference is to store the vector in the object_info_t structure because
+typically both are accessed together, and because this makes it easier to
+cache both in the same object cache. We will keep metadata and memory
+overheads low by only storing the vector when it is needed.
+
+Care is required to ensure that existing clusters can be upgraded. The absence
+of the vector of version numbers implies that an object has never had a
+partial update and therefore all shards are expected to have the same version
+number for the object and the existing backfill algorithm can be used.
+
+Code references
+"""""""""""""""
+
+PrimaryLogPG::scan_range - this function creates a map of objects and their
+version numbers, on the primary it tries to get this information from the
+object cache, otherwise it reads to OI attribute. This will need changes to
+deal with the vectors. To conserve memory it will need to be provided with the
+set of backfill targets so it can select which part of the vector to keep.
+
+PrimaryLogPG::recover_backfill - this function call scan_range for the local
+(primary) and sends MOSDPGScan to the backfill targets to get them to perform
+the same scan. Once it has collected all the version numbers it compares the
+primary and backfill targets to work out which objects need to be
+recovered. This will also need changes to deal with the vectors when comparing
+version numbers.
+
+PGBackend::run_recovery_op - recovers a single object. For an EC pool this
+involves reconstructing the data for the shards that need backfilling (read
+other shards and use decode to recover). This code shouldn't need any changes.
+
+Version number and last modification time for clients
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Clients can read the object version number and set expectations about what the
+minimum version number is when making updates. Clients can also read the last
+modification time. There are use cases where it is important that these values
+can be read and give consistent results, but there is also a large number of
+scenarios where this information is not required.
+
+If the object version number is only being updated on a known M+1 shards for
+partial writes, then where this information is required it will need to
+involve a metadata access to one of those shards. We have arranged for the
+primary to be one of the M+1 shards so I/Os submitted to the primary will
+always have access to the up to date information.
+
+Direct write I/Os need to update the M+1 shards, so it is not difficult to
+also return this information to the client when completing the I/O.
+
+Direct read I/Os are the problem case, these will only access the local shard
+and will not necessarily have access to the latest version and modification
+time. For simplicity we will require clients that require this information to
+send requests to the primary rather than using the direct I/O
+optimization. Where a client does not need this information they can use the
+direct I/O optimizations.
+
+The direct read I/O optimization will still return a (potentially stale)
+object version number. This may still be of use to clients to help understand
+the ordering of I/Os to a chunk.
+
+Direct Write with Metadata updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here's the full picture of what a direct write performing a parity-delta-write
+looks like with all the control messages:
+
+.. ditaa::
+
+ RADOS Client
+
+ | ^
+ | |
+ 1 28
+ +-----+ | |
+ | |<------27-------+--+
+ | | | |
+ | | +-------------|->|
+ | | | | |
+ | |<-|----2--------+ |<--------------------------------------------+
+ | Seq | | | |<----------------------------+ |
+ | | | +----3---+ | | |
+ | | | | +--|-----------------------5-----|---+ |
+ | | | | +--|-------4---------+ | | |
+ | +--|-10-|------->| | | | | |
+ | | | | +---+ | | | | |
+ | | | | | | | | | | |
+ | | | | v | | | | | |
+ +----++ | | +---+ | | | | | |
+ ^ | | | |XOR+-|--|----------15-----|-----------|---|-----+ |
+ | | | | |13 +-|--|-------14--------|-----+ | | | |
+ | | | | +---+ | | | | | | | |
+ | | | | ^ | | | v | | v |
+ | | | | | | | | +------+ | | +------+ |
+ 6 11 | | | | | | | XOR | | | | GF | |
+ | | | | | | | | | 18 | | | | 21 | |
+ | | | | 12 16 | | +----+-+ | | +----+-+ |
+ | | | | | | | | ^ | | | ^ | |
+ | | | | | | | | | | | | | | |
+ | | | | | | | | 17 19 | | 20 22 |
+ | | | | | | | | | | | | | | |
+ | | | | | v | | | v | | | v |
+ | | | | +-+----+ | | +-+----+ | | +-+----+ |
+ | | | +->|Extent| | +->|Extent| | +->|Extent| |
+ | | 23 |Cache | 24 |Cache | 25 |Cache | 26
+ | | | +----+-+ | +----+-+ | +----+-+ |
+ | | | ^ | | ^ | | ^ | |
+ | | | | | | | | | | | |
+ | +---+ 7 +---+ 8 +---+ 9 +---+
+ | | | | | | | |
+ | v | v | v | v
+ .-----. .-----. .-----. .-----. .-----. .-----.
+ ( ) ( ) ( ) ( ) ( ) ( )
+ |`-----'| |`-----'| |`-----'| |`-----'| |`-----'| |`-----'|
+ | | | | | | | | | | | |
+ | | | | | | | | | | | |
+ ( ) ( ) ( ) ( ) ( ) ( )
+ `-----' `-----' `-----' `-----' `-----' `-----'
+ Primary OSD 2 OSD 3 OSD 4 OSD P OSD Q
+ OSD
+
+ * Xattr * No Xattr * Xattr * Xattr
+ * OI * Stale OI * OI * OI
+ * PG log * Partial PG log * PG log * PG log
+ * PG stats * No PG stats * PG stats * PG stats
+
+Note: Only the primary OSD and parity coding OSDs (the M+1 shards) have Xattr,
+up to date object info, PG log and PG stats. Only one of these OSDs is
+permitted to become the (acting) primary. The other data OSDs 2,3 and 4 (the
+K-1 shards) do not have Xattrs or PG stats, may have state object info and
+only have PG log entries for their own updates. OSDs 2,3 and 4 may have stale
+OI with an old version number. The other OSDs have the latest OI and a vector
+with the expected version numbers for OSDs 2,3 and 4.
+
+1. Data message with Write I/O from client (MOSDOp)
+2. Control message to Primary with Xattr (new msg MOSDEcSubOpSequence)
+
+Note: the primary needs to be told about any xattr update so it can update its
+copy, but the main purpose of this message is to allow the primary to sequence
+the write I/O. The reply message at step 10 is what allows the write to start
+and provides the PG stats and new object info including the new version
+number. If necessary the primary can delay this to ensure that
+recovery/backfill of the object is completed first and deal with overlapping
+writes. Data may be read (prefetched) before the reply, but obviously no
+transactions can start.
+
+3. Prefetch request to local extent cache
+4. Control message to P to prefetch to extent cache (new msg
+ MOSDEcSubOpPrefetch equivalent of MOSDEcSubOpRead)
+5. Control message to Q to prefetch to extent cache (new msg
+ MOSDEcSubOpPrefetch equivalent of MOSDEcSubOpRead)
+6. Primary reads object info
+7. Prefetch old data
+8. Prefetch old P
+9. Prefetch old Q
+
+Note: The objective of these prefetches is to get the old data, P and Q reads
+started as quickly as possible to reduce the latency of the whole I/O. There
+may be error scenarios where the extent cache is not able to retain this and
+it will need to be re-read. This includes the rare/pathological scenarios
+where there is a mixture of writes sent to the primary and writes sent
+directly to the data OSD for the same object.
+
+10. Control message to data OSD with new object info + PG stats (new msg
+ MOSDEcSubOpSequenceReply)
+11. Transaction to update object info + PG log + PG stats
+12. Fetch old data (hopefully cached)
+
+Note: For best performance we want to pipeline writes to the same stripe. The
+primary assigns the version number to each write and consequently defines the
+order in which writes should be processed. It is important that the data shard
+and the coding parity shards apply overlapping writes in the same order. The
+primary knows what set of writes are in flight so can detect this situation
+and indicate in its reply message at step 10 that an update must wait until an
+earlier update has been applied. This information needs to be forwarded to the
+coding parities (steps 14 and 15) so they can also ensure updates are applied
+in the same order.
+
+13. XOR new and old data to create delta
+14. Data message to P with delta + Xattr + object info + PG log + PG stats
+ (new msg MOSDEcSubOpDelta equivalent of MOSDEcSubOpWrite)
+15. Data message to Q with delta + Xattr + object info + PG log + PG stats
+ (new msg MOSDEcSubOpDelta equivalent of MOSDEcSubOpWrite)
+16. Transaction to update data + object info + PG log
+17. Fetch old P (hopefully cached)
+18. XOR delta and old P to create new P
+19. Transaction to update P + Xattr + object info + PG log + PG stats
+20. Fetch old Q (hopefully cached)
+21. XOR delta and old Q to create new Q
+22. Transaction to update Q + Xattr + object info + PG log + PG stats
+23. Control message to data OSD for commit (new msg MOSDEcSubOpDeltaReply
+ equivalent of MOSDEcSubOpWriteReply)
+24. Local commit notification
+25. Control message to data OSD for commit (new msg MOSDEcSubOpDeltaReply
+ equivalent of MOSDEcSubOpWriteReply)
+26. Control message to data OSD for commit (new msg MOSDEcSubOpDeltaReply
+ equivalent of MOSDEcSubOpWriteReply)
+27. Control message to Primary to signal end of write (variant of new msg
+ MOSDEcSubOpSequence)
+28. Control message reply to client (MOSDOpReply)
+
+Upgrade and backwards compatibility
+-----------------------------------
+
+A few of the optimizations can be made just by changing code on the primary
+OSD with no backwards compatibility concerns regarding clients or the other
+OSDs. These optimizations will be enabled as soon as the primary OSD upgrades
+and will replace the existing code paths.
+
+The remainder of the changes will be new I/O code paths that will exist
+alongside the existing code paths.
+
+Similar to EC Overwrites many of the changes will need to ensure that all OSDs
+are running new code and that the EC plugins support new interfaces required
+for parity-delta-writes. A new pool level flag will be required to enforce
+this. It will be possible to enable this flag (and hence enable the new
+performance optimizations) after upgrading an existing cluster. Once set it
+will not be possible to add down level OSDs to the pool. It will not be
+possible to turn this flag off other than by deleting the pool. Downgrade is
+not supported because:
+
+1. It is not trivial to quiesce all I/O to a pool to ensure that none of the
+ new I/O code paths are in use when the flag is cleared.
+
+2. The PG log format for new I/Os will not be understood by down level
+ OSDs. It would be necessary to ensure the log has been trimmed of all new
+ format entries before clearing the flag to ensure that down level OSDs will
+ be able to interpret the log.
+
+3. Additional xattr data will be stored by the new I/O code paths and used by
+ backfill. Down level code will not understand how to backfill a pool that
+ has been running the new I/O paths and will get confused by the
+ inconsistent object version numbers. While it is theoretically possible to
+ disable partial updates and then scan and update all the metadata to return
+ the pool to a state where a downgrade is possible, we have no intention of
+ writing this code.
+
+The direct I/O changes will additionally require clients to be running new
+code. These will require that the pool has the new flag set and that a new
+client is used. Old clients can use pools with the new flag set, just without
+the direct I/O optimization.
+
+Not under consideration
+-----------------------
+
+There is a list of enhancements discussed in
+doc/dev/osd_internals/erasure_coding/proposals.rst, the following are not
+under consideration:
+
+1. RADOS Client Acknowledgement Generation optimization
+
+When updating K+M shards in an erasure coded pool, in theory you don’t have to
+wait for all the updates to complete before completing the update to the
+client, because so long as K updates have completed any viable subset of
+shards should be able to roll forward the update.
+
+For partial writes where only M+1 shards are updated this optimization does
+not apply as all M+1 updates need to complete before the update is completed
+to the client.
+
+This optimization would require changes to the peering code to work out
+whether partially completed updates need to be rolled forwards or
+backwards. To roll an update forwards it would be simplest to mark the object
+as missing and use the recovery path to reconstruct and push the update to
+OSDs that are behind.
+
+2. Avoid sending read request to local OSD via Messenger
+
+The EC backend code has an optimization for writes to the local OSD which
+avoids sending a message and reply via messenger. The equivalent optimization
+could be made for reads as well, although a bit more care is required because
+the read is synchronous and will block the thread waiting for the I/O to
+complete.
+
+Pull request https://github.com/ceph/ceph/pull/57237 is making this
+optimization
+
+Stories
+=======
+
+This is our high level breakdown of the work. Our intention is to deliver this
+work as a series of PRs. The stories are roughly in the order we plan to
+develop. Each story is at least one PR, where possible they will be broken up
+further. The earlier stories can be implemented as stand alone pieces of work
+and will not introduce upgrade/backwards compatibility issues. The later
+stories will start breaking backwards compatibility, here we plan to add a new
+flag to the pool to enable these new features. Initially this will be an
+experimental flag while the later stories are developed.
+
+Test tools - enhanced I/O generator for testing erasure coding
+--------------------------------------------------------------
+
+* Extend rados bench to be able to generate more interesting patterns of I/O
+ for erasure coding, in particular reading and writing at different offsets
+ and for different lengths and making sure we get good coverage of boundary
+ conditions such as the sub-chunk size, chunk size and stripe size
+* Improve data integrity checking by using a seed to generate data patterns
+ and remembering which seed is used for each block that is written so that
+ data can later be validated
+
+Test tools - offline consistency checking tool
+----------------------------------------------
+
+* Test tools for performing offline consistency checks combining use of
+ objectstore_tool with ceph-erasure-code-tool
+* Enhance some of the teuthology standalone erasure code checks to use this
+ tool
+
+Test tools - online consistency checking tool
+---------------------------------------------
+
+* New CLI to be able to perform online consistency checking for an object or a
+ range of objects that reads all the data and coding parity shards and
+ re-encodes the data to validate the coding parities
+
+Switch for JErasure to ISA-L
+----------------------------
+
+The JErasure library has not been updated since 2014, the ISA-L library is
+maintained and exploits newer instructions sets (e.g. AVX512, AVX2) which
+provides faster encoding/decoding
+
+* Change defaults to ISA-L in upstream ceph
+* Benchmark Jerasure and ISA-L
+* Refactor Ceph isa_encode region_xor() to use AVX when M=1
+* Documentation updates
+* Present results at performance weekly
+
+Sub Stripe Reads
+----------------
+
+Ceph currently reads an integer number of stripes and discards unneeded
+data. In particular for small random reads it will be more efficient to just
+read the required data
+
+* Help finish Pull Request https://github.com/ceph/ceph/pull/55196 if not
+ already complete
+* Further changes to issue sub-chunk reads rather than full-chunk reads
+
+Simple Optimizations to Overwrite
+---------------------------------
+
+Ceph overwrites currently read an integer number of stripes, merge the new
+data and write an integer number of stripes. This story makes simple
+improvements by making the same optimizations as for sub stripe reads and for
+small (sub-chunk) updates reducing the amount of data being read/written to
+each shard.
+
+* Only read chunks that are not being fully overwritten (code currently reads
+ whole stripe and then merges new data)
+* Perform sub-chunk reads for sub-chunk updates
+* Perform sub-chunk writes for sub-chunk updates
+
+Eliminate unnecessary chunk writes but keep metadata transactions
+-----------------------------------------------------------------
+
+This story avoids re-writing data that has not been modified. A transaction is
+still applied to every OSD to update object metadata, the PG log and PG stats.
+
+* Continue to create transactions for all chunks but without the new write data
+* Add sub-chunk writes to transactions where data is being modified
+
+Avoid zero padding objects to a full stripe
+-------------------------------------------
+
+Objects are rounded up to an integer number of stripes by adding zero
+padding. These buffers of zeros are then sent in messages to other OSDs and
+written to the OS consuming storage. This story make optimizations to remove
+the need for this padding
+
+* Modifications to reconstruct reads to avoid reading zero-padding at the end
+ of an object - just fill the read buffer with zeros instead
+* Avoid transfers/writes of buffers of zero padding. Still send transactions
+ to all shards and create the object, just don't populate it with zeros
+* Modifications to encode/decode functions to avoid having to pass in buffers
+ of zeros when objects are padded
+
+Erasure coding plugin changes to support distributed partial writes
+-------------------------------------------------------------------
+
+This is preparatory work for future stories, it adds new APIs to the erasure
+code plugins.
+
+* Add a new interface to create a delta by XORing old and new data together
+ and implement this for the ISA-L and JErasure plugins
+* Add a new interface to apply a delta to one coding parity by using XOR/GF
+ and implement this for the ISA-L and JErasure plugins
+* Add a new interface which reports which erasure codes support this feature
+ (ISA-L and JErasure will support it, others will not)
+
+Erasure coding interface to allow RADOS clients to direct I/Os to OSD storing the data
+--------------------------------------------------------------------------------------
+
+This is preparatory work for future stories, its adds a new API for clients
+
+* New interface to convert the pair (pg, offset) to {OSD, remaining chunk
+ length}
+
+We do not want clients to have to dynamically link to the erasure code plugins
+so this code will need to be part of librados. However this interface needs to
+understand how erasure codes distribute data and coding chunks to be able to
+perform this translation.
+
+We will only support ISA-L and JErasure plugins where there is a trivial
+striping of data chunks to OSDs.
+
+Changes to object_info_t
+------------------------
+
+This is preparatory work for future stories.
+
+This adds the vector of version numbers to object_info_t which will be used
+for partial updates. For replicated pools and for erasure coded objects that
+are not overwritten we will avoid storing extra data in object_info_t.
+
+Changes to PGLog and Peering to support updating a subset of OSDs
+-----------------------------------------------------------------
+
+This is preparatory work for future stories.
+
+* Modify the PG log entry to store a record of which OSDs are being updated
+* Modify peering to use this extra data to work out OSDs that are missing
+ updates
+
+Change to selection of (acting) primary
+---------------------------------------
+
+This is preparatory work for future stories.
+
+Constrain the choice of primary to be the first data OSD or one of the erasure
+coding parities. If none of these OSDs are available and up to date then the
+pool must be offline.
+
+Implement parity-delta-write with all computation on the primary
+----------------------------------------------------------------
+
+* Calculate whether its more efficient for an update to perform a full stripe
+ overwrite or a parity-delta-write
+* Implement new code paths to perform the parity-delta-write
+* Test tool enhancements. We want to make sure that both parity-delta-write
+ and full-stripe write are tested. We will add a new conf file option with a
+ choice of 'parity-delta', 'full-stripe', 'mixture for testing' or
+ 'automatic' and update teuthology test cases to predominately use a mixture.
+
+Upgrades and backwards compatibility
+------------------------------------
+
+* Add a new feature flag for erasure coded pools
+* All OSDs must be running new code to enable the flag on the pool
+* Clients may only issue direct I/Os if the flag is set
+* OSDs running old code may not join a pool with the flag set
+* Its not possible to turn the feature flag off (other than by deleting the
+ pool)
+
+Changes to Backfill to use the vector in object_info_t
+------------------------------------------------------
+
+This is preparatory work for future stories.
+
+* Modify the backfill process to use the vector of version numbers in
+ object_info_t so that when partial updates occur we do not backfill OSDs
+ which did not participate in the partial update.
+* When there is a single backfill target extract the appropriate version
+ number from the vector (no additional storage required)
+* When there are multiple backfill targets extract the subset of the vector
+ required by the backfill targets and select the appropriate entry when
+ comparing version numbers in PrimaryLogPG::recover_backfill
+
+Test tools - offline metadata validation tool
+---------------------------------------------
+
+* Test tools for performing offline consistency checking of metadata, in
+ particular checking the vector of version numbers in object_info_t matches
+ the versions on each OSD, but also for validating PG log entries
+
+Eliminate transactions on OSDs not updating data chunks
+-------------------------------------------------------
+
+Peering, log recovery and backfill can now all cope with partial updates using
+the vector of version numbers in object_info_t.
+
+* Modify the overwrite I/O path to not bother with metadata only transactions
+ (except to the Primary OSD)
+* Modify the update of the version numbers in object_info_t to use the vector
+ and only update entries that are receiving a transaction
+* Modify the generation of the PG log entry to record which OSDs are being
+ updated
+
+Direct reads to OSDs (single chunk only)
+----------------------------------------
+
+* Modify OSDClient to route single chunk read I/Os to the OSD storing the data
+* Modify OSD to accept reads from non-primary OSD (expand existing changes for
+ replicated pools to work with EC pools as well)
+* If necessary fail the read with EAGAIN if the OSD is unable to process the
+ read directly
+* Modify OSDClient to retry read by submitting to Primary OSD if read is
+ failed with EAGAIN
+* Test tool enhancements. We want to make sure that both direct reads and
+ reads to the primary are tested. We will add a new conf file option with a
+ choice of 'prefer direct', 'primary only' or 'mixture for testing' and
+ update teuthology test cases to predominately use a mixture.
+
+The changes will be made to the OSDC part of the RADOS client so will be
+applicable to rbd, rgw and cephfs.
+
+We will not make changes to other code that has its own version of RADOS
+client code such as krbd, although this could be done in the future.
+
+Direct reads to OSDs (multiple chunks)
+--------------------------------------
+
+* Add a new OSDC flag NONATOMIC which allows OSDC to split a read into
+ multiple requests
+* Modify OSDC to split reads spanning multiple chunks into separate requests
+ to each OSD if the NONATOMIC flag is set
+* Modifications to OSDC to coalesce results (if any sub read fails the whole
+ read needs to fail)
+* Changes to librbd client to set NONATOMIC flag for reads
+* Changes to cephfs client to set NONATOMIC flag for reads
+
+We are only changing a very limited set of clients, focusing on those that
+issue smaller reads and are latency sensitive. Future work could look at
+extending the set of clients (including krbd).
+
+Implement distributed parity-delta-write
+----------------------------------------
+
+* Implement new message MOSDEcSubOpDelta and MOSDEcSubOpDeltaReply
+* Change primary to calculate delta and send MOSDEcSubOpDelta message to
+ coding parity OSDs
+* Modify coding parity OSDs to apply the delta and send MOSDEcSubOpDeltaReply
+ message
+
+Note: This change will increase latency because the coding parity reads start
+after the old data read. Future work will fix this.
+
+Test tools - EC error injection thrasher
+----------------------------------------
+
+* Implement a new type of thrasher that specifically injects faults to stress
+ erasure coded pools
+* Take one or multiple (up to M) OSDs down, more focus on taking different
+ subsets of OSDs down to drive all the different EC recovery paths than
+ stressing out peering/recovery/backfill (the existing OSD thrasher excels at
+ this)
+* Inject read I/O failures to force reconstructs using decode for single and
+ multiple failures
+* Inject delays using osd tell type interface to make it easier to test OSD
+ down at all the interesting stages of EC I/Os
+* Inject delays using osd tell type interface to slow down an OSD transaction
+ or message to expose the less common completion orders for parallel work
+
+Implement prefetch message MOSDEcSubOpPrefetch and modify extent cache
+----------------------------------------------------------------------
+
+* Implement new message MOSDEcSubOpPrefetch
+* Change primary to issue this message to the coding parity OSDs before
+ starting read of old data
+* Change the extent cache so that each OSD caches its own data rather than
+ caching everything on the primary
+* Change coding parity OSDs to handle this message and read the old coding
+ parity into the extent cache
+* Changes to extent cache to retain the prefetched old parity until the
+ MOSDEcSubOpDelta message is received, and to discard this on error paths
+ (e.g. new OSDMap)
+
+Implement sequencing message MOSDEcSubOpSequence
+------------------------------------------------
+
+* Implement new message MODSEcSubOpSequence and MOSDEcSubOpSequenceReply
+* Modify primary code to create these messages and route them locally to
+ itself in preparation for direct writes
+
+Direct writes to OSD (single chunk only)
+----------------------------------------
+
+* Modify OSDC to route single chunk write I/Os to the OSD storing the data
+* Changes to issue MOSDEcSubOpSequence and MOSDEcSubOpSequenceReply between
+ data OSD and primary OSD
+
+Direct writes to OSD (multiple chunks)
+--------------------------------------
+
+* Modifications to OSDC to split multiple chunk writes into separate requests
+ if NONATOMIC flag is set
+* Further changes to coalescing completions (in particular reporting version
+ number correctly)
+* Changes to librbd client to set NONATOMIC flag for reads
+* Changes to cephfs client to set NONATOMIC flag for reads
+
+We are only changing a very limited set of clients, focusing on those that
+issue smaller writes and are latency sensitive. Future work could look at
+extending the set of clients.
+
+Deep scrub / CRC
+----------------
+
+* Disable CRC generation in the EC code for overwrites, delete hinfo Xattr
+ when first overwrite occurs
+* For objects in pool with new feature flag set that have not been overwritten
+ check CRC, even if pool overwrite flag is set. The presence/absence of hinfo
+ can be used to determine if the object has been overwritten
+* For deep scrub requests XOR the contents of the shard to create a
+ longitudinal check (8 bytes wide?)
+* Return the longitudinal check in the scrub reply message, have the primary
+ encode the set of longitudinal replies to check for inconsistencies
+
+Variable chunk size erasure coding
+----------------------------------
+
+* Implement new pool option for automatic/variable chunk size
+* When object size is small use a small chunk size (4K) when the pool is using
+ the new option
+* When object size is large use a large chunk size (64K or 256K?)
+* Convert the chunk size by reading and re-writing the whole object when a
+ small object grows (append)
+* Convert the chunk size by reading and re-writing the whole object when a
+ large object shrinks (truncate)
+* Use the object size hint to avoid creating small objects and then almost
+ immediately converting them to a larger chunk size
+
+CLAY Erasure Codes
+------------------
+
+In theory CLAY erasure codes should be good for K+M erasure codes with larger
+values of M, in particular when these erasure codes are used with multiple
+OSDs in the same failure domain (e.g. an 8+6 erasure code with 5 servers each
+with 4 OSDs). We would like to improve the test coverage for CLAY and perform
+some more benchmarking to collect data to help substantiate when people should
+consider using CLAY.
+
+* Benchmark CLAY erasure codes - in particular the number of I/O required for
+ backfills when multiple OSDs fail
+* Enhance test cases to validate the implementation
+* See also https://bugzilla.redhat.com/show_bug.cgi?id=2004256
projects / ceph.git / commitdiff