From: Maya Gilad Date: Mon, 13 Sep 2021 17:11:39 +0000 (+0300) Subject: doc added msgr on-wire-compression X-Git-Tag: v17.1.0~644^2 X-Git-Url: http://git.apps.os.sepia.ceph.com/?a=commitdiff_plain;h=3f5dca49773bfe3b686c4c1449ed922bda640e92;p=ceph.git doc added msgr on-wire-compression Signed-off-by: Maya Gilad --- diff --git a/doc/dev/msgr2.rst b/doc/dev/msgr2.rst index db8f3fa2e194c..af1421a3b878f 100644 --- a/doc/dev/msgr2.rst +++ b/doc/dev/msgr2.rst @@ -104,7 +104,8 @@ Each frame has a 32-byte preamble:: __le32 segment length __le16 segment alignment } * 4 - reserved (2 bytes) + __u8 flags + reserved (1 byte) __le32 preamble crc An empty frame has one empty segment. A non-empty frame can have @@ -114,6 +115,10 @@ empty. If there are less than four segments, unused (trailing) segment length and segment alignment fields are zeroed. +### Currently supported flags + + 1. FRAME_EARLY_DATA_COMPRESSED (see :ref:`msgr-post-compression`) + The reserved bytes are zeroed. The preamble checksum is CRC32-C. It covers everything up to @@ -528,6 +533,64 @@ where epilogue is:: late_status has the same meaning as in msgr2.1-crc mode. +Compression +----------- +Compression handshake is implemented using msgr2 feature-based handshaking. +In this phase, the client will indicate the server if on-wire-compression can be used for message transmitting, +in addition to the list of supported compression methods. If on-wire-compression is enabled for both client and server, +the server will choose a compression method based on client's request and its' own preferences. +Once the handshake is completed, both peers have setup their compression handlers (if desired). + +* TAG_COMPRESSION_REQUEST (client->server): declares compression capabilities and requirements:: + + bool is_compress + std::vector preferred_methods + + - if the client identifies that both peers support compression feature, it initiates the handshake. + - is_compress flag indicates whether the client's configuration is to use compression. + - preferred_methods is a list of compression algorithms that are supported by the client. + +* TAG_COMPRESSION_DONE (server->client) : determines on compression settings:: + + bool is_compress + uint32_t method + + - the server determines whether compression is possible according to its' configuration. + - if it is possible, it will pick its' most prioritizied compression method that is also supprorted by the client. + - if none exists, it will determine that session between the peers will be handled without compression. + +.. ditaa:: + + +---------+ +--------+ + | Client | | Server | + +---------+ +--------+ + | compression request | + |----------------------->| + |<-----------------------| + | compression done | + +# msgr2.x-secure mode + +Combining compression with encryption introduces security implications. +Compression will not be possible when using secure mode, unless configured specifically by an admin. + +.. _msgr-post-compression: + +Post-compression frame format +----------------------------- +Depending on the negotiated connection mode from TAG_COMPRESSION_DONE, the connection is able to acccept/send compressed frames or process all frames as decompressed. + +# msgr2.x-force mode + +All subsequent frames that will be sent via the connection will be compressed if compression requirements are met (e.g, the frames size). + +For compressed frames, the sending peer will enable the FRAME_EARLY_DATA_COMPRESSED flag, thus allowing the accepting peer to detect it and decompress the frame. + +# msgr2.x-none mode + +FRAME_EARLY_DATA_COMPRESSED flag will be disabled in preamble. + + Message flow handshake ---------------------- diff --git a/doc/rados/configuration/msgr2.rst b/doc/rados/configuration/msgr2.rst index 685422720cfe2..e1beaf5dc4cc3 100644 --- a/doc/rados/configuration/msgr2.rst +++ b/doc/rados/configuration/msgr2.rst @@ -146,6 +146,41 @@ secure) requirements on communication with the monitors. .. confval:: ms_mon_service_mode .. confval:: ms_mon_client_mode + +Compression modes +----------------- + +The v2 protocol supports two compression modes: + +* *force* mode provides: + + - In multi-availability zones deployment, compressing replication messages between OSDs saves latency. + - In the public cloud, inter-AZ communications are expensive. Thus, minimizing message + size reduces network costs to cloud provider. + - When using instance storage on AWS (probably other public clouds as well) the instances with NVMe + provide low network bandwidth relative to the device bandwidth. + In this case, NW compression can improve the overall performance since this is clearly + the bottleneck. + +* *none* mode provides: + + - messages are transmitted without compression. + + +Compression mode configuration options +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For all connections, there is an option that controls compression usage in secure mode + +.. confval:: ms_compress_secure + +There is a parallel set of options that apply specifically to OSDs, +allowing administrators to set different requirements on communication between OSDs. + +.. confval:: ms_osd_compress_mode +.. confval:: ms_osd_compress_min_size +.. confval:: ms_osd_compression_algorithm + Transitioning from v1-only to v2-plus-v1 ----------------------------------------