From: Feng, Hualong Date: Wed, 17 Jul 2024 06:53:32 +0000 (+0000) Subject: doc/radosgw/qat-accel: Update and Add QATlib information X-Git-Tag: v20.0.0~1411^2 X-Git-Url: http://git.apps.os.sepia.ceph.com/?a=commitdiff_plain;h=98723d51b969041a672d505a26d4e0a8df4115d3;p=ceph.git doc/radosgw/qat-accel: Update and Add QATlib information Signed-off-by: Feng, Hualong --- diff --git a/doc/radosgw/qat-accel.rst b/doc/radosgw/qat-accel.rst index b275e8a194626..5022f80e81bdd 100644 --- a/doc/radosgw/qat-accel.rst +++ b/doc/radosgw/qat-accel.rst @@ -33,13 +33,20 @@ QAT Environment Setup encryption and compression services. And QAT driver in kernel space have to be loaded to drive the hardware. -The driver package can be downloaded from `Intel Quickassist Technology`_. +The out-of-tree QAT driver package can be downloaded from `Intel Quickassist +Technology`_. -2. The implementation for QAT based encryption is directly base on QAT API which - is included the driver package. But QAT support for compression depends on - QATzip project, which is a user space library which builds on top of the QAT - API. Currently, QATzip speeds up gzip compression and decompression at the - time of writing. +The QATlib can be downloaded from `qatlib`_, which is used for the in-tree QAT +driver. + + .. note:: + The out-of-tree QAT driver is gradually turning to intree driver+QATlib. + +2. The implementation of QAT-based encryption is directly based on the QAT API, + which is included the driver package. However, QAT support for compression + depends on the QATzip project, which is a userspace library that builds on + top of the QAT API. At the time of writing (July 2024), QATzip speeds up + gzip compression and decompression. See `QATzip`_. @@ -48,36 +55,40 @@ Implementation 1. QAT based Encryption for RGW `OpenSSL support for RGW encryption`_ has been merged into Ceph, and Intel also -provides one `QAT Engine`_ for OpenSSL. So, theoretically speaking, QAT based -encryption in Ceph can be directly supported through OpenSSl+QAT Engine. +provides one `QAT Engine`_ for OpenSSL. Theoretically, QAT-based encryption in +Ceph can be directly supported through OpenSSl+QAT Engine. -But the QAT Engine for OpenSSL currently supports chained operations only, and -so Ceph will not be able to utilize QAT hardware feature for crypto operations -based on OpenSSL crypto plugin. As a result, one QAT plugin based on native -QAT API is added into crypto framework. +However, the QAT Engine for OpenSSL currently supports only chained operations, +which means that Ceph will not be able to utilize QAT hardware features for +crypto operations based on the OpenSSL crypto plugin. As a result, one QAT plugin +based on native QAT API is added into the crypto framework. 2. QAT Support for Compression -As mentioned above, QAT support for compression is based on QATzip library in -user space, which is designed to take full advantage of the performance provided -by QuickAssist Technology. Unlike QAT based encryption, QAT based compression -is supported through a tool class for QAT acceleration rather than a compressor -plugin. The common tool class can transparently accelerate the existing compression -types, but only zlib compressor can be supported at the time of writing. So -user is allowed to use it to speed up zlib compressor as long as the QAT -hardware is available and QAT is capable to handle it. +As mentioned above, QAT support for compression is based on the QATzip library +in user space, which is designed to take full advantage of the performance that +QuickAssist Technology provides. Unlike QAT-based encryption, QAT-based +compression is supported through a tool class for QAT acceleration rather than +a compressor plugin. This common tool class can transparently accelerate the +existing compression types, but only zlib compressor is supported at the +time of writing. This means that the user can use this tool class to speed up +zlib compressor if the QAT hardware is available and if QAT is capable of +handling it. Configuration ============= #. Prerequisites - Make sure the QAT driver with version v1.7.L.4.14.0 or higher has been installed. - Remember to set an environment variable "ICP_ROOT" for your QAT driver package - root directory. + **For out-of-tree QAT** - To enable the QAT based encryption and compression, user needs to modify the QAT - configuration files. For example, for Intel QuickAssist Adapter 8970 product, revise - c6xx_dev0/1/2.conf in the directory ``/etc/`` and keep them the same, e.g.: + Make sure the out-of-tree QAT driver with version v1.7.L.4.14.0 or higher + has been installed. Remember to set an environment variable ``ICP_ROOT`` + for your QAT driver package root directory. + + To enable the QAT based encryption and compression, the user must modify the + QAT configuration files. For example, for the Intel QuickAssist Adapter 8970 + product, revise ``c6xx_dev0/1/2.conf`` in the directory ``/etc/`` and keep them + the same. For example: .. code-block:: ini @@ -101,51 +112,121 @@ Configuration # List of core affinities Dc0CoreAffinity = 0 -#. QAT based Encryption for RGW + **For in-tree QAT** + + There are some prerequisites for using QATlib. Make sure that your system + meets the `QATlib System Requirements`_ . + + * To properly use the QATlib library, the Intel VT-d and SR-IOV parameters + must be enabled in the platform BIOS. + * Some qatlib features require a recent kernel driver or firmware version. + See `QATlib Kernel Driver Releases`_. + * The supported platform contains a 4xxx Intel Communications device or + newer. + * The ``intel_iommu`` parameter must be enabled. Verify that this setting is + enabled by running the following commands: + + .. prompt:: bash $ - The CMake option ``WITH_QAT=ON`` must be configured. If you build Ceph from + cat /proc/cmdline | grep intel_iommu=on + sudo sh -c 'echo "@qat - memlock 204800" >> /etc/security/limits.conf' + sudo su -l $USER + + For configuration and Tuning see `QATlib Configuration and Tuning`_. + +#. QAT-based Encryption for RGW + + The CMake option ``WITH_QATDRV=ON`` must be set. If you build Ceph from source code (see: :ref:`build-ceph`), navigate to your cloned Ceph repository and execute the following: .. prompt:: bash $ cd ceph - ./do_cmake.sh -DWITH_QAT=ON + ./do_cmake.sh -DWITH_QATDRV=ON cd build ininja - .. note:: - The section name of the QAT configuration files must be ``CEPH`` since - the section name is set as "CEPH" in Ceph crypto source code. + .. note:: The section name of the QAT configuration files must be ``CEPH``, + because the section name is set as ``CEPH`` in the Ceph crypto source code. - Then, edit the Ceph configuration file to make use of QAT based crypto plugin:: + Edit the Ceph configuration file (usually ``ceph.conf``) to make use of the + QAT-based crypto plugin:: plugin crypto accelerator = crypto_qat #. QAT Support for Compression - Before starting, make sure both QAT driver and `QATzip`_ have been installed. Besides - "ICP_ROOT", remember to set the environment variable "QZ_ROOT" for the root directory - of your QATzip source tree. + **For out-of-tree QAT** - The following CMake options have to be configured to trigger QAT based compression - when building Ceph: + For the out-of-tree QAT driver package, before starting ensure that both QAT + driver and `QATzip`_ have been installed. Besides ``ICP_ROOT``, remember to + set the environment variable ``QZ_ROOT`` to the root directory of your QATzip + source tree. + + The following CMake options have to be configured to trigger QAT-based + compression when building Ceph: .. prompt:: bash $ - ./do_cmake.sh -DWITH_QAT=ON -DWITH_QATZIP=ON + ./do_cmake.sh -DWITH_QATDRV=ON -DWITH_QATZIP=ON -DWITH_SYSTEM_QATZIP=ON -DWITH_QATLIB=OFF - Then, set an environment variable to clarify the section name of User Process Instance - Section in QAT configuration files, e.g.: + Set an environment variable to clarify the section name of User Process + Instance Section in the QAT configuration files. For example: .. prompt:: bash $ export QAT_SECTION_NAME=CEPH - Next, edit the Ceph configuration file to enable QAT support for compression:: + **For in-tree QAT** + + For in-tree QAT, make sure that your system meets the `QATlib System + Requirements`_. QATlib can be installed from pre-built packages or from + source code. See `QATlib Installation`_ . After QATlib is installed, you + can run ``cpa_sample_code`` to check if the QAT environment is OK. + + If you are using QATlib source code, Ceph cmake build has the compilation + options of qatlib and qatzip enabled by default. Our normal compilation + already includes QAT-compressor-related code. + + .. prompt:: bash $ + + ./do_cmake.sh + + If you are using pre-built packages installed on the system, the following + CMake options have to be configured when building Ceph: + + .. prompt:: bash $ + + ./do_cmake.sh -DWITH_SYSTEM_QATLIB=ON -DWITH_SYSTEM_QATZIP=ON + + + **For both out-of-tree QAT and in-tree QAT** + + Edit the Ceph configuration file (usually ``ceph.conf``) to enable QAT + support for *zlib* compression:: qat compressor enabled=true + Set the RGW compression method: + + .. prompt:: bash $ + + # for storage class(STANDARD) + radosgw-admin zone placement modify --rgw-zone=default --placement-id=default-placement --compression=zlib + # or create a new storage class(COLD) and define data pool(default.rgw.cold.data) + radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class COLD + radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class COLD --compression zlib --data-pool default.rgw.cold.data + +CONFIG REFERENCE +================ +The following QAT-related settings can be added to the Ceph configuration file +(usually `ceph.conf`) under the ``[client.rgw.{instance-name}]`` section. + +.. confval:: qat_compressor_session_max_number +.. confval:: qat_compressor_busy_polling + + .. _QAT Support for Compression: https://github.com/ceph/ceph/pull/19714 .. _QAT based Encryption for RGW: https://github.com/ceph/ceph/pull/19386 @@ -153,3 +234,9 @@ Configuration .. _QATzip: https://github.com/intel/QATzip .. _OpenSSL support for RGW encryption: https://github.com/ceph/ceph/pull/15168 .. _QAT Engine: https://github.com/intel/QAT_Engine +.. _qatlib: https://github.com/intel/qatlib +.. _QATlib User's Guide: https://intel.github.io/quickassist/qatlib/index.html +.. _QATlib System Requirements: https://intel.github.io/quickassist/qatlib/requirements.html +.. _QATlib Installation: https://intel.github.io/quickassist/qatlib/install.html +.. _QATlib Configuration and Tuning: https://intel.github.io/quickassist/qatlib/configuration.html +.. _QATlib Kernel Driver Releases: https://intel.github.io/quickassist/RN/In-Tree/in_tree_firmware_RN.html#qat-kernel-driver-releases-features