From: Emmanuel Ameh Date: Tue, 7 Jul 2026 13:52:48 +0000 (+0100) Subject: doc/rbd: point rbd-kubernetes at the Ceph-CSI chapter X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=b937b96f4e042a574539224a7984c5677f24f564;p=ceph.git doc/rbd: point rbd-kubernetes at the Ceph-CSI chapter The page taught a manual manifest deployment that is superseded by the Ceph-CSI-Operator. Keep the URL alive as a signpost to the new csi chapter instead. Signed-off-by: Emmanuel Ameh --- diff --git a/doc/rbd/rbd-kubernetes.rst b/doc/rbd/rbd-kubernetes.rst index ccec4813a3ab..3fed879e4a8f 100644 --- a/doc/rbd/rbd-kubernetes.rst +++ b/doc/rbd/rbd-kubernetes.rst @@ -2,363 +2,31 @@ Block Devices and Kubernetes ============================== -You may use Ceph Block Device images with Kubernetes v1.13 and later through -`ceph-csi`_, which dynamically provisions RBD images to back Kubernetes -`volumes`_ and maps these RBD images as block devices (optionally mounting -a file system contained within the image) on worker nodes running -`pods`_ that reference an RBD-backed volume. Ceph stripes block device images as -objects across the cluster, which means that large Ceph Block Device images have -better performance than a standalone server! +Ceph Block Device images are provisioned for Kubernetes by `Ceph-CSI`_, +which dynamically creates RBD images to back Kubernetes `volumes`_ and +maps them as block devices on the worker nodes that run `pods`_ +referencing the volumes. -To use Ceph Block Devices with Kubernetes v1.13 and higher, you must install -and configure ``ceph-csi`` within your Kubernetes environment. The following -diagram depicts the Kubernetes/Ceph technology stack. +The documentation for using Ceph with Kubernetes and other container +platforms has moved to its own chapter: -.. ditaa:: - +---------------------------------------------------+ - | Kubernetes | - +---------------------------------------------------+ - | ceph--csi | - +------------------------+--------------------------+ - | - | configures - v - +------------------------+ +------------------------+ - | | | rbd--nbd | - | Kernel Modules | +------------------------+ - | | | librbd | - +------------------------+-+------------------------+ - | RADOS Protocol | - +------------------------+-+------------------------+ - | OSDs | | Monitors | - +------------------------+ +------------------------+ +- :ref:`ceph-csi` introduces Ceph-CSI and its storage backends. +- :ref:`csi-deployment` covers deployment, including the recommended + Ceph-CSI-Operator. +- :ref:`csi-rbd` covers the Ceph-side preparation for RBD volumes: + pools, cephx users, and StorageClass essentials. - -.. important:: - ``ceph-csi`` uses the RBD kernel modules by default which may not support all - Ceph `CRUSH tunables`_ or `RBD image features`_. - -Create a Pool -============= - -By default, Ceph block devices use the ``rbd`` pool. Create a pool for -Kubernetes volume storage. Ensure your Ceph cluster is running, then create -the pool. :: - - $ ceph osd pool create kubernetes - -See `Create a Pool`_ for details on specifying the number of placement groups -for your pools, and `Placement Groups`_ for details on the number of placement -groups you should set for your pools. - -A newly created pool must be initialized prior to use. Use the ``rbd`` tool -to initialize the pool:: - - $ rbd pool init kubernetes - -Configure ceph-csi -================== - -Setup Ceph Client Authentication --------------------------------- - -Create a new user for Kubernetes and `ceph-csi`. Execute the following and -record the generated key:: - - $ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes' - [client.kubernetes] - key = AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg== - -Generate `ceph-csi` `ConfigMap` -------------------------------- - -The `ceph-csi` requires a `ConfigMap` object stored in Kubernetes to define the -the Ceph monitor addresses for the Ceph cluster. Collect both the Ceph cluster -unique `fsid` and the monitor addresses:: - - $ ceph mon dump - <...> - fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8 - <...> - 0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a - 1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b - 2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c +CephFS, NFS, and NVMe-oF backends are covered in the same chapter. .. note:: - ``ceph-csi`` currently only supports the `legacy V1 protocol`_. - -Generate a `csi-config-map.yaml` file similar to the example below, substituting -the `fsid` for "clusterID", and the monitor addresses for "monitors":: - - $ cat < csi-config-map.yaml - --- - apiVersion: v1 - kind: ConfigMap - data: - config.json: |- - [ - { - "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8", - "monitors": [ - "192.168.1.1:6789", - "192.168.1.2:6789", - "192.168.1.3:6789" - ] - } - ] - metadata: - name: ceph-csi-config - EOF - -Once generated, store the new `ConfigMap` object in Kubernetes:: - - $ kubectl apply -f csi-config-map.yaml - -Recent versions of `ceph-csi` also require an additional `ConfigMap` object to -define Key Management Service (KMS) provider details. If KMS isn't set up, put -an empty configuration in a `csi-kms-config-map.yaml` file or refer to examples -at https://github.com/ceph/ceph-csi/tree/master/examples/kms:: - - $ cat < csi-kms-config-map.yaml - --- - apiVersion: v1 - kind: ConfigMap - data: - config.json: |- - {} - metadata: - name: ceph-csi-encryption-kms-config - EOF - -Once generated, store the new `ConfigMap` object in Kubernetes:: - - $ kubectl apply -f csi-kms-config-map.yaml - -Recent versions of `ceph-csi` also require yet another `ConfigMap` object -to define Ceph configuration to add to ceph.conf file inside CSI containers:: - - $ cat < ceph-config-map.yaml - --- - apiVersion: v1 - kind: ConfigMap - data: - ceph.conf: | - [global] - auth_cluster_required = cephx - auth_service_required = cephx - auth_client_required = cephx - # keyring is a required key and its value should be empty - keyring: | - metadata: - name: ceph-config - EOF - -Once generated, store the new `ConfigMap` object in Kubernetes:: - - $ kubectl apply -f ceph-config-map.yaml - -Generate `ceph-csi` cephx `Secret` ----------------------------------- - -`ceph-csi` requires the cephx credentials for communicating with the Ceph -cluster. Generate a `csi-rbd-secret.yaml` file similar to the example below, -using the newly created Kubernetes user id and cephx key:: - - $ cat < csi-rbd-secret.yaml - --- - apiVersion: v1 - kind: Secret - metadata: - name: csi-rbd-secret - namespace: default - stringData: - userID: kubernetes - userKey: AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg== - EOF - -Once generated, store the new `Secret` object in Kubernetes:: - - $ kubectl apply -f csi-rbd-secret.yaml - -Configure `ceph-csi` Plugins ----------------------------- - -Create the required `ServiceAccount` and RBAC `ClusterRole`/`ClusterRoleBinding` -Kubernetes objects. These objects do not necessarily need to be customized for -your Kubernetes environment and therefore can be used as-is from the `ceph-csi` -deployment YAMLs:: - - $ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml - $ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml - -Finally, create the `ceph-csi` provisioner and node plugins. With the -possible exception of the `ceph-csi` container release version, these objects do -not necessarily need to be customized for your Kubernetes environment and -therefore can be used as-is from the `ceph-csi` deployment YAMLs:: - - $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml - $ kubectl apply -f csi-rbdplugin-provisioner.yaml - $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml - $ kubectl apply -f csi-rbdplugin.yaml - -.. important:: - The provisioner and node plugin YAMLs will, by default, pull the development - release of the `ceph-csi` container (quay.io/cephcsi/cephcsi:canary). - The YAMLs should be updated to use a release version container for - production workloads. - -Using Ceph Block Devices -======================== - -Create a `StorageClass` ------------------------ - -The Kubernetes `StorageClass` defines a class of storage. Multiple `StorageClass` -objects can be created to map to different quality-of-service levels (i.e. NVMe -vs HDD-based pools) and features. - -For example, to create a `ceph-csi` `StorageClass` that maps to the `kubernetes` -pool created above, the following YAML file can be used after ensuring that the -"clusterID" property matches your Ceph cluster's `fsid`:: - - $ cat < csi-rbd-sc.yaml - --- - apiVersion: storage.k8s.io/v1 - kind: StorageClass - metadata: - name: csi-rbd-sc - provisioner: rbd.csi.ceph.com - parameters: - clusterID: b9127830-b0cc-4e34-aa47-9d1a2e9949a8 - pool: kubernetes - imageFeatures: layering - csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret - csi.storage.k8s.io/provisioner-secret-namespace: default - csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret - csi.storage.k8s.io/controller-expand-secret-namespace: default - csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret - csi.storage.k8s.io/node-stage-secret-namespace: default - reclaimPolicy: Delete - allowVolumeExpansion: true - mountOptions: - - discard - EOF - $ kubectl apply -f csi-rbd-sc.yaml - -Note that in Kubernetes v1.14 and v1.15 volume expansion feature was in alpha -status and required enabling `ExpandCSIVolumes` feature gate. - -Create a `PersistentVolumeClaim` --------------------------------- - -A `PersistentVolumeClaim` is a request for abstract storage resources by a user. -The `PersistentVolumeClaim` would then be associated to a `Pod` resource to -provision a `PersistentVolume`, which would be backed by a Ceph block image. -An optional `volumeMode` can be included to select between a mounted file system -(default) or raw block device-based volume. - -Using `ceph-csi`, specifying `Filesystem` for `volumeMode` can support both -`ReadWriteOnce` and `ReadOnlyMany` `accessMode` claims, and specifying `Block` -for `volumeMode` can support `ReadWriteOnce`, `ReadWriteMany`, and -`ReadOnlyMany` `accessMode` claims. - -For example, to create a block-based `PersistentVolumeClaim` that utilizes -the `ceph-csi`-based `StorageClass` created above, the following YAML can be -used to request raw block storage from the `csi-rbd-sc` `StorageClass`:: - - $ cat < raw-block-pvc.yaml - --- - apiVersion: v1 - kind: PersistentVolumeClaim - metadata: - name: raw-block-pvc - spec: - accessModes: - - ReadWriteOnce - volumeMode: Block - resources: - requests: - storage: 1Gi - storageClassName: csi-rbd-sc - EOF - $ kubectl apply -f raw-block-pvc.yaml - -The following demonstrates and example of binding the above -`PersistentVolumeClaim` to a `Pod` resource as a raw block device:: - - $ cat < raw-block-pod.yaml - --- - apiVersion: v1 - kind: Pod - metadata: - name: pod-with-raw-block-volume - spec: - containers: - - name: fc-container - image: fedora:26 - command: ["/bin/sh", "-c"] - args: ["tail -f /dev/null"] - volumeDevices: - - name: data - devicePath: /dev/xvda - volumes: - - name: data - persistentVolumeClaim: - claimName: raw-block-pvc - EOF - $ kubectl apply -f raw-block-pod.yaml - -To create a file-system-based `PersistentVolumeClaim` that utilizes the -`ceph-csi`-based `StorageClass` created above, the following YAML can be used to -request a mounted file system (backed by an RBD image) from the `csi-rbd-sc` -`StorageClass`:: - - $ cat < pvc.yaml - --- - apiVersion: v1 - kind: PersistentVolumeClaim - metadata: - name: rbd-pvc - spec: - accessModes: - - ReadWriteOnce - volumeMode: Filesystem - resources: - requests: - storage: 1Gi - storageClassName: csi-rbd-sc - EOF - $ kubectl apply -f pvc.yaml - -The following demonstrates and example of binding the above -`PersistentVolumeClaim` to a `Pod` resource as a mounted file system:: - $ cat < pod.yaml - --- - apiVersion: v1 - kind: Pod - metadata: - name: csi-rbd-demo-pod - spec: - containers: - - name: web-server - image: nginx - volumeMounts: - - name: mypvc - mountPath: /var/lib/www/html - volumes: - - name: mypvc - persistentVolumeClaim: - claimName: rbd-pvc - readOnly: false - EOF - $ kubectl apply -f pod.yaml + Earlier versions of this page walked through deploying Ceph-CSI by + hand-editing raw manifests. That deployment method is superseded by + the Ceph-CSI-Operator and Helm charts described in + :ref:`csi-deployment`; the manifests remain available in the + `Ceph-CSI repository`_. -.. _ceph-csi: https://github.com/ceph/ceph-csi/ +.. _Ceph-CSI: https://github.com/ceph/ceph-csi/ +.. _Ceph-CSI repository: https://github.com/ceph/ceph-csi/tree/devel/deploy .. _volumes: https://kubernetes.io/docs/concepts/storage/volumes/ -.. _pods: https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/ -.. _Create a Pool: ../../rados/operations/pools#createpool -.. _Placement Groups: ../../rados/operations/placement-groups -.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables -.. _RBD image features: ../rbd-config-ref/#image-features -.. _legacy V1 protocol: ../../rados/configuration/msgr2/#address-formats +.. _pods: https://kubernetes.io/docs/concepts/workloads/pods/