--- /dev/null
+.. _csi-cephfs:
+
+======================
+ CephFS with Ceph-CSI
+======================
+
+The CephFS driver (``cephfs.csi.ceph.com``) provisions shared POSIX file
+systems backed by :ref:`CephFS<ceph-file-system>`. Volumes support
+``ReadWriteMany`` access, so many pods on many nodes can mount the same
+volume simultaneously. Each provisioned volume is a CephFS subvolume
+(see :ref:`fs-volumes-and-subvolumes`).
+
+
+Prepare a File System
+=====================
+
+Create a CephFS volume, or reuse an existing one:
+
+.. prompt:: bash $
+
+ ceph fs volume create cephfs
+
+The driver stores provisioned volumes as subvolumes in a subvolume
+group. Create the group that your StorageClass will use (the Ceph-CSI
+default group name is ``csi``):
+
+.. prompt:: bash $
+
+ ceph fs subvolumegroup create cephfs csi
+
+
+Create a cephx User
+===================
+
+Create a user for the driver, restricted to the file system and the
+subvolume group created above:
+
+.. prompt:: bash $
+
+ ceph auth get-or-create client.csi-cephfs \
+ mgr 'allow rw' \
+ mon 'allow r fsname=cephfs' \
+ mds 'allow r fsname=cephfs path=/volumes, allow rws fsname=cephfs path=/volumes/csi' \
+ osd 'allow rwx tag cephfs metadata=cephfs, allow rw tag cephfs data=cephfs'
+
+Record the generated key. It is stored in a Kubernetes ``Secret`` that
+the StorageClass references. See :doc:`/cephfs/client-auth` for
+background on CephFS authentication and the up-to-date capability
+requirements in the `Ceph-CSI capabilities documentation`_.
+
+
+Define a StorageClass
+=====================
+
+A minimal StorageClass for the CephFS driver looks like this:
+
+.. code-block:: yaml
+
+ apiVersion: storage.k8s.io/v1
+ kind: StorageClass
+ metadata:
+ name: ceph-cephfs
+ provisioner: cephfs.csi.ceph.com
+ parameters:
+ clusterID: <cluster id>
+ fsName: cephfs
+ reclaimPolicy: Delete
+ allowVolumeExpansion: true
+
+Complete examples, including the secret references that the provisioner
+and node plugin require, are maintained in the `Ceph-CSI CephFS
+examples`_. See :ref:`csi-deployment` for how the ``clusterID`` value
+is determined.
+
+
+Features
+========
+
+The CephFS driver supports volume expansion, snapshots and clones
+through the standard Kubernetes APIs, and both kernel and FUSE mounters
+on the nodes. See the `Ceph-CSI documentation`_ for the full feature
+matrix and configuration details.
+
+.. _Ceph-CSI capabilities documentation: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md
+.. _Ceph-CSI CephFS examples: https://github.com/ceph/ceph-csi/tree/devel/examples/cephfs
+.. _Ceph-CSI documentation: https://ceph.github.io/ceph-csi/
--- /dev/null
+.. _csi-deployment:
+
+====================
+ Deploying Ceph-CSI
+====================
+
+There are several ways to deploy the Ceph-CSI drivers into a Kubernetes
+cluster. The `Ceph-CSI-Operator`_ is the recommended method for new
+deployments. Helm charts and raw manifests remain available, and Rook
+users get Ceph-CSI automatically.
+
+.. note::
+
+ If you deploy Ceph with `Rook`_, Rook installs and manages Ceph-CSI
+ for you. Do not deploy Ceph-CSI by hand in a Rook cluster.
+
+
+Gather Ceph Cluster Information
+===============================
+
+Whichever method you choose, the drivers need to know how to reach the
+Ceph cluster. Collect the cluster ``fsid`` and the monitor addresses:
+
+.. prompt:: bash $
+
+ ceph mon dump
+
+The monitor addresses are listed in the driver's cluster connection
+configuration. The ``clusterID`` parameter that StorageClasses use to
+select a cluster must match the name of the corresponding
+``ClientProfile`` resource when deploying with the operator; Helm and
+raw manifest deployments conventionally use the cluster ``fsid`` as
+the ``clusterID``. Each backend page in this chapter describes the
+cephx user that the drivers authenticate with.
+
+
+Ceph-CSI-Operator (Recommended)
+===============================
+
+The `Ceph-CSI-Operator`_ manages the deployment, configuration, and
+lifecycle of Ceph-CSI drivers through Kubernetes custom resources.
+Install the operator:
+
+.. prompt:: bash $
+
+ kubectl create -f https://raw.githubusercontent.com/ceph/ceph-csi-operator/main/deploy/all-in-one/install.yaml
+
+Then deploy a driver by creating a ``Driver`` resource. For example, to
+deploy the RBD driver:
+
+.. code-block:: yaml
+
+ apiVersion: csi.ceph.io/v1
+ kind: Driver
+ metadata:
+ name: rbd.csi.ceph.com
+ namespace: ceph-csi-operator-system
+
+The CephFS and NFS drivers are deployed the same way, using the names
+``cephfs.csi.ceph.com`` and ``nfs.csi.ceph.com``. Connection details
+for the Ceph cluster are supplied through the operator's
+``CephConnection`` and ``ClientProfile`` resources; cephx credentials
+are stored in Kubernetes Secrets referenced from the StorageClass. See
+the
+`operator quick start`_ and `operator installation guide`_ for complete
+walkthroughs, supported Kubernetes versions, and Helm-based installation
+of the operator itself.
+
+
+Helm Charts
+===========
+
+The Ceph-CSI project publishes per-driver Helm charts:
+
+.. prompt:: bash $
+
+ helm repo add ceph-csi https://ceph.github.io/csi-charts
+ helm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd \
+ --create-namespace
+
+A ``ceph-csi-cephfs`` chart is available for the CephFS driver. Chart
+values are documented in the `Ceph-CSI charts`_ directory.
+
+
+Raw Manifests
+=============
+
+Per-driver Kubernetes manifests are maintained in the ``deploy/``
+directory of the Ceph-CSI repository, together with deployment
+instructions in the `Ceph-CSI documentation`_. This method offers the
+most control but requires you to track and apply manifest changes on
+every upgrade.
+
+.. _Ceph-CSI-Operator: https://github.com/ceph/ceph-csi-operator
+.. _Rook: https://rook.io
+.. _operator quick start: https://github.com/ceph/ceph-csi-operator/blob/main/docs/quick-start.md
+.. _operator installation guide: https://github.com/ceph/ceph-csi-operator/blob/main/docs/installation.md
+.. _Ceph-CSI charts: https://github.com/ceph/ceph-csi/tree/devel/charts
+.. _Ceph-CSI documentation: https://ceph.github.io/ceph-csi/
--- /dev/null
+.. _ceph-csi:
+
+=========================================
+ Ceph Container Storage Interface (CSI)
+=========================================
+
+`Ceph-CSI <https://github.com/ceph/ceph-csi>`__ implements the
+`Container Storage Interface`_ specification for Ceph storage.
+Container orchestrators such as Kubernetes use it to dynamically
+provision Ceph volumes and to attach those volumes to workloads.
+Kubernetes clusters that dynamically provision Ceph volumes rely on
+Ceph-CSI, whether Ceph runs inside the cluster under `Rook`_ or as an
+external cluster. Other CSI-capable platforms, including HashiCorp
+Nomad, can use Ceph-CSI as well.
+
+.. ditaa::
+
+ +---------------------------------------------------+
+ | Kubernetes and other CSI platforms |
+ +---------------------------------------------------+
+ | Ceph CSI |
+ +------------------------+--------------------------+
+ |
+ | provisions and maps
+ v
+ +---------------------------------------------------+
+ | Ceph Cluster |
+ +---------------------------------------------------+
+
+
+Storage Backends
+================
+
+Ceph-CSI ships several drivers in a single binary. Each driver exposes a
+different Ceph interface to containerized workloads:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 15 35 50
+
+ * - Driver
+ - Access model
+ - Typical use
+ * - RBD
+ - Block volumes. ``ReadWriteOnce`` file systems, ``ReadWriteMany``
+ raw block.
+ - Databases, virtual machines, and most single-writer workloads.
+ See :ref:`csi-rbd`.
+ * - CephFS
+ - Shared POSIX file system. ``ReadWriteMany``.
+ - Workloads that share data across many pods. See :ref:`csi-cephfs`.
+ * - NFS
+ - CephFS subvolumes exported over NFS.
+ - Clients that cannot run Ceph client code. See :ref:`csi-nfs`.
+ * - NVMe-oF
+ - Block volumes over standard NVMe/TCP initiators.
+ - Under active development. See :ref:`csi-nvmeof`.
+
+
+Where the Documentation Lives
+=============================
+
+Ceph-CSI is developed and released independently of Ceph itself, and one
+Ceph-CSI release supports multiple Ceph releases. For that reason the
+canonical driver documentation is maintained by the Ceph-CSI project and
+published at `ceph.github.io/ceph-csi`_.
+
+The pages in this chapter cover the Ceph side of a Ceph-CSI deployment:
+preparing pools, file systems, and cephx users, and choosing a deployment
+method. For driver internals, complete example manifests, version
+compatibility, and upgrade procedures, follow the links to the canonical
+documentation in each page.
+
+.. toctree::
+ :maxdepth: 1
+
+ Deployment <deployment>
+ RBD <rbd>
+ CephFS <cephfs>
+ NFS <nfs>
+ NVMe-oF <nvmeof>
+
+.. _Container Storage Interface: https://github.com/container-storage-interface/spec
+.. _Rook: https://rook.io
+.. _ceph.github.io/ceph-csi: https://ceph.github.io/ceph-csi/
--- /dev/null
+.. _csi-nfs:
+
+===================
+ NFS with Ceph-CSI
+===================
+
+The NFS driver (``nfs.csi.ceph.com``) provisions CephFS subvolumes and
+exports them through an NFS-Ganesha cluster that is managed by the Ceph
+:ref:`NFS manager module<mgr-nfs>`. Exports are created and deleted
+dynamically as volumes are provisioned and destroyed. Pods mount the
+volumes with the ordinary NFS client, so the worker nodes need no Ceph
+client code at all. This makes the NFS driver a good fit for platforms
+where installing Ceph-specific components on every node is not
+practical.
+
+
+Prepare a File System and an NFS Cluster
+========================================
+
+The NFS driver builds on the CephFS driver: provision a CephFS volume
+and a cephx user as described in :ref:`csi-cephfs`. Then create an NFS
+cluster to serve the exports:
+
+.. prompt:: bash $
+
+ ceph nfs cluster create mynfs
+
+Confirm that the cluster is up and note the address that clients will
+mount from:
+
+.. prompt:: bash $
+
+ ceph nfs cluster info mynfs
+
+See :ref:`mgr-nfs` for placement options, high availability with an
+ingress service, and the rest of the NFS cluster management commands.
+
+
+Define a StorageClass
+=====================
+
+A minimal StorageClass for the NFS driver looks like this:
+
+.. code-block:: yaml
+
+ apiVersion: storage.k8s.io/v1
+ kind: StorageClass
+ metadata:
+ name: ceph-nfs
+ provisioner: nfs.csi.ceph.com
+ parameters:
+ clusterID: <cluster id>
+ fsName: cephfs
+ nfsCluster: mynfs
+ server: <NFS cluster address>
+ reclaimPolicy: Delete
+ allowVolumeExpansion: true
+
+Complete examples, including the secret references that the provisioner
+requires, are maintained in the `Ceph-CSI NFS examples`_. See
+:ref:`csi-deployment` for how the ``clusterID`` value is determined.
+
+.. note::
+
+ The worker nodes need NFS client support in the kernel. No Ceph
+ packages are required on the nodes.
+
+.. _Ceph-CSI NFS examples: https://github.com/ceph/ceph-csi/tree/devel/examples/nfs
--- /dev/null
+.. _csi-nvmeof:
+
+=======================
+ NVMe-oF with Ceph-CSI
+=======================
+
+.. note::
+
+ NVMe-oF support in Ceph-CSI is under active development and is not
+ yet generally available. Check the `Ceph-CSI project`_ for the
+ current status before planning a deployment on it.
+
+The NVMe-oF driver exposes RBD images to containerized workloads
+through the :ref:`Ceph NVMe-oF gateway<ceph-nvmeof>` rather than
+through the RBD client on each node. Worker nodes attach volumes with
+the standard NVMe/TCP initiator that ships with the Linux kernel, so,
+as with the NFS driver, no Ceph client code is needed on the nodes.
+This suits environments that standardize on NVMe/TCP for storage
+access or that cannot run RBD kernel clients.
+
+
+Prepare the Gateway
+===================
+
+The driver requires a running NVMe-oF gateway group in the Ceph
+cluster. See :ref:`ceph-nvmeof` for the gateway architecture, hardware
+sizing, and deployment instructions, and :doc:`/rbd/nvmeof-requirements`
+for gateway requirements.
+
+The gateway exposes RBD images from a pool, so the Ceph-side pool
+preparation matches the RBD driver: see :ref:`csi-rbd` for creating and
+initializing a pool.
+
+.. _Ceph-CSI project: https://github.com/ceph/ceph-csi
--- /dev/null
+.. _csi-rbd:
+
+===================
+ RBD with Ceph-CSI
+===================
+
+The RBD driver (``rbd.csi.ceph.com``) provisions
+:ref:`Ceph Block Device<ceph_block_device>` images and maps them on
+the nodes that run the pods
+which consume them. Volumes can be mounted with a file system in
+``ReadWriteOnce`` mode, or attached as raw block devices, including
+``ReadWriteMany`` raw block for clustered applications that coordinate
+their own access.
+
+
+Prepare a Pool
+==============
+
+Create a pool for container volumes and initialize it for use by RBD:
+
+.. prompt:: bash $
+
+ ceph osd pool create kubernetes
+ rbd pool init kubernetes
+
+See :ref:`rados_pools` for guidance on pool creation, including
+placement group counts.
+
+
+Create a cephx User
+===================
+
+Create a user for the driver, restricted to the pool created above. The
+same user is used for provisioning and for mapping images on the nodes:
+
+.. prompt:: bash $
+
+ ceph auth get-or-create client.csi-rbd \
+ mon 'profile rbd' \
+ osd 'profile rbd pool=kubernetes' \
+ mgr 'profile rbd pool=kubernetes'
+
+Record the generated key. It is stored in a Kubernetes ``Secret`` that
+the StorageClass references. See :ref:`user-management` for background
+on cephx capabilities and the up-to-date capability requirements in the
+`Ceph-CSI capabilities documentation`_.
+
+
+Define a StorageClass
+=====================
+
+A minimal StorageClass for the RBD driver looks like this:
+
+.. code-block:: yaml
+
+ apiVersion: storage.k8s.io/v1
+ kind: StorageClass
+ metadata:
+ name: ceph-rbd
+ provisioner: rbd.csi.ceph.com
+ parameters:
+ clusterID: <cluster id>
+ pool: kubernetes
+ imageFeatures: layering
+ reclaimPolicy: Delete
+ allowVolumeExpansion: true
+
+Complete examples, including the secret references that the provisioner
+and node plugin require, are maintained in the `Ceph-CSI RBD examples`_.
+See :ref:`csi-deployment` for how the ``clusterID`` value is
+determined.
+
+.. important::
+
+ The driver maps images with the RBD kernel client by default, which
+ may not support all :ref:`CRUSH tunables<crush-map-tunables>` or RBD
+ image features. ``imageFeatures: layering`` is a safe baseline;
+ consult the canonical documentation before enabling more features.
+
+
+Features
+========
+
+The RBD driver supports snapshots and clones through the standard
+Kubernetes APIs, volume expansion, topology-aware provisioning, and
+RBD mirroring for disaster recovery through `CSI-Addons`_. Feature
+maturity varies; see the feature matrix in the `Ceph-CSI
+documentation`_ for the current status and configuration details.
+
+.. _Ceph-CSI capabilities documentation: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md
+.. _Ceph-CSI RBD examples: https://github.com/ceph/ceph-csi/tree/devel/examples/rbd
+.. _CSI-Addons: https://github.com/csi-addons/kubernetes-csi-addons
+.. _Ceph-CSI documentation: https://ceph.github.io/ceph-csi/
cephfs/index
rbd/index
radosgw/index
+ csi/index
mgr/index
mgr/dashboard
monitoring/index
projects / ceph.git / commitdiff