From: Redouane Kachach Date: Mon, 8 Jun 2026 09:54:17 +0000 (+0200) Subject: doc/mgr/ceph_secrets: add documentation for the ceph_secrets module X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=ead4ef7f9af55088b27a43969bcc0e093ea85ed4;p=ceph.git doc/mgr/ceph_secrets: add documentation for the ceph_secrets module Document CLI commands (set/get/get-value/ls/rm), the Python API via CephSecretsClient, secret URI embedding and resolution, and epoch-based change detection. Fixes: https://tracker.ceph.com/issues/74562 Assisted-by: Claude Assisted-by: ChatGPT Signed-off-by: Redouane Kachach --- diff --git a/doc/mgr/ceph_secrets.rst b/doc/mgr/ceph_secrets.rst new file mode 100644 index 00000000000..45675c002d2 --- /dev/null +++ b/doc/mgr/ceph_secrets.rst @@ -0,0 +1,437 @@ +.. _mgr-ceph-secrets: + +=================== +Ceph Secrets Module +=================== + +The ``ceph_secrets`` manager module provides centralised secret storage for +Ceph operators and Ceph manager modules. Instead of embedding plaintext +credentials in service specifications or configuration objects, secrets are +stored once and referenced by URI. Ceph manager modules such as cephadm +and rook can store secret payloads centrally, reference them by URI, and +resolve them to plaintext only when needed at deploy time. + +For example, a cephadm-managed service may need an API token. Instead of +embedding that token directly in the service specification, an operator can +store it once: + +.. prompt:: bash $ + + ceph secret set cephadm/service/my-service/api_token -i /tmp/api-token + +and then use the URI +``secret:/cephadm/service/my-service/api_token`` in the +service spec instead of the plaintext token. A cephadm integration can then +resolve the URI at deploy time and write the token only into the daemon files +that need it. + +Secrets are stored in the Mon KV store under the ``secret_store/v1/`` prefix +and are organised by namespace, scope, and name. Each secret is versioned and +carries ``created``/``updated`` timestamps. A per-namespace epoch counter +is incremented on every ``set`` and on any ``rm`` that actually removes a +secret, allowing consumers to detect changes without fetching the full secret +list. + +.. note:: + + The ``mon`` backend stores secrets in the Mon KV store, which is not an + external KMS or vault. Users and MGR modules with sufficient Ceph + permissions can still reveal or resolve stored secret values. Namespaces + provide logical and storage isolation, not an authorisation boundary. + +If the module is not already enabled, run:: + + ceph mgr module enable ceph_secrets + +.. contents:: + :local: + :depth: 2 + + +Concepts +======== + +Namespace +--------- + +A namespace is a logical storage boundary, typically matching the name of +the consuming MGR module (e.g. ``cephadm``, ``rook``). Secrets in different +namespaces are stored independently and have independent epoch counters. +Namespaces are not an authorisation boundary; any caller with access to the +``ceph_secrets`` module can read secrets from any namespace. + +Scope +----- + +Within a namespace, every secret has a scope that encodes what the secret +belongs to: + +.. list-table:: + :header-rows: 1 + :widths: 15 40 45 + + * - Scope + - Meaning + - CLI path form + * - ``global`` + - Cluster-wide secret, not tied to any specific target + - ``/global/`` + * - ``service`` + - Secret for a specific named service + - ``/service//`` + * - ``host`` + - Secret for a specific host + - ``/host//`` + * - ``custom`` + - Arbitrary slash-delimited path under the namespace + - ``/custom/`` + +Path grammar +------------ + +All path segments (namespace, scope target, name, and each segment of a +custom path) must match ``[A-Za-z0-9._-]+`` and must not end with ``'.'``. +Empty segments, percent-encoding, and leading/trailing whitespace are +rejected. + +Custom scope paths may contain multiple slash-separated segments (e.g. +``cephadm/custom/app/db/password``). Two custom paths that share a common +prefix are independent secrets; ``cephadm/custom/a/b`` and +``cephadm/custom/a/b/c`` do not conflict. + +Secret URIs +----------- + +Internally, secrets are identified by URIs of the form:: + + secret://global/ + secret://service// + secret://host// + secret://custom/ + +These URIs appear in the output of :ref:`scan_refs ` +and may be embedded in configuration objects as opaque references to be +resolved at deploy time. + + +CLI Reference +============= + +All CLI commands use the path form described above. Responses are JSON by +default; pass ``--format yaml`` for YAML output. + +.. _mgr-ceph-secrets-set: + +secret set +---------- + +Create or update a secret. The input file content is stored as an opaque string:: + + ceph secret set -i + +Secret data must not be empty. Other contents, including leading or trailing +whitespace and final newlines, are preserved exactly. Callers that need to +store structured data should encode it themselves, for example as JSON, and +decode it after retrieval or resolution. + +If the secret already exists its data is replaced and its version +incremented, unless the existing secret policy marks it non-editable (set via +the Python API with ``editable=False``), in which case the update is rejected. +The ``created`` timestamp is set on the first write and is never changed +thereafter; ``updated`` is refreshed on every write. + +Example: + +.. prompt:: bash $ + + ceph secret set cephadm/service/my-service/api_token -i /tmp/api-token + {"metadata": {"version": 1, "created": "2025-06-01T12:00:00Z", "updated": "2025-06-01T12:00:00Z"}} + +.. _mgr-ceph-secrets-get: + +secret get +---------- + +Retrieve the metadata (and, optionally, the data) for a secret:: + + ceph secret get [--reveal] [--format {json|yaml}] + +Without ``--reveal``, the ``data`` field is omitted from the response to +avoid accidental exposure in terminal output or logs. With ``--reveal``, +``data`` contains the stored opaque string. + +Example: + +.. prompt:: bash $ + + ceph secret get cephadm/service/my-service/api_token + { + "metadata": { + "version": 1, + "created": "2025-06-01T12:00:00Z", + "updated": "2025-06-01T12:00:00Z" + } + } + + ceph secret get cephadm/service/my-service/api_token --reveal + { + "metadata": { + "version": 1, + "created": "2025-06-01T12:00:00Z", + "updated": "2025-06-01T12:00:00Z" + }, + "data": "api-token-value" + } + +.. _mgr-ceph-secrets-get-value: + +secret get-value +---------------- + +Return the raw secret data string directly, with no JSON envelope:: + + ceph secret get-value + +Unlike ``secret get --reveal``, the output is the stored string itself, +making it suitable for use in shell scripts and pipelines. Returns an +error if the secret does not exist. + +Example: + +.. prompt:: bash $ + + ceph secret get-value cephadm/service/my-service/api_token + api-token-value + +.. _mgr-ceph-secrets-ls: + +secret ls +--------- + +List secrets, optionally filtered by namespace, scope, and/or target:: + + ceph secret ls [--namespace ] [--scope ] + [--sec_target ] [--reveal] [--show_internals] + [--format {json|yaml}] + +Without filters, all secrets across all namespaces are listed. +``--reveal`` includes the stored opaque data string in each output record. +``--show_internals`` additionally includes the ``policy`` object (``user_made`` +and ``editable`` flags) in each record. + +Example: + +.. prompt:: bash $ + + ceph secret ls --namespace cephadm --scope host --sec_target node1 + { + "cephadm/host/node1/ssh_key": { + "metadata": { + "version": 3, + "created": "2025-05-10T08:00:00Z", + "updated": "2025-06-01T09:00:00Z" + }, + "ref": { + "namespace": "cephadm", + "scope": "host", + "target": "node1", + "name": "ssh_key" + } + } + } + +.. _mgr-ceph-secrets-rm: + +secret rm +--------- + +Remove a secret:: + + ceph secret rm + +The operation is idempotent: removing a secret that does not exist succeeds +and reports ``"status": "not_found"`` rather than an error. + +Example: + +.. prompt:: bash $ + + ceph secret rm cephadm/service/my-service/api_token + {"status": "removed"} + + ceph secret rm cephadm/service/my-service/api_token + {"status": "not_found"} + +.. _mgr-ceph-secrets-get-epoch: + +Epoch +----- + +The epoch for a namespace is accessible via the Python API only (see +:ref:`Epoch-based change detection `). There is +no CLI command for it. + + +Module API +========== + +Other MGR modules should consume ``ceph_secrets`` via ``CephSecretsClient`` +(``src/pybind/mgr/ceph_secrets_client.py``) rather than calling +``mgr.remote()`` directly. The client file, along with +``ceph_secrets_types.py``, lives at the top level of ``src/pybind/mgr/`` so +any MGR module can import it without depending on the ``ceph_secrets`` package +internals. + +.. code-block:: python + + from ceph_secrets_client import CephSecretsClient + from ceph_secrets_types import SecretScope + + client = CephSecretsClient(self) # self is a MgrModule instance + + # Store a secret as an opaque string + client.secret_set( + namespace="cephadm", + scope=SecretScope.HOST, + target="node1", + name="ssh_key", + data="AQB...==", + ) + + # Retrieve metadata (no data unless reveal=True) + rec = client.secret_get("cephadm", SecretScope.HOST, "node1", "ssh_key") + if rec: + version = rec["metadata"]["version"] + + # Remove + client.secret_rm("cephadm", SecretScope.HOST, "node1", "ssh_key") + +CephSecretsClient methods +-------------------------- + +.. list-table:: + :header-rows: 1 + :widths: 35 65 + + * - Method + - Description + * - ``secret_set(namespace, scope, target, name, data, ...)`` + - Create or update a secret. ``data`` must be a non-empty opaque + string. Increments the version and refreshes ``updated`` on each + call; ``created`` is set only on the first write. Optional + ``user_made`` and ``editable`` flags control the stored policy; + ``editable=False`` blocks future updates but does not prevent removal. + * - ``secret_get(namespace, scope, target, name, reveal=False)`` + - Return the secret's metadata dict, plus the opaque ``data`` string if + *reveal* is True. Returns ``None`` if the secret does not exist. + * - ``secret_get_value(namespace, scope, target, name)`` + - Return the stored opaque string directly, or ``None`` if the secret + does not exist. Use this when only the value is needed and the + metadata envelope is not required. + * - ``secret_get_version(namespace, scope, target, name)`` + - Return the current version integer, or ``None`` if not found. A + cheaper alternative to ``secret_get`` when only change-detection is needed. + * - ``secret_get_versions(uris)`` + - Batch-fetch version numbers for a list of canonical ``secret:/...`` + URIs. Returns a dict keyed by URI; missing secrets map to ``None``. + Malformed URIs are skipped entirely, so a missing key indicates + malformed input rather than an absent secret. + * - ``secret_rm(namespace, scope, target, name)`` + - Remove a secret. Idempotent: returns ``False`` if not found. + * - ``secret_get_epoch(namespace)`` + - Return the current epoch for the namespace. Use as a cheap + change-detector before a full refresh. + * - ``scan_refs(obj, namespace)`` + - Walk a JSON-like object and return secret-like reference strings found + within it. This includes valid whole-value secret URIs and malformed + or embedded secret-like references that should be reported to the + caller. Only canonical ``secret:/...`` URIs should be passed to + ``secret_get_versions``. + * - ``scan_unresolved_refs(obj, namespace)`` + - Like ``scan_refs`` but returns only references that are missing, + malformed, embedded inside a larger string, or cannot currently be + read successfully. Useful for pre-flight validation. + * - ``resolve_object(obj)`` + - Walk a JSON-like object and replace every whole-value ``secret:/...`` + URI string with the stored opaque string for the referenced secret. + +.. _mgr-ceph-secrets-scan: + +Secret URI embedding and resolution +----------------------------------- + +Configuration objects that need to reference secrets without embedding +plaintext can store ``secret:/...`` URIs as string values. The module +resolves them on demand: + +.. code-block:: python + + spec = { + "service_type": "my-service", + "spec": { + "api_token": " secret:/cephadm/service/my-service/api_token ", + }, + } + + # Check for missing, malformed, embedded, or unresolvable secret references + unresolved = client.scan_unresolved_refs(spec, namespace="cephadm") + if unresolved: + raise RuntimeError(f"Unresolved secret references: {unresolved}") + + # Replace URI references with plaintext values + resolved = client.resolve_object(spec) + # resolved["spec"]["api_token"] now holds the stored API token + +.. note:: + + Resolution replaces the entire string value of a field. Surrounding + whitespace around a URI reference is ignored, so values such as + ``" secret:/cephadm/global/foo "`` are accepted. Embedding a secret URI + inside a larger string (for example ``"Bearer secret:/..."``) is not + supported; after trimming surrounding whitespace, the field value must be + the URI. + + The resolved value is the stored opaque string. The module does not parse + stored data as JSON and does not support field-level URI selection. If a + caller stores structured data, it must encode and decode that structure + itself. + +.. _mgr-ceph-secrets-epoch: + +Epoch-based change detection +---------------------------- + +Consumers that maintain a local cache of secrets can use the epoch to avoid +unnecessary full refreshes: + +.. code-block:: python + + def maybe_refresh(client, namespace, last_epoch): + current_epoch = client.secret_get_epoch(namespace) + if current_epoch == last_epoch: + return last_epoch # nothing changed + # ... do a full refresh ... + return current_epoch + + last_epoch = 0 + last_epoch = maybe_refresh(client, "cephadm", last_epoch) + +The epoch is incremented by every successful ``set`` operation and by +``rm`` only when an existing secret is actually removed (an idempotent +``rm`` returning ``"not_found"`` does not bump the epoch). It starts at 0 +for a namespace that has never been written to, and mutations in one namespace +never affect another namespace's epoch. + + +Configuration +============= + +.. mgr_module:: ceph_secrets +.. confval:: secrets_backend + + :type: str + :default: ``mon`` + + The storage backend used for secrets. Currently only the Mon KV store + (``mon``) is supported. This option is reserved for future backends + (e.g. HashiCorp Vault). diff --git a/doc/mgr/index.rst b/doc/mgr/index.rst index e3c9f688204..25b629dbb80 100644 --- a/doc/mgr/index.rst +++ b/doc/mgr/index.rst @@ -47,5 +47,6 @@ sensible. MDS Autoscaler module NFS module SMB module + Secrets module Progress Module CLI API Commands module