And add miscellaneous clarity / wording improvements.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
High Availability Authentication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The ``cephx`` authentication system is used by Ceph to authenticate users and
-daemons and to protect against man-in-the-middle attacks.
-
-.. note:: The ``cephx`` protocol does not address data encryption in transport
+The CephX authentication system is used by Ceph to authenticate users and
+daemons and to protect against man-in-the-middle attacks. Simultaneously, CephX
+provides a mechanism to grant authorizations to clients for using Ceph
+services. These authorizations are encoded as CephX Capabilities which are
+interpreted by the respective services as to what they grant the client to do.
+The authorizations take the form of time-limited tickets which must be
+periodically refreshed by the Ceph client.
+
+.. note:: The CephX protocol does not address data encryption in transport
(for example, SSL/TLS) or encryption at rest.
-``cephx`` uses shared secret keys for authentication. This means that both the
-client and the monitor cluster keep a copy of the client's secret key.
+CephX uses shared secret keys for authentication. This means that both the
+client and the Monitors keep a copy of the client's secret key.
-The ``cephx`` protocol makes it possible for each party to prove to the other
-that it has a copy of the key without revealing it. This provides mutual
-authentication and allows the cluster to confirm (1) that the user has the
-secret key and (2) that the user can be confident that the cluster has a copy
-of the secret key.
+The CephX protocol makes it possible for each party to prove to the other
+that it has a copy of the key without revealing it while preventing an
+adversary to learn the key or replay messages to masquerade key ownership.
As stated in :ref:`Scalability and High Availability
<arch_scalability_and_high_availability>`, Ceph does not have any centralized
centralized interface, Ceph avoids the bottlenecks that attend such centralized
interfaces. However, this means that clients must interact directly with OSDs.
Direct interactions between Ceph clients and OSDs require authenticated
-connections. The ``cephx`` authentication system establishes and sustains these
+connections. The CephX authentication system establishes and sustains these
authenticated connections.
-The ``cephx`` protocol operates in a manner similar to `Kerberos`_.
-
-A user invokes a Ceph client to contact a monitor. Unlike Kerberos, each
-monitor can authenticate users and distribute keys, which means that there is
-no single point of failure and no bottleneck when using ``cephx``. The monitor
-returns an authentication data structure that is similar to a Kerberos ticket.
-This authentication data structure contains a session key for use in obtaining
-Ceph services. The session key is itself encrypted with the user's permanent
-secret key, which means that only the user can request services from the Ceph
-Monitors. The client then uses the session key to request services from the
-monitors, and the monitors provide the client with a ticket that authenticates
-the client against the OSDs that actually handle data. Ceph Monitors and OSDs
-share a secret, which means that the clients can use the ticket provided by the
-monitors to authenticate against any OSD or metadata server in the cluster.
-
-Like Kerberos tickets, ``cephx`` tickets expire. An attacker cannot use an
+The CephX protocol operates in a manner similar to `Kerberos`_. A user invokes
+a Ceph client which automatically contacts a Monitor. Like Kerberos, each
+Monitor can independently authenticate users and distribute tickets to clients.
+Consequently, there is no single point of failure and no bottleneck when using
+CephX.
+
+Like Kerberos, a Monitor responding to a client authentication request will
+return an session key encrypted by the credential secret and a set of encrypted
+tickets for use by the client to access Ceph services which its credential is
+authorized to use. The session key and auth service ticket together can be used
+for obtaining new or refreshed tickets for Ceph services. For each Ceph service
+(AUTH, MON, MGR, OSD, MDS), a client might receive a ticket for each of the
+rotating service keys (usually 3).
+
+CephX uses rotating service keys which are securely distributed to each service
+type, e.g. the CephFS Metadata Servers (MDS). The tickets encrypted with those
+rotating service keys can then be decrypted by any service daemon to learn the
+identity of the client (entity name), the client's global identifier or
+incarnation, the time when the ticket was created and will expire, and all
+capabilities (authorizations) the Client has with the service.
+
+Like Kerberos tickets, CephX tickets expire. An attacker cannot use an
expired ticket or session key that has been obtained surreptitiously. This form
of authentication prevents attackers who have access to the communications
medium from creating bogus messages under another user's identity and prevents
attackers from altering another user's legitimate messages, as long as the
user's secret key is not divulged before it expires.
-An administrator must set up users before using ``cephx``. In the following
-diagram, the ``client.admin`` user invokes ``ceph auth get-or-create-key`` from
-the command line to generate a username and secret key. Ceph's ``auth``
-subsystem generates the username and key, stores a copy on the monitor(s), and
-transmits the user's secret back to the ``client.admin`` user. This means that
-the client and the monitor share a secret key.
+CephX also supports rotating a credential's secret key to address leaks
+or scope changes when desired. It is expected to be routine in future
+Ceph installations (2025+) that the service daemon keys (e.g. ``mgr.x``)
+might be rotated whenever the daemon is relocated or even restarted.
+The same can be done for any entity type, including clients. Finally, this
+mechanism can be used to effect key type upgrades as necessary when ciphers
+are upgraded to improve security.
+
+To use CephX, an administrator must set up each user/credential in advance. The
+first credential routinely created for new Ceph installs is the
+``client.admin`` key which can be used to administer the cluster. In the
+following diagram, that ``client.admin`` user may invoke ``ceph auth
+get-or-create-key`` from the command line to generate a new entity and secret
+key. Ceph's ``auth`` subsystem generates the username and key, stores a copy on
+all Monitors, and returns the new entity's secret back to administrator.
.. note:: The ``client.admin`` user must provide the user ID and
secret key to the user in a secure manner.
| transmit key |
| |
-Here is how a client authenticates with a monitor. The client passes the user
-name to the monitor. The monitor generates a session key that is encrypted with
-the secret key associated with the ``username``. The monitor transmits the
-encrypted ticket to the client. The client uses the shared secret key to
-decrypt the payload. The session key identifies the user, and this act of
-identification will last for the duration of the session. The client requests
-a ticket for the user, and the ticket is signed with the session key. The
-monitor generates a ticket and uses the user's secret key to encrypt it. The
-encrypted ticket is transmitted to the client. The client decrypts the ticket
-and uses it to sign requests to OSDs and to metadata servers in the cluster.
+
+Here is how a client authenticates with a Monitor with entity
+``client.username``. The client passes the user name to the Monitor. The
+Monitor generates an **auth session key** that is encrypted with the secret key
+associated with ``client.username``. The Monitor transmits the encrypted
+``auth`` ticket to the client. The client uses its principal's secret key to
+decrypt the payload. The newly established **auth session key** now identifies
+the user and will persist for the duration of its instance (incarnation).
+
+When the client requests a service ticket, the Monitor will generate new
+tickets with the latest rotating service secrets for the requested service
+type. Within each ticket is a fresh **service session key** only for use with
+that ticket. The **service session key** is encrypted twice: once using the
+**auth session key** and a second time within the encrypted service ticket
+which can only be decrypted by the service daemon. This new **service session
+key** thereby allows the client and service daemon to mutually authenticate
+each other when establishing a session.
+
.. ditaa::
|<----+ |
-The ``cephx`` protocol authenticates ongoing communications between the clients
+The CephX protocol authenticates ongoing communications between the clients
and Ceph daemons. After initial authentication, each message sent between a
-client and a daemon is signed using a ticket that can be verified by monitors,
+client and a daemon is signed using a ticket that can be verified by Monitors,
OSDs, and metadata daemons. This ticket is verified by using the secret shared
between the client and the daemon.
This authentication protects only the connections between Ceph clients and Ceph
daemons. The authentication is not extended beyond the Ceph client. If a user
-accesses the Ceph client from a remote host, cephx authentication will not be
+accesses the Ceph client from a remote host, CephX authentication will not be
applied to the connection between the user's host and the client host.
-See :ref:`rados-cephx-config-ref` for more on configuration details.
+See :ref:`rados-cephx-config-ref` for more on configuration and operational
+details for CephX.
-See :ref:`user-management` for more on user management.
+See :ref:`user-management` for more on user management and service
+authorizations (CephX Capabilities).
-See :ref:`A Detailed Description of the Cephx Authentication Protocol
+See :ref:`A Detailed Description of the CephX Authentication Protocol
<cephx_2012_peter>` for more on the distinction between authorization and
-authentication and for a step-by-step explanation of the setup of ``cephx``
+authentication and for a step-by-step explanation of the setup of CephX
tickets and session keys.
.. index:: architecture; smart daemons and scalability
means limiting its cache size.
+.. _manual-mds:
+
Adding an MDS
=============
-#. Create an mds directory ``/var/lib/ceph/mds/ceph-${id}``. The daemon only uses this directory to store its keyring.
-#. Create the authentication key, if you use CephX: ::
+In the below instructions, ``{id}`` is an arbitrary name, such as the hostname of the machine.
+
+#. Create the mds data directory.
+
+ .. code:: bash
+
+ mkdir -p /var/lib/ceph/mds/{cluster-name}-{id}
+
+#. Create and import the keyring.
+
+ .. code:: bash
+
+ ceph auth get-or-create mds.{id} mon "allow profile mds" mgr "allow profile mds" osd "allow rw tag cephfs *=*" mds "allow" > /var/lib/ceph/mds/{cluster-name}-{id}/keyring
+
+#. Start the daemon the manual way.
+
+ .. code:: bash
+
+ ceph-mds --cluster {cluster-name} -i {id} -m {mon-hostname}:{mon-port} [-f]
+
+#. Alternatively, start the daemon using a service manager.
+
+ .. code:: bash
+
+ sudo systemctl start ceph-mds@${id}
+
+#. If starting the daemon fails with this error:
- $ sudo ceph auth get-or-create mds.${id} mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-${id}/keyring
+ ::
-#. Start the service: ::
+ mds.-1.0 ERROR: failed to authenticate: (22) Invalid argument
- $ sudo systemctl start ceph-mds@${id}
+ Then make sure you do not have a keyring set in ceph.conf in the global
+ section; move it to the client section; or add a keyring setting specific to
+ this mds daemon. And verify that you see the same key in the mds data
+ directory and ``ceph auth get mds.{id}`` output.
-#. The status of the cluster should show: ::
+#. Optionally, configure the file system the MDS should join (:ref:`mds-join-fs`):
- mds: ${id}:1 {0=${id}=up:active} 2 up:standby
+ ::
-#. Optionally, configure the file system the MDS should join (:ref:`mds-join-fs`): ::
+ $ ceph config set mds.${id} mds_join_fs ${fs}
- $ ceph config set mds.${id} mds_join_fs ${fs}
+#. Now you are ready to :ref:`create-fs`.
Removing an MDS
+.. _create-fs:
+
=========================
Create a Ceph file system
=========================
$ ceph fs ls # Pool should now show up
.... data pools: [cephfs_data cephfs_data_ssd ]
-Make sure that your cephx keys allows the client to access this new pool.
+Make sure that your CephX keys allows the client to access this new pool.
You can then update the layout on a directory in CephFS to use the pool you added:
Terminology
-----------
-``{foo, bar}^secret`` denotes encryption by secret.
+``{X, Y, ...}^secret`` denotes encryption of values ``X + Y + ...`` by ``secret``.
Context
CephXAuthenticate {
u8 2 # 2 means nautilus+
u64 client_challenge # random (by client)
- u64 key = {client_challenge ^ server_challenge}^principal_secret # (roughly)
+ u64 key = digest({client_challenge, server_challenge}^principal_secret) # (roughly)
blob old_ticket # old ticket, if we are reconnecting or renewing
u32 other_keys # bit mask of service keys we want
}
CephXAuthenticate {
u8 1 # 2 means nautilus+
u64 client_challenge # random (by client)
- u64 key = {client_challenge + server_challenge}^principal_secret # (roughly)
+ u64 key = digest({client_challenge, server_challenge}^principal_secret) # (roughly)
blob old_ticket # old ticket, if we are reconnecting or renewing
}
-The monitor looks up principal_secret in database, and verifies the
-key is correct. If old_ticket is present, verify it is valid, and we
-can reuse the same global_id. (Otherwise, a new global_id is assigned
-by the monitor.)::
+.. note:: This is not an authentication step in this protocol. The
+ ``server_challenge`` is vulnerable to MitM attacks. This can be
+ considered a DoS filter and little else.
+
+The monitor looks up ``principal_secret`` in database, and verifies the key is
+correct. If ``old_ticket`` is present, verify that it is valid and reuse the
+same ``global_id``. Otherwise, the monitors assign a new ``global_id``::
a->p :
CephxReplyHeader {
}
u8 encoding_version = 1
u32 num_tickets ( = 1)
- ticket_info # (N = 1)
+ ticket_info # (N = 1) for AUTH ticket
plus (for Nautilus and later)::
}
service_ticket_info {
- u32 service_id # CEPH_ENTITY_TYPE_{OSD,MDS,MGR}
+ u32 service_id # CEPH_ENTITY_TYPE_{MON,MGR,OSD,MDS}
u8 msg_version (1)
- {CephXServiceTicket service_ticket}^principal_secret
+ {CephXServiceTicket service_ticket}^session_key
CephxTicketBlob ticket_blob
}
CephxTicketBlob {
u64 secret_id # which service ticket encrypted this; -1 == monsecret, otherwise service's rotating key id
- {CephXServiceTicketInfo ticket}^mon_secret
+ {CephxServiceTicketInfo ticket}^rotating_service_secret
}
CephxServiceTicketInfo {
u32 flags = 0 # unused
}
+.. note:: The rotating_service_secret is one of three (by default) secrets that
+ services of the given type share. The Monitors have two service types for
+ this purpose: CEPH_ENTITY_TYPE_MON for protecting mon authorizations and
+ CEPH_ENTITY_TYPE_AUTH for securing the "auth" tickets. The "auth" tickets
+ permit a client to reacquire an assigned global_id.
+
So: for each ticket, principal gets a part that it decrypts with its
secret to get the session_key (CephxServiceTicket). And the
CephxTicketBlob is opaque (secured by the mon secret) but can be used
u64 server_challenge_plus_one = 0 # not used here
}
-The monitor validates the authorizer by decrypting the auth_ticket
-with ``mon_secret`` and confirming that it says this principal is who
-they say they are in the CephxAuthorizer fields. Note that the nonce
-random bytes aren't used here (the field exists for Phase III below).
+The monitor validates the authorizer by decrypting the auth_ticket with
+``rotating_service_secret`` and confirming that it says this principal is who
+they say they are in the CephxAuthorizer fields. Note that the nonce random
+bytes aren't used here (the field exists for Phase III below).
Assuming all is well, the authorizer can generate service tickets
based on the CEPH_ENTITY_TYPE_* bits in the ``keys`` bitmask.
In order to initialize a new monitor, it must always be fed:
#. a logical name
-#. secret keys
+#. The ``mon.`` secret key. Note this is not qualified with an id since all Monitors share the same secret.
#. a cluster fsid (uuid)
In addition, a monitor needs to know two things:
name will be ``mon.foo``.
For most users, there is no more than one monitor per host, which
-makes the short hostname logical choice.
+makes the short hostname a logical choice.
Secret keys
===========
The resulting keyring is fed to ``ceph-mon --mkfs`` with the ``--keyring <keyring>`` command-line argument.
+.. note:: The Monitor keyring file is a fallback store for the KeyServer used
+ to store keys via the usual `ceph auth` command suite. For this
+ reason, the ``client.admin`` key may be placed in this keyring but a future
+ rotation or update to the ``client.admin`` credential may obviate it.
+
Cluster fsid
============
ceph-mon -i <myid> --mon-initial-members foo,bar,baz
to prevent any risk of split-brain.
-
-
-
-
-
CephX
The Ceph authentication protocol. CephX authenticates users and
daemons. CephX operates like Kerberos, but it has no single
- point of failure. See the :ref:`High-availability
- Authentication section<arch_high_availability_authentication>`
- of the Architecture document and the :ref:`CephX Configuration
+ point of failure and allows authentication to a class of
+ services. See the :ref:`High-availability Authentication
+ section<arch_high_availability_authentication>` of the
+ Architecture document and the :ref:`CephX Configuration
Reference<rados-cephx-config-ref>`.
Client
+.. _manual-deployment:
+
===================
Manual Deployment
===================
secret key. You must generate a keyring with a monitor secret and provide
it when bootstrapping the initial monitor(s).
+* **Configure Cephx**: The monmap will use the most secure defaults but
+ you may want to relax the choice in ciphers to support legacy clients.
+
- **Administrator Keyring**: To use the ``ceph`` CLI tools, you must have
a ``client.admin`` user. So you must generate the admin user and keyring,
and you must also add the ``client.admin`` user to the monitor keyring.
The procedure is as follows:
-#. Log in to the initial monitor node(s)::
+#. Log in to the initial monitor node(s):
- ssh {hostname}
+ .. code:: bash
- For example::
+ ssh {hostname}
+
+ For example:
+
+ .. code:: bash
- ssh mon-node1
+ ssh mon-node1
#. Ensure you have a directory for the Ceph configuration file. By default,
Ceph uses ``/etc/ceph``. When you install ``ceph``, the installer will
- create the ``/etc/ceph`` directory automatically. ::
+ create the ``/etc/ceph`` directory automatically.
- ls /etc/ceph
+ .. code:: bash
+
+ ls /etc/ceph
#. Create a Ceph configuration file. By default, Ceph uses
``ceph.conf``, where ``ceph`` reflects the cluster name. Add a line
- containing "[global]" to the configuration file. ::
+ containing "[global]" to the configuration file.
- sudo vim /etc/ceph/ceph.conf
+ .. code:: bash
+ sudo vim /etc/ceph/ceph.conf
-#. Generate a unique ID (i.e., ``fsid``) for your cluster. ::
- uuidgen
+#. Generate a unique ID (i.e., ``fsid``) for your cluster.
+ .. code:: bash
-#. Add the unique ID to your Ceph configuration file. ::
+ uuidgen
- fsid = {UUID}
- For example::
+#. Add the unique ID to your Ceph configuration file.
- fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
+ .. code::
+ fsid = {UUID}
-#. Add the initial monitor(s) to your Ceph configuration file. ::
+ For example:
- mon_initial_members = {hostname}[,{hostname}]
+ .. code::
+
+ fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
- For example::
- mon_initial_members = mon-node1
+#. Add the initial monitor(s) to your Ceph configuration file.
+
+ .. code::
+
+ mon_initial_members = {hostname}[,{hostname}]
+
+ For example:
+
+ .. code::
+
+ mon_initial_members = mon-node1
#. Add the IP address(es) of the initial monitor(s) to your Ceph configuration
- file and save the file. ::
+ file and save the file.
- mon_host = {ip-address}[,{ip-address}]
+ .. code::
- For example::
+ mon_host = {ip-address}[,{ip-address}]
+
+ For example
+
+ .. code::
- mon_host = 192.168.0.1
+ mon_host = 192.168.0.1
- **Note:** You may use IPv6 addresses instead of IPv4 addresses, but
- you must set ``ms_bind_ipv6`` to ``true``. See `Network Configuration
- Reference`_ for details about network configuration.
+ .. note:: You may use IPv6 addresses instead of IPv4 addresses, but you must set ``ms_bind_ipv6`` to ``true``. See `Network Configuration Reference`_ for details about network configuration.
-#. Create a keyring for your cluster and generate a monitor secret key. ::
+#. Create a keyring for your cluster and generate a monitor secret key.
- sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
+ .. code:: bash
+ sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon.
+
+ .. note:: The ``mon.`` credential does not require any capabilities. All Monitors share this single key.
#. Generate an administrator keyring, generate a ``client.admin`` user and add
- the user to the keyring. ::
+ the user to the keyring.
+
+ .. code:: bash
- sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
+ sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
#. Generate a bootstrap-osd keyring, generate a ``client.bootstrap-osd`` user and add
- the user to the keyring. ::
+ the user to the keyring.
+
+ .. code:: bash
+
+ sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
+
+#. Add the generated keys to the ``ceph.mon.keyring``.
+
+ .. code:: bash
+
+ sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
+ sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
+
+#. Change the owner for ``ceph.mon.keyring``.
+
+ .. code:: bash
+
+ sudo chown ceph:ceph /tmp/ceph.mon.keyring
+
+#. Configure CephX for cluster
+
+ .. note:: This step can also be done after cluster creation.
+
+ If you need to support legacy clients authenticating with the cluster, you will want to allow older insecure ciphers:
+
+ .. code:: bash
+
+ AUTH_SETTINGS="--auth-allowed-ciphers=aes,aes256k"
+
+ If you also want new keys to use the legacy (and insecure) ``aes`` cipher by default:
- sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
+ .. code:: bash
-#. Add the generated keys to the ``ceph.mon.keyring``. ::
+ AUTH_SETTINGS="$AUTH_SETTINGS --auth-preferred-cipher=aes"
- sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
- sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
+ The service cipher type can also be configured but this should generally not need to be done unless an older version of a service daemon needs to run in the cluster:
-#. Change the owner for ``ceph.mon.keyring``. ::
+ .. code:: bash
- sudo chown ceph:ceph /tmp/ceph.mon.keyring
+ AUTH_SETTINGS="$AUTH_SETTINGS --auth-service-cipher=aes"
+
+ .. note:: Clients do not and cannot decrypt the service cipher.
#. Generate a monitor map using the hostname(s), host IP address(es) and the FSID.
- Save it as ``/tmp/monmap``::
+ Save it as ``/tmp/monmap``:
- monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap
+ .. code:: bash
- For example::
+ monmaptool --create $AUTH_SETTINGS --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap
- monmaptool --create --add mon-node1 192.168.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap
+ For example:
+ .. code:: bash
-#. Create a default data directory (or directories) on the monitor host(s). ::
+ monmaptool --create $AUTH_SETTINGS --add mon-node1 192.168.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap
- sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}
- For example::
+#. Create a default data directory (or directories) on the monitor host(s).
+
+ .. code:: bash
- sudo -u ceph mkdir /var/lib/ceph/mon/ceph-mon-node1
+ sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}
+
+ For example:
+
+ .. code:: bash
+
+ sudo -u ceph mkdir /var/lib/ceph/mon/ceph-mon-node1
See `Monitor Config Reference - Data`_ for details.
-#. Populate the monitor daemon(s) with the monitor map and keyring. ::
+#. Populate the monitor daemon(s) with the monitor map and keyring.
- sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
+ .. code:: bash
- For example::
+ sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
- sudo -u ceph ceph-mon --mkfs -i mon-node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
+ For example:
+ .. code:: bash
-#. Consider settings for a Ceph configuration file. Common settings include
- the following::
+ sudo -u ceph ceph-mon --mkfs -i mon-node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
- [global]
- fsid = {cluster-id}
- mon_initial_members = {hostname}[, {hostname}]
- mon_host = {ip-address}[, {ip-address}]
- public_network = {network}[, {network}]
- cluster_network = {network}[, {network}]
- auth_cluster required = cephx
- auth_service required = cephx
- auth_client required = cephx
- osd_pool_default_size = {n} # Write an object n times.
- osd_pool_default_min_size = {n} # Allow writing n copies in a degraded state.
- osd_pool_default_pg_num = {n}
- osd_crush_chooseleaf_type = {n}
+#. Consider minimal settings for a Ceph configuration file. Common settings include
+ the following:
+
+ .. code::
+
+ [global]
+ fsid = {cluster-id}
+ mon_initial_members = {hostname}[, {hostname}]
+ mon_host = {ip-address}[, {ip-address}]
+ public_network = {network}[, {network}]
+ cluster_network = {network}[, {network}]
In the foregoing example, the ``[global]`` section of the configuration might
- look like this::
+ look like this:
- [global]
- fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
- mon_initial_members = mon-node1
- mon_host = 192.168.0.1
- public_network = 192.168.0.0/24
- auth_cluster_required = cephx
- auth_service_required = cephx
- auth_client_required = cephx
- osd_pool_default_size = 3
- osd_pool_default_min_size = 2
- osd_pool_default_pg_num = 333
- osd_crush_chooseleaf_type = 1
+ .. code::
+ [global]
+ fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
+ mon_initial_members = mon-node1
+ mon_host = 192.168.0.1
+ public_network = 192.168.0.0/24
-#. Start the monitor(s).
+ .. note:: Your preference as an operator should be to effect configuration changes through the ``ceph config`` API rather than in the ``ceph.conf`` file. See also :ref:`configuring-ceph-api`.
- Start the service with systemd::
+#. Start the monitor(s) with systemd.
- sudo systemctl start ceph-mon@mon-node1
+ .. code:: bash
+
+ sudo systemctl start ceph-mon@mon-node1
#. Ensure to open firewall ports for ceph-mon.
- Open the ports with firewalld::
+ Open the ports with firewalld:
+
+ .. code:: bash
- sudo firewall-cmd --zone=public --add-service=ceph-mon
- sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent
+ sudo firewall-cmd --zone=public --add-service=ceph-mon
+ sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent
-#. Verify that the monitor is running. ::
- sudo ceph -s
+#. Verify that the monitor is running.
+
+ .. code:: bash
+
+ sudo ceph -s
You should see output that the monitor you started is up and running, and
you should see a health error indicating that placement groups are stuck
- inactive. It should look something like this::
+ inactive. It should look something like this:
+
+ ::
cluster:
id: a7f64266-0894-4f1e-a635-d0aeaca0e993
pgs:
- **Note:** Once you add OSDs and start them, the placement group health errors
- should disappear. See `Adding OSDs`_ for details.
+ .. note:: Once you add OSDs and start them, any placement group health errors should disappear. See `Adding OSDs`_ for details.
+
+ .. warning:: If you have enabled legacy cipher types then the Monitors may raise health warnings. You may mute (see :ref:`health-mute`) the warnings. See the warning descriptions in :ref:`health-checks` for more information on the different warnings.
Manager daemon configuration
============================
Adding MDS
==========
-In the below instructions, ``{id}`` is an arbitrary name, such as the hostname of the machine.
-
-#. Create the mds data directory.::
-
- mkdir -p /var/lib/ceph/mds/{cluster-name}-{id}
-
-#. Create a keyring.::
-
- ceph-authtool --create-keyring /var/lib/ceph/mds/{cluster-name}-{id}/keyring --gen-key -n mds.{id}
-
-#. Import the keyring and set caps.::
-
- ceph auth add mds.{id} osd "allow rwx" mds "allow *" mon "allow profile mds" -i /var/lib/ceph/mds/{cluster}-{id}/keyring
-
-#. Add to ceph.conf.::
-
- [mds.{id}]
- host = {id}
-
-#. Start the daemon the manual way.::
-
- ceph-mds --cluster {cluster-name} -i {id} -m {mon-hostname}:{mon-port} [-f]
-
-#. Start the daemon the right way (using ceph.conf entry).::
-
- service ceph start
-
-#. If starting the daemon fails with this error::
-
- mds.-1.0 ERROR: failed to authenticate: (22) Invalid argument
-
- Then make sure you do not have a keyring set in ceph.conf in the global section; move it to the client section; or add a keyring setting specific to this mds daemon. And verify that you see the same key in the mds data directory and ``ceph auth get mds.{id}`` output.
-
-#. Now you are ready to `create a Ceph file system`_.
+Please see the section on manual deployment in :ref:`manual-mds`.
Manually Installing RADOSGW
===========================
.. _Add/Remove OSDs: ../../rados/operations/add-or-rm-osds
.. _Network Configuration Reference: ../../rados/configuration/network-config-ref
.. _Monitor Config Reference - Data: ../../rados/configuration/mon-config-ref#data
-.. _create a Ceph file system: ../../cephfs/createfs
Usage::
- ceph auth add <entity> {<caps> [<caps>...]}
+ ceph auth add [--key-type=<cipher>] <entity> {<caps> [<caps>...]}
+
+No key information is output. If ``--key-type`` is omitted, the cluster's perferred cipher is used.
Subcommand ``caps`` updates caps for **name** from caps specified in the command.
Usage::
- ceph auth get-or-create <entity> {<caps> [<caps>...]}
+ ceph auth get-or-create [--key-type=<cipher>] <entity> {<caps> [<caps>...]}
+
+The output is the new key in the ``.ini`` file format. If ``--key-type`` is omitted, the cluster's perferred cipher is used.
Subcommand ``get-or-create-key`` gets or adds key for ``name`` from system/caps
pairs specified in the command. If key already exists, any given caps must match
ceph auth get-or-create-key <entity> {<caps> [<caps>...]}
+The output is the new key in plain text. If ``--key-type`` is omitted, the cluster's perferred cipher is used.
+
Subcommand ``import`` reads keyring from input file.
Usage::
ceph auth print_key <entity>
+Subcommand ``rotate`` rotates the key for the given entity.
+
+Usage::
+
+ ceph auth rotate [--key-type=<cipher>] <entity>
+
+The output is the new key in the ``.ini`` file format. If ``--key-type`` is omitted, the cluster's perferred cipher is used.
+
+Subcommand ``dump-keys`` dumps the entire key database including rotating service keys.
+
+Usage::
+
+ ceph --format=json auth dump-keys
+
+The output format must be ``json`` or ``json-pretty``.
+
+Subcommand ``wipe-rotating-service-keys`` wipes all rotating service keys and forces the Monitors to refresh all keys. **Do not** run this command without understanding its full effects and purpose. Consult the Ceph documentation for more information.
+
+Usage::
+
+ ceph auth wipe-rotating-service-keys
+
+
compact
-------
Usage::
- ceph fs authorize <fs_name> client.<client_id> <path> <perms> [<path> <perms>...]
+ ceph fs authorize [--key-type=<cipher>] <fs_name> client.<client_id> <path> <perms> [<path> <perms>...]
+
+The output is the new key in the ``.ini`` file format with caps. If ``--key-type`` is omitted, the cluster's perferred cipher is used.
Subcommand ``dump`` displays the FSMap at the given epoch (default: current).
This includes all file system settings, MDS daemons and the ranks they hold
ceph mon stat
+Subcommand ``set`` sets various Monitor map settings.
+
+Usage::
+
+ ceph mon set <name> <value>
+
+Valid names include ``auth_service_cipher``, ``auth_allowed_ciphers``, and
+``auth_preferred_cipher``. The meaning of these settings and the valid values
+are as described in :manpage:`monmaptool(8)`.
+
+
mgr
---
Subcommand ``new`` can be used to create a new OSD or to recreate a previously
destroyed OSD with a specific *id*. The new OSD will have the specified *uuid*,
-and the command expects a JSON file containing the base64 cephx key for auth
+and the command expects a JSON file containing the base64 CephX key for auth
entity *client.osd.<id>*, as well as optional base64 cepx key for dm-crypt
lockbox access and a dm-crypt key. Specifying a dm-crypt requires specifying
-the accompanying lockbox cephx key.
+the accompanying lockbox CephX key.
Usage::
ceph osd rm <ids> [<ids>...]
-Subcommand ``destroy`` marks OSD *id* as *destroyed*, removing its cephx
+Subcommand ``destroy`` marks OSD *id* as *destroyed*, removing its CephX
entity's keys and all of its dm-crypt and daemon-private config key
entries.
set the min_mon_release.
+.. option:: --auth-service-cipher=<cipher>
+
+ set the cipher type for rotating service keys. The default is "aes256k".
+
+.. option:: --auth-allowed-ciphers=<cipher1,cipher2,...>
+
+ set the cipher types permitted for authentication by daemons (including Monitors) or clients. The default is "aes256k".
+
+.. option:: --auth-preferred-cipher=<cipher>
+
+ set the preferred cipher key type for new credentials or for key rotations. The default is "aes256k".
+
+
Example
=======
.. option:: --id ID
- Set ID portion of my name.
+ Set ID portion of a client entity. e.g. admin for client.admin
.. option:: -n TYPE.ID, --name TYPE.ID
- Set cephx user name.
+ Set the full entity name. e.g. client.admin
.. option:: --cluster NAME
rbd snap rm mypool/myimage@mysnap
-To map an image via the kernel with cephx enabled::
+To map an image via the kernel with CephX enabled::
rbd device map mypool/myimage --id admin --keyfile secretfile
install/deploy tools, your admin client should get this capability
automatically. If you use tooling from elsewhere, you may get EACCES errors
when invoking certain ceph cluster commands. To fix that, add a ``mgr allow
-\*`` stanza to your client's cephx capabilities by `Modifying User
+\*`` stanza to your client's CephX capabilities by `Modifying User
Capabilities`_.
High availability
CephX provides has some computational costs, though they should generally be
quite low. If the network environment connecting your client and server hosts
is very safe and you cannot afford authentication, you can disable it.
-**Disabling authentication is not generally recommended**.
-.. note:: If you disable authentication, you will be at risk of a
- man-in-the-middle attack that alters your client/server messages, which
- could have disastrous security effects.
+.. warning:: Disabling authentication is usually a very bad choice. If you
+ disable authentication, any access to the cluster will be
+ permitted no matter the origin or identity of the client.
For information about creating users, see `User Management`_. For details on
the architecture of CephX, see `Architecture - High Availability
Manual Deployment
-----------------
-When you deploy a cluster manually, it is necessary to bootstrap the monitors
+When you deploy a cluster manually, it is necessary to bootstrap the Monitors
manually and to create the ``client.admin`` user and keyring. To bootstrap
-monitors, follow the steps in `Monitor Bootstrapping`_. Follow these steps when
+Monitors, follow the steps in `Monitor Bootstrapping`_. Follow these steps when
using third-party deployment tools (for example, Chef, Puppet, and Juju).
Enabling/Disabling CephX
========================
-Enabling CephX is possible only if the keys for your monitors, OSDs, and
-metadata servers have already been deployed. If you are simply toggling CephX
-on or off, it is not necessary to repeat the bootstrapping procedures.
+Enabling CephX is possible only if the keys for your Monitors, OSD, and MDS
+have already been deployed. If you are simply toggling CephX on or off, it is
+not necessary to repeat the bootstrapping procedures.
+
+Authentication is explicitly enabled or disabled for all entities via the
+``global`` section of Ceph configuration. The following configurations affect
+this.
+
+.. confval:: auth_cluster_required
+.. confval:: auth_service_required
+.. confval:: auth_client_required
+
Enabling CephX
--------------
#. Create a ``client.admin`` key, and save a copy of the key for your client
host:
- .. prompt:: bash $
+.. prompt:: bash $
- ceph auth get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *' -o /etc/ceph/ceph.client.admin.keyring
+ ceph auth get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *' -o /etc/ceph/ceph.client.admin.keyring
- **Warning:** This step will clobber any existing
- ``/etc/ceph/client.admin.keyring`` file. Do not perform this step if a
- deployment tool has already generated a keyring file for you. Be careful!
+.. warning:: This step will clobber any existing ``/etc/ceph/client.admin.keyring`` file. Do not perform this step if a deployment tool has already generated a keyring file for you. Be careful!
#. Create a monitor keyring and generate a monitor secret key:
.. code-block:: ini
+ [global]
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
.. code-block:: ini
+ [global]
auth_cluster_required = none
auth_service_required = none
auth_client_required = none
Configuration Settings
======================
-Enablement
-----------
-
-
-``auth_cluster_required``
-
-:Description: If this configuration setting is enabled, the Ceph Storage
- Cluster daemons (that is, ``ceph-mon``, ``ceph-osd``,
- ``ceph-mds``, and ``ceph-mgr``) are required to authenticate with
- each other. Valid settings are ``cephx`` or ``none``.
-
-:Type: String
-:Required: No
-:Default: ``cephx``.
-
-
-``auth_service_required``
-
-:Description: If this configuration setting is enabled, then Ceph clients can
- access Ceph services only if those clients authenticate with the
- Ceph Storage Cluster. Valid settings are ``cephx`` or ``none``.
-
-:Type: String
-:Required: No
-:Default: ``cephx``.
-
-
-``auth_client_required``
-
-:Description: If this configuration setting is enabled, then communication
- between the Ceph client and Ceph Storage Cluster can be
- established only if the Ceph Storage Cluster authenticates
- against the Ceph client. Valid settings are ``cephx`` or
- ``none``.
-
-:Type: String
-:Required: No
-:Default: ``cephx``.
-
-
.. index:: keys; keyring
Keys
path to a keyfile by using the ``keyfile`` setting in the Ceph configuration
file.
-``keyring``
-
-:Description: The path to the keyring file.
-:Type: String
-:Required: No
-:Default: ``/etc/ceph/$cluster.$name.keyring,/etc/ceph/$cluster.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin``
-
-
-``keyfile``
-
-:Description: The path to a keyfile (that is, a file containing only the key).
-:Type: String
-:Required: No
-:Default: None
-
-
-``key``
-
-:Description: The key (that is, the text string of the key itself). We do not
- recommend that you use this setting unless you know what you're
- doing.
-:Type: String
-:Required: No
-:Default: None
+.. confval:: keyring
+.. confval:: keyfile
+.. confval:: key
Daemon Keyrings
Administrative users or deployment tools (for example, ``cephadm``) generate
daemon keyrings in the same way that they generate user keyrings. By default,
-Ceph stores the keyring of a daemon inside that daemon's data directory. The
-default keyring locations and the capabilities that are necessary for the
-daemon to function are shown below.
-
-``ceph-mon``
-
-:Location: ``$mon_data/keyring``
-:Capabilities: ``mon 'allow *'``
+Ceph stores the keyring of a daemon inside that daemon's data directory.
+Consult each components documentation for capabilities expected for the
+service.
-``ceph-osd``
+To bootstrap a cluster, consult the :ref:`manual-deployment` documentation.
-:Location: ``$osd_data/keyring``
-:Capabilities: ``mgr 'allow profile osd' mon 'allow profile osd' osd 'allow *'``
-
-``ceph-mds``
-
-:Location: ``$mds_data/keyring``
-:Capabilities: ``mds 'allow' mgr 'allow profile mds' mon 'allow profile mds' osd 'allow rwx'``
-
-``ceph-mgr``
-
-:Location: ``$mgr_data/keyring``
-:Capabilities: ``mon 'allow profile mgr' mds 'allow *' osd 'allow *'``
-
-``radosgw``
-
-:Location: ``$rgw_data/keyring``
-:Capabilities: ``mon 'allow rwx' osd 'allow rwx'``
-
-
-.. note:: The monitor keyring (that is, ``mon.``) contains a key but no
- capabilities, and this keyring is not part of the cluster ``auth`` database.
-
-The daemon's data-directory locations default to directories of the form::
+Each daemon's data-directory locations defaults to a path of the form::
/var/lib/ceph/$type/$cluster-$id
Note that even when signatures are enabled data is not encrypted in flight.
-``cephx_require_signatures``
+.. confval:: cephx_require_signatures
-:Description: If this configuration setting is set to ``true``, Ceph requires
- signatures on all message traffic between the Ceph client and the
- Ceph Storage Cluster, and between daemons within the Ceph Storage
- Cluster.
+.. confval:: cephx_cluster_require_signatures
-.. note::
- **ANTIQUATED NOTE:**
+.. confval:: cephx_service_require_signatures
- Neither Ceph Argonaut nor Linux kernel versions prior to 3.19
- support signatures; if one of these clients is in use, ``cephx_require_signatures``
- can be disabled in order to allow the client to connect.
+.. confval:: cephx_sign_messages
-:Type: Boolean
-:Required: No
-:Default: ``false``
+Time to Live
+------------
+.. confval:: auth_mon_ticket_ttl
+.. confval:: auth_service_ticket_ttl
-``cephx_cluster_require_signatures``
-:Description: If this configuration setting is set to ``true``, Ceph requires
- signatures on all message traffic between Ceph daemons within the
- Ceph Storage Cluster.
-:Type: Boolean
-:Required: No
-:Default: ``false``
+.. _cephx-upgrade:
+Upgrading and Rotating CephX Keys
+=================================
-``cephx_service_require_signatures``
+In 2025, it became necessary to upgrade the cipher key type for all CephX keys
+due to the vulnerabilities in the older encryption schemes (see
+`CVE-2025-30156 `_). To effect this upgrade, it's necessary to do the
+upgrade in several steps.
-:Description: If this configuration setting is set to ``true``, Ceph requires
- signatures on all message traffic between Ceph clients and the
- Ceph Storage Cluster.
+.. _CVE-2025-30156: https://github.com/ceph/ceph/security/advisories/GHSA-7q3q-3975-qw3q
-:Type: Boolean
-:Required: No
-:Default: ``false``
+.. note:: cephadm and Rook automate this process for you. cephadm however does not handle the client key changes.
-``cephx_sign_messages``
+#. **Allow the newer key types for authentication.** For upgraded clusters, this should be done automatically by the Monitors.
-:Description: If this configuration setting is set to ``true``, and if the Ceph
- version supports message signing, then Ceph will sign all
- messages so that they are more difficult to spoof.
+ Confirm the upgrade:
-:Type: Boolean
-:Default: ``true``
+ .. code:: bash
+ ceph --format=json mon dump | jq -r '.auth_allowed_ciphers | map(.name) | join (",")'
-Time to Live
-------------
+ should output something like:
+
+ ::
+
+ aes,aes256k
+
+ where ``aes256k`` is the new more secure cipher type.
+
+ If not included, you can explicitly enable it:
+
+ .. code:: bash
+
+ ceph mon set auth_allowed_ciphers aes,aes256k
+
+ Then confirm the change:
+
+ .. code:: bash
+
+ ceph --format=json mon dump | jq -r '.auth_allowed_ciphers | map(.name) | join (",")'
+
+
+#. **Set the preferred default cipher type for new keys.** You may choose **not** to do this if you want new keys, by default, to use the older cipher type until your client applications can be upgraded.
+
+ Check the current value:
+
+ .. code:: bash
+
+ ceph --format=json mon dump | jq -r '.auth_preferred_cipher.name'
+
+ might output:
+
+ ::
+
+ aes
+
+ To upgrade to ``aes256k`` as the new default cipher type, execute:
+
+ .. code:: bash
+
+ ceph mon set auth_preferred_cipher aes256k
+
+ Confirm the change:
+
+ .. code:: bash
+
+ ceph --format=json mon dump | jq -r '.auth_preferred_cipher.name'
+
+ should output:
+
+ ::
+
+ aes256k
+
+
+#. **Rotate the keys for all service daemon credentials.** These include **mon**, **mgr**, **osd**, and **mds**.
+
+ .. warning:: Changing the key will make the existing daemon unable to reauthenticate.
+
+ .. note:: The ``mon.`` historically has not been managed by the Monitor auth database; it exists soley in each Monitor's keyring inside its data directory. This suggested rotation procedure now puts the authoritative copy in the auth database alongside other keys. The Monitor keyring persists as a fallback or emergency key.
+
+ Begin with the ``mon.`` key:
+
+ .. code:: bash
+
+ ceph auth rotate --key-type=aes256k mon. | tee mon.keyring
+
+ Save the ``mon.keyring`` file in a safe place. It should **not** be necessary to update the keyring files for each Monitor.
+
+ Restart each Monitor:
+
+ .. code:: bash
+
+ systemctl restart ceph-mon@$ID
+
+ .. warning:: If a Monitor was out-of-quorum during the Monitor key rotation, it will not have the new key. You must put the saved ``mon.keyring`` in its keyring file so it can authenticate.
+
+
+ Now, for each other service daemon type (**mgr**, **osd**, and **mds**):
+
+ Stop the daemon:
+
+ .. code:: bash
+
+ systemctl stop ceph-$TYPE@$ID
+
+ If it is an OSD:
+
+ .. code:: bash
+
+ ceph osd down $ID
+
+ Rotate the entity's key:
+
+ .. code:: bash
+
+ ceph auth rotate --key-type=aes256k $TYPE.$ID | tee keyring
+
+ .. note:: If you have updated ``auth_preferred_cipher`` then you can omit ``--key-type``.
+
+ Copy the ``keyring`` file to the machine running the daemon, then execute:
+
+ .. code:: bash
+
+ ceph-authtool --import-keyring $COPIED_KEYRING /var/lib/ceph/$TYPE/ceph-$ID/keyring
+
+ Adjust the above script based on where the data directory for your daemons are located.
+
+ Finally, restart the daemon:
+
+ .. code:: bash
+
+ systemctl restart ceph-$TYPE@$ID
+
+#. **Confirm the** :ref:`auth-insecure-service-key-type` **is cleared.**
+
+ .. code:: bash
+
+ ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_SERVICE_KEY_TYPE") | not'
+
+ output gives ``false``.
+
+ If it outputs ``true``, there is another daemon that needs to be upgraded.
+ Check the output of ``ceph health detail``.
+
+#. **Upgrade the cipher for rotating service keys.**
+
+ .. code:: bash
+
+ ceph mon set auth_service_cipher aes256k
+
+ Confirm the change:
+
+ .. code:: bash
+
+ ceph --format=json mon dump | jq -r '.auth_service_cipher.name'
+
+ should output
+
+ ::
+
+ aes256k
+
+ Verify the :ref:`auth-insecure-service-tickets` is resolved:
+
+ .. code:: bash
+
+ ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_SERVICE_TICKETS") | not'
+
+#. **Wipe the rotating service keys.**
+
+ .. warning:: This **is not recommended** for most deployments. It is best to let your rotating service keys expire after a few hours (using default TTL).
+
+ .. warning:: **Only perform this step if all service daemons have upgraded binaries that understand the new cipher type.**
+
+ If you want to immediately clear the :ref:`auth-insecure-rotating-service-key-type` warning, you can wipe the existing rotating service key database on the Monitors:
+
+ .. code::
+
+ ceph auth wipe-rotating-service-keys
+
+ should output:
+
+ ::
+
+ wiped rotating service keys!
+
+ This will cause all service daemons to refresh the rotating service keys. Upgraded clients will similarly refresh their tickets with the Monitors.
+
+ .. note:: This operation has no effect on the existing sessions the Clients have established with service daemons.
+
+#. **Prevent creation of new insecure keys.**
+
+ When the Monitor setting ``auth_allowed_ciphers`` setting includes an
+ insecure key type, the **default value** of the Monitor config
+ ``mon_auth_allow_insecure_key`` will be altered at runtime to ``true``. For an upgraded
+ cluster, you should therefore expect see the
+ :ref:`auth-insecure-keys-creatable` health warning.
+
+ You can disable this configuration manually to prevent new insecure keys
+ from being created. Alternatively, once the ``auth_allowed_ciphers`` omits
+ insecure key types (in a future step of this process), this configuration
+ will have its default value changed and that should also clear
+ ``AUTH_INSECURE_KEYS_CREATABLE``.
+
+ To manually disable the creation of insecure keys:
+
+ .. code:: bash
+
+ ceph config set mon 'mon auth allow insecure key' false
+
+ Verify the ``AUTH_INSECURE_KEYS_CREATABLE`` is resolved:
+
+ .. code:: bash
+
+ ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_KEYS_CREATABLE") | not'
+
+ output gives ``false``.
+
+ For more information, see :ref:`auth_allow_insecure_keys`.
+
+#. **Rotate the admin key.**
+
+ .. warning:: Rotating the admin key requires special care as recovering from a mistake is complicated. Be careful.
+
+ .. note:: It is common for the ``client.admin`` credential's key to be copied to several nodes that may need to execute administrative commands. The new key will need to be copied to each node.
+
+ Create a backup emergency admin key in case of mistakes:
+
+ .. code:: bash
+
+ ceph auth get-or-create client.admin-backup mon "allow *" | tee ./client.admin-backup.keyring
+
+ Confirm the backup key works:
+
+ .. code:: bash
+
+ ceph -n client.admin-backup -k ./client.admin-backup.keyring auth ls
+
+ Now, rotate the ``client.admin`` key:
+
+ .. code:: bash
+
+ ceph auth rotate --key-type=aes256k client.admin | tee ./client.admin.keyring
+
+ .. note:: If you have updated ``auth_preferred_cipher`` then you can omit ``--key-type``.
+
+ .. warning:: The client.admin key is now changed. You cannot execute new Ceph commands as ``client.admin`` until you import the new key into your keyring.
+
+ Import the new ``client.admin`` key into your system's keyring file:
+
+ .. code:: bash
+
+ ceph-authtool --import-keyring ./client.admin.keyring /etc/ceph/ceph.client.admin.keyring
+
+ .. warning:: Your system's keyring file may be in a different location! Check ``/etc/ceph`` and your local Ceph configuration.
+
+ Verify the key works:
+
+ .. code:: bash
+
+ ceph -n client.admin -k /etc/ceph/ceph.client.admin.keyring ceph auth ls
+
+ If everything looks good, remove the backup key:
+
+ .. code:: bash
+
+ ceph auth rm client.admin-backup
+
+#. **Rotate other client keys.**
+
+ The process to rotate other client keys is similar to the admin key.
+
+ To view ``client`` credentials with insecure keys:
+
+ .. code:: bash
+
+ ceph health detail
+
+ should include output with :ref:`auth-insecure-client-key-type`:
+
+ ::
+
+ [WRN] AUTH_INSECURE_CLIENT_KEY_TYPE: 2 auth client entities with insecure key types
+ entity client.fs using insecure key type: aes
+ entity client.fs_a using insecure key type: aes
+
+ which tells you that two keys need to be updated.
+
+ If the client's software (e.g. ``ceph-fuse`` or the Linux kernel driver) is
+ up-to-date on all machines using the key, you may rotate the key and
+ distribute it.
+
+ .. code:: bash
+
+ ceph auth rotate --key-type=aes256k client.$ID | tee ./client.$ID.keyring
+
+ .. note:: If you have updated ``auth_preferred_cipher`` then you can omit ``--key-type``.
+
+ Then copy and import the key to each machine using that ``client.$ID`` credential.
+
+ Once all client credentials have been upgraded, you should see the ``AUTH_INSECURE_CLIENT_KEY_TYPE`` health warning clear.
+
+ .. code:: bash
+
+ ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_CLIENT_KEY_TYPE") | not'
+
+ output gives ``false``.
+
+ If you cannot rotate a particular client key yet, you may prefer to mute the
+ health warning until you can complete upgrading all of the client keys. We
+ expect this to be typical situation for some clusters.
+
+ .. code::
+
+ ceph health mute AUTH_INSECURE_CLIENT_KEY_TYPE 8w
+
+ to mute the warning for 8 weeks. Alternatively, use ``--sticky`` to make it permanent.
+
+
+#. **Disallow insecure keys for authentication.**
+
+ Now that all serivce daemon and client keys have been rotated, you can remove the insecure cipher key type
+ from the list of types allowed for authentication.
+
+ .. code:: bash
+
+ ceph mon set auth_allowed_ciphers aes256k
+
+ .. note:: This will now disable the default value for ``mon_auth_allow_insecure_key`` and clear the ``AUTH_INSECURE_KEYS_CREATABLE`` warning.
+
+ .. warning:: If you remove the key type for the ``client.admin`` key or for service daemon keys, you may break authentication in your cluster. That situation will require rescue via :ref:`auth_emergency_allowed_ciphers`. Ensure that ``AUTH_INSECURE_CLIENT_KEY_TYPE`` and ``AUTH_INSECURE_SERVICE_KEY_TYPE`` health warnings are clear!
+
+ Once changed, you should see the :ref:`auth-insecure-keys-allowed` health warning clear.
+
+ .. code:: bash
+
+ ceph --format=json health detail | jq '.checks | has("AUTH_INSECURE_KEYS_ALLOWED") | not'
+
+ output gives ``false``.
+
+At this point, your CephX ciphers and keys should be upgraded.
+
+
+.. _auth_rotate:
+
+Rotating CephX Keys
+-------------------
+
+The Monitors provide a mechanism to only update the key for an entity via the
+``auth rotate`` command.
+
+.. code:: bash
+
+ ceph auth rotate $TYPE.$ID
+
+For example:
+
+.. code:: bash
+
+ ceph auth rotate client.fs | tee ./client.fs.keyring
+
+The output of the command is the new key:
+
+::
+
+ [client.fs]
+ key = <redacted>
+ caps mds = "allow rwp"
+ caps mon = "allow r"
+ caps osd = "allow rw tag cephfs data=*"
+
+This can be imported into a new keyring using ``ceph-authtool``:
+
+.. code:: bash
+
+ ceph-authtool --import-keyring ./client.fs.keyring /etc/ceph/client.fs.keyring
+
+
+.. note:: The key must be distributed to all locations where the key is in use.
+
+
+.. _auth_emergency_allowed_ciphers:
+
+Emergency Allowed Ciphers
+-------------------------
+
+The Monitors maintain the set of allowed ciphers for credential keys in the ``MonMap``. This is normally set live on the cluster using:
+
+.. prompt: bash
+
+ ceph mon set auth_allowed_ciphers <cipher1,cipher2,...>
+
+It's possible to set a cipher for which no key exists to authenticate during
+key upgrades. To work around this, the Monitors may be rescued using the local
+startup configuration:
+
+.. confval:: mon_auth_emergency_allowed_ciphers
+
+This will allow your existing ``client.admin`` or other administrative key to authenticate as normal.
+
+When this configuration is set, the Monitors will raise the
+:ref:`auth-emergency-ciphers-set` health warning. It should only be set on a
+temporary basis to rescue the cluster.
+
+
+.. _auth_allow_insecure_keys:
+
+Allow Creation of Insecure Keys
+-------------------------------
+
+By default, the Monitors will allow creation of keys with a cipher type known
+to be insecure so long as the Monitors also allow that cipher to authenticate.
+When that cipher type is removed from the authentication list, the Monitors
+will also disable the default value of the ``mon_auth_allow_insecure_key``
+configuration.
+
+.. confval:: mon_auth_allow_insecure_key
+
+When disabled, ceph commands can no longer create keys with an insecure cipher
+type.
+
+When this configuration is enabled by default or otherwise, the Monitors will
+raise the :ref:`auth-insecure-keys-creatable` health warning.
-``auth_service_ticket_ttl``
-:Description: When the Ceph Storage Cluster sends a ticket for authentication
- to a Ceph client, the Ceph Storage Cluster assigns that ticket a
- Time To Live (TTL).
+.. _auth_dump_keys:
+
+Dump Existing Keys
+------------------
+
+The Monitors provide a command to dump all CephX credentials and key metadata
+as well as all rotating service key metadata.
+
+.. code:: bash
+
+ ceph --format=json-pretty auth dump-keys
+
+produces truncated output like:
+
+::
+
+ {
+ "data": {
+ "version": 15,
+ "rotating_version": 1,
+ "secrets": [
+ {
+ "entity": {
+ "type": 2,
+ "type_str": "mds",
+ "id": "a"
+ },
+ "auth": {
+ "key": {
+ "type": 1,
+ "type_str": "aes",
+ "created": "2025-07-29T21:53:30.978646-0400"
+ },
+ "pending_key": {
+ "type": 0,
+ "type_str": "none",
+ "created": "0.000000"
+ },
+ "caps": [
+ {
+ "service_name": "mds",
+ "access_spec": "\u0005\u0000\u0000\u0000allow"
+ },
+ {
+ "service_name": "mgr",
+ "access_spec": "\u0011\u0000\u0000\u0000allow profile mds"
+ },
+ {
+ "service_name": "mon",
+ "access_spec": "\u0011\u0000\u0000\u0000allow profile mds"
+ },
+ {
+ "service_name": "osd",
+ "access_spec": "\u0017\u0000\u0000\u0000allow rw tag cephfs *=*"
+ }
+ ]
+ }
+ },
+ ...
+ ],
+ "rotating_secrets": [
+ {
+ "entity": {
+ "type": 1,
+ "type_str": "mon",
+ "id": "*"
+ },
+ "secrets": {
+ "max_ver": 3,
+ "keys": [
+ {
+ "id": 1,
+ "expiring_key": {
+ "key": {
+ "type": 1,
+ "type_str": "aes",
+ "created": "2025-07-29T21:53:04.632712-0400"
+ },
+ "expiration": "2025-07-29T22:53:04.632703-0400"
+ }
+ },
+ {
+ "id": 2,
+ "expiring_key": {
+ "key": {
+ "type": 1,
+ "type_str": "aes",
+ "created": "2025-07-29T21:53:04.632715-0400"
+ },
+ "expiration": "2025-07-29T23:53:04.632703-0400"
+ }
+ },
+ {
+ "id": 3,
+ "expiring_key": {
+ "key": {
+ "type": 1,
+ "type_str": "aes",
+ "created": "2025-07-29T21:53:04.632718-0400"
+ },
+ "expiration": "2025-07-30T00:53:04.632703-0400"
+ }
+ }
+ ]
+ }
+ },
+ ]
+ }
+ }
+
+This command only works for format types ``json`` or ``json-pretty``.
+
+You may use this information to monitor the entities in the Monitor auth
+database as well as key types. For example, this information lets the operator
+check if insecure key types are in use. Consider this a low-level API. For
+example, the caps listed are in a binary format that is unsuitable for analysis.
+
+.. note:: Generally, the Monitors will warn you if there is a dangerous situation such as insecure key types are in use.
-:Type: Double
-:Default: ``60*60``
.. _Monitor Bootstrapping: ../../../install/manual-deployment#monitor-bootstrapping
When configuration options are displayed, the section name and any mask are presented
in separate fields or columns to make them more readable.
-Commands
---------
+.. _configuring-ceph-api:
+
+Configuration Commands
+----------------------
The following CLI commands are used to configure the cluster:
* ``ceph config show-with-defaults <who>`` is like the above,
but also shows all defaults, and for changed values, where they are changed.
+* ``ceph config generate-minimal-conf`` generates a minimal ``ceph.conf`` file
+ necessary to bootstrap connections to the cluster. Further configuration
+ can be done via the ``ceph config`` interface.
+
+ .. note:: Some configs only affect startup behavior for clients or daemons. In that case, modifying the local ``ceph.conf`` is required.
+
* ``ceph config assimilate-conf -i <input_file> -o <output_file>`` ingests a
configuration file from *input_file* and sets any valid options found into the
Monitor configuration database. Any settings that are unrecognized, are
.. versionadded:: Bobtail 0.56
-Authentication is explicitly enabled or disabled in the ``[global]`` section of
-the Ceph configuration file, as shown here:
-
-.. code-block:: ini
-
- auth_cluster_required = cephx
- auth_service_required = cephx
- auth_client_required = cephx
-
-In addition, you should enable message signing. For details, see `Cephx Config
-Reference`_.
+Presently, CephX is the default and sole authentication mechanism for Ceph.
+Full documentation should be consulted in `Cephx Config Reference`_.
.. _Cephx Config Reference: ../auth-config-ref
ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
+.. _auth-insecure-keys-creatable:
AUTH_INSECURE_KEYS_CREATABLE
____________________________
.. note:: cephadm and Rook should automate this process for you.
+.. _auth-insecure-service-tickets:
AUTH_INSECURE_SERVICE_TICKETS
_____________________________
.. note:: cephadm and Rook should automate this process for you.
+.. _auth-insecure-service-key-type:
AUTH_INSECURE_SERVICE_KEY_TYPE
______________________________
.. note:: cephadm and Rook should automate this process for you.
+.. _auth-insecure-rotating-service-key-type:
AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE
_______________________________________
.. note:: cephadm and Rook should automate this process for you.
+.. _auth-insecure-client-key-type:
AUTH_INSECURE_CLIENT_KEY_TYPE
_____________________________
.. note:: Rook should automate this process for you. cephadm does not generally administer client credentials.
+.. _auth-insecure-keys-allowed:
+
AUTH_INSECURE_KEYS_ALLOWED
__________________________
.. note:: Rook should automate this process for you. cephadm will not change this as it may adversely affect existing clients.
+.. _auth-emergency-ciphers-set:
AUTH_EMERGENCY_CIPHERS_SET
__________________________
},
...
-
+.. _health-mute:
Muting Health Checks
--------------------
projects / ceph-ci.git / commitdiff