Adam King [Wed, 14 May 2025 17:16:43 +0000 (13:16 -0400)]
mgr/cephadm: rotate keyring for core ceph daemons during upgrade
Specifically, this causes us to rotate the mgr, mon, OSD,
and mds keyrings. The mgr and mon keyring are done as soon
as we see all the mons have been upgraded and OSD/mds happens
when we reach them in the upgrade order.
The routine was structured to print warnings frequently whenever the
config is refreshed (manually or normally). This confused Rook with
verbose debugging warnings so I tried to clean that up in an earlier
commit. This unfortunately broke command-line keyfile/key arguments.
I've resolved that now robustly by putting the error message in a
throwaway buffer that can be printed only from the init routines.
Lastly, this commit changes the preference for sourcing the key to
configs "key", then "keyfile", then "keyring". Before, it checked the
keyring file but, as this is usually set in the ceph.conf, that prevents
overriding with another key.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Generally speaking, these keys are constructed by
"CryptoKey::decode" which does not know the context for how the
key will be used, so usage can't be set here. In a brief
experiment, these usages for keys were invoked by keys decoded
under these routines:
3 4 5 CephxClientHandler::handle_response
11 13 15 CephXTicketHandler::verify_service_ticket_reply
Marcus Watts [Sat, 15 Nov 2025 02:57:53 +0000 (21:57 -0500)]
auth: CryptoKey, dynamic usage keys
Make it possible to specify usage that was not initially declared
when the secret was set in CryptoKey. As a prerequisite, simplify
the per-usage key storage structures so that is is less heavy-weight
and thread-safe.
Also, extend various encrypt and decrypt apis to enable passing
suage down to CryptoKey.
Marcus Watts [Tue, 4 Nov 2025 01:50:24 +0000 (20:50 -0500)]
auth: CryptoKey, use secret in CryptoKeyHandler
Keep only one copy of secret in CryptoKeyHandler. This will reduce
the number of copies made in memory. Also introduce bool() and ==
opeartors so we can hide implementation details.
Patrick Donnelly [Tue, 16 Sep 2025 20:02:05 +0000 (16:02 -0400)]
auth: remove superfluous error log message
It's also possible that _refresh_config can be called multiple times before the
keyring config has been set (by an arg/env for instance). This would pollute
the log with erroneous error warnings.
MonClient::authenticate already warns about this.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Patrick Donnelly [Tue, 24 Jun 2025 03:27:31 +0000 (23:27 -0400)]
mon: provide emergency mechanism to rescue allowed_ciphers
If the administrator accidentally revokes auth to client.admin, they cannot fix
it because the setting is stored in the monmap. Provide a config to restore
access in such an emergency.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
This commit prompted the previous refactor as it was inconvenient to check for
health warnings as part of AuthMonitor::tick and then pass those up via
PaxosService::encode_health.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
This was motivated by confusing persistence of some health warnings during
testing of health warnings for cephx upgrades. Some services are only doing
health checks during ::encode_pending and others during ::tick. Make it
consistent.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Patrick Donnelly [Thu, 29 May 2025 15:57:55 +0000 (11:57 -0400)]
auth: check service key is valid before decryption
CryptoKey::empty is the correct mechanism to check for an invalid key (and this
is codified elsewhere, fixed in this commit). Decryption would fail with an
abort if the key handler was unset. This would happen after rotating the "mon."
key and then restarting one of the mons.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Patrick Donnelly [Thu, 29 May 2025 15:04:00 +0000 (11:04 -0400)]
mon: provide emergency mechanism to use mon keyring
If they key is lost for the `mon.` credential, it's very inconvenient to get it
out of the "auth" database in the mon store. So, allow the operator to create a
new keyring for the mons and use it instead to get mons in quorum again.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Patrick Donnelly [Thu, 29 May 2025 14:13:40 +0000 (10:13 -0400)]
mon: cycle through keyring or key_server for auth with mons
After commit `mon: use key_server for looking up mon key`, the mons will now
use the key_server to lookup the `mon.` key when a mon connects. We need to
make the mons prefer using that key with authenticating during probing other
mons. However, the protocol doesn't allow falling back to another key. This is
necessary if what's in the key_server database is out-of-date due to an earlier
loss of quorum. In that case, the operator should update the local keyring file
and the mon should give that a try if auth fails.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Patrick Donnelly [Thu, 29 May 2025 14:07:52 +0000 (10:07 -0400)]
mon: use key_server for looking up mon key
Note: the key_server is already configured to fallback (via
KeyServerData::extra_secrets) to the Monitor::keyring which is sourced from the
mon's keyring file.
Using the Monitor::key_server allows us to maintain the mon's secret in the
auth database alongside all other secrets. This makes rotating the mons' keys
the same as all other entities in Ceph. Before this, to rotate the mons' key
you would need to turn off all montitors and then rotate the key files
manually. This is obviously disruptive since it's not a rolling upgrade.
If the key is sourced from the Monitor::key_server, then the key can be rotated
and all mons are aware of the new key. The mons can then proceed to restart as
needed in a non-disruptive fashion.
A followup commit will cleanup the monitor to try either its local keyring key
or the key in the key_server (if present) when authenticating with other mons.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Patrick Donnelly [Wed, 26 Mar 2025 02:05:09 +0000 (22:05 -0400)]
tools/ceph_authtool: allow configuring a preferred cipher
This makes testing easier as we can configure all keys in the cluster to be the
given "old" type without modifying each location that ceph-authtool is used.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
mon/MonClient: wipe secrets and invalidate tickets on auth epoch change
* This causes service daemons to drop all known service tickets and request new
ones from the auth server.
* This causes the clients (and service daemons) to request new tickets from the
auth server which will include tickets signed with the new service keys.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
This will be used to indicate to clients / service daemons that the auth
service keys have been rotated. Clients and service daemons are expected to
invalidate their tickets and reauth. Service daemons should wipe their service
keys.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Patrick Donnelly [Wed, 26 Mar 2025 01:59:34 +0000 (21:59 -0400)]
mon/AuthMonitor: add dump-keys and wipe-rotating-service-keys
`auth dump-keys` allows examining the key types for each entity and also the
rotating session keys. This lets us confirm key upgrades are done as expected.
`wipe-rotating-service-keys` clears out existing non-auth service keys so that we do not
need to wait for the rotating key expiration. It is not disruptive so long as clients
renew their tickets when prompted by the auth epoch change.
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
projects / ceph-ci.git / log