Alex Ainscow [Fri, 24 Apr 2026 14:57:55 +0000 (15:57 +0100)]
osd: Fix incorrect rollback logic for partial write OI
Before this fix, when rolling back an OI, the system used the OI
from the primary to rollback to. This is wrong if the previous
write was a partial write. This may have a few consequences
during recovery (although its not clear any are serious) and in
EC direct reads, where a false-positive version mismatch will be
detected.
The test provided recreates the issue.
The fix provided modifies the rollback as it is being written.
Fixes: https://tracker.ceph.com/issues/76213 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
# Conflicts:
# src/test/osd/TestECFailoverWithPeering.cc
Alex Ainscow [Mon, 27 Apr 2026 16:46:40 +0000 (17:46 +0100)]
osd: Avoid assertion on empty object read when reading multiple objects
Tracker 75432 hits an assert which is attempting to protect the system
against hanging, due to generating a read request which sends no messages.
The assert fired because recovery was attempting to read multiple objects
in a single read request. One object did not require any further shard
reads in order to recover, while the other did. The consequence is that
the assert fired on one of the objects.
The problem is simply that the assert is in the wrong place.
Fixes: https://tracker.ceph.com/issues/75432 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Alex Ainscow [Wed, 18 Mar 2026 09:22:26 +0000 (09:22 +0000)]
osd: PGLog Attach correct version to missing list when ignoring log entries
A previous fix for PR 66698 fixed an issue where log entries associated with
partial writes were being processed incorrectly (see that PR and associated
tracker for details). The fix was to ignore log entries that should not have
been present on the non-primary shard.
The problem with that approach is that in a more complex scenario, where the
log contained a partial write, followed by a full write AND the shard is
backfilling, then the missing list was being given the version prior to the
full write, rather than prior to the clone.
Our fix here corrects how the missing list version is calculated.
See the associated tracker for instructions on how to recreate
Fixes: https://tracker.ceph.com/issues/75211 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
Alex Ainscow [Mon, 27 Apr 2026 13:24:45 +0000 (14:24 +0100)]
osd/test: Add EC peering test infrastructure and recovery test cases
This commit enhances the EC peering test framework and adds test cases
for erasure-coded pool recovery scenarios:
NOTE: Many of the tests cases are disabled as they recreate certain
problems. Later commits will enable these tests and fix the production
issues, but under different PRs.
Test Infrastructure Improvements:
- Add MockStore wrapper with read error injection capabilities for testing
error handling in EC recovery
- Enhance ECPeeringTestFixture with recovery callback verification
- Add support for pg_upmap to better simulate OSD placement
- Implement write_attribute() for testing partial vs full stripe writes
- Add read_shard_object_info() to verify on-disk version consistency
- Improve logging with missing object stats (m=, u=, mbc=)
- Add support for doing object recovery in Fast EC.
- Add set_config() helper for runtime configuration changes
- Preserve xinfo features when marking OSDs up/down
- Fix pg_temp handling for EC pools with optimizations
Patrick Donnelly [Tue, 28 Apr 2026 22:25:44 +0000 (15:25 -0700)]
doc/start/os-recommendations: update for Umbrella and future releases
Overhaul the OS recommendations documentation to reflect deployment
practices and map out the support matrices for upcoming releases through
Ceph X (24.x).
Key changes include:
* Emphasized container-based deployments: Added a new section strongly
recommending containerized deployments via `cephadm` over legacy
package-based installations to simplify upgrades and avoid host-level
dependency conflicts.
* Expanded support tables: Updated the Platforms and Container Hosts
tables to include Umbrella (21.x), Vampire (22.x), W (23.x), and
X (24.x). Removed EOL releases like Reef.
* Added EOL visibility: Included End-of-Life dates for Linux
distributions and anticipated EOL dates for Ceph releases to help
administrators plan lifecycle events.
* Updated OS targets: Added support tracking for Ubuntu 24.04 (Noble),
Ubuntu 26.04, Ubuntu 28.04, Rocky Linux 10, and Rocky Linux 11.
* Addressed CentOS transition: Added a warning that CentOS 10+ will no
longer be built or tested by upstream. Documented that Rocky Linux 10
is the new default container base image for Umbrella, while clarifying
that the bare-metal host OS can remain any supported distribution.
* Added horizontal upgrade guidance: Introduced a new section outlining
safe "horizontal" bare-metal OS upgrade paths (e.g., CentOS 9 to
Rocky 10, Ubuntu 22.04 to 24.04) so users can safely migrate their
nodes outside of Ceph version upgrade windows.
AI-Assisted: Gemini Pro, through numerous prompts Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
ceph-volume: make TPM2 PCR policy configurable (default to PCR 7)
tpm enrollment for dmcrypt OSDs is hardcoded to systemd-cryptenroll
--tpm2-pcrs 9+12 which ties the LUKS key to initrd and kernel
command line measurements, which is brittle on RHEL image mode
systems: after a bootc switch, the kernel, initrd, or cmdline often
change, the PCRs move, and the volume won't unlock until you re-enroll
or fall back to another key.
typical error:
```
Apr 27 14:17:25 ceph-jx5fq20u bash[4289]: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /usr/lib/systemd/systemd-cryptsetup attach M3zE7r-qsGZ-xs0T-610d-SJNZ-U89x-J0cJq8 /dev/ceph-cac05fb6-51d3-4a60-9fc1-4958c568b433/osd-block-b1a495a0-e1a4-4888-baf9-7990f45f1e56 - tpm2-device=auto,discard,headless=true,nofail
Apr 27 14:17:26 ceph-jx5fq20u ceph-e5520e2c-420d-11f1-a7b9-5254001191fb-osd-0-activate[4300]: stderr: Failed to unseal secret using TPM2: Operation not permitted
Apr 27 14:17:26 ceph-jx5fq20u bash[4289]: stderr: Failed to unseal secret using TPM2: Operation not permitted
```
The patch makes the PCR set configurable and defaults to 7 so bootc style
deployments behave correctly.
mgr/dashboard: Update permissions for pool-manager role
Fixes https://tracker.ceph.com/issues/76307
- says denied access when clicked on create pool table action
- this was happening due to the failing monitor API added for stretch cluster configuration
- also updates overview nav permissions
ceph-volume: raw activate should ignore lvm backed OSD devices
the generic activate (`ceph-volume activate`) runs the
raw path before LVM. Raw.activate was walking lsblk / raw
list entries and could hit block devices that are actually
logical volumes from `ceph-volume lvm prepare` or `lvm batch`
(with ceph lvm tags on the lv).
That made raw activation poke at LVM backed OSDs instead of
leaving it to `lvm activate`.
with this commit ceph-volume now builds the set of LV paths
that carry those tags once (`lvs` via ceph_volume_lvm_prepare_lv_paths)
and skip any candidate path that matches, so only real raw
OSDs go through the 'raw activate path'.
Also, we now pass `with_tpm` through luks_open() calls for db and
wal so encrypted metadata uses the same systemd-cryptsetup path
as the block LV when ceph.with_tpm is set.
mgr/cephadm: replace md5_hash with FIPS-safe config_hash
Replace md5_hash() usages in cephadm dependency hashing with an
algorithm-agnostic config_hash() helper. config_hash() is backed by
SHA-256, making dependency hash generation unconditionally FIPS-safe
while preserving change-detection behavior.
Ville Ojamo [Wed, 22 Apr 2026 06:51:34 +0000 (13:51 +0700)]
doc/rados: improve troubleshooting-mon.rst
Don't ceph tell mon_status and then claim it passes the help command.
Improve language and link to cephadm doc on asok usage. Add label and
note about accessing asok from the host in troubleshooting.rst.
Capitalize and use double backticks consistently.
Add some missing articles and other minor word changes.
Fix indentation.
Use ref and link definitions consistently, use automatic bold.
Use privileged prompts for CLI commands where necessary.
Remove spaces at end of lines and change tabs to four spaces.
Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>
Afreen Misbah [Fri, 27 Mar 2026 16:06:38 +0000 (21:36 +0530)]
mgr/dashboard: Add gray10 theme base color to all pages
- applies #f4f4f4 - $background to all pages as base page
- earlier the base color of page was white
- also updates tabs/navs/tables css to adapt
- some fixes of spacings in alerts tabs, nvmeof
Afreen Misbah [Thu, 26 Mar 2026 13:25:18 +0000 (18:55 +0530)]
mgr/dashboard: Remove tooltip and popover defaults
Fixes https://tracker.ceph.com/issues/75410
These defaults are not required as carbon adds blackish color to tooltips and moving forward we want to align to CDS.
If anything breaks then add / fix in the used component
cephadm: replace call_throws with call in command_inspect_image
Problem:
During the upgrade, when inspecting the new ceph image for the first time, an error is printed to the ceph-mgr log instead of displaying a user-friendly message.
Root cause: During an upgrade, inspect-image is called on each node to check if the target image exists locally before pulling it. This flow, where inspect-image always precedes the pull, occurs on nodes other than the first.
Code Fixes:
1. src/cephadm/cephadm.py:
Replace call_throws with call in command_inspect_image. call_throws raises a RuntimeError on any non-zero exit code, producing a full traceback in the logs. call returns the exit code instead of raising, so the function exits cleanly with errno.ENOENT when the image is not found.
cephadm: convert lists back to tuples when loading last_client_files
Problem: ceph mgr fail or active ceph mgr restart causes unnecessary client files recreation on _admin hosts. Files such as /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring are rewritten even when their content has not changed.
Root cause:
update_client_file() stores client file metadata as a Python tuple (digest, mode, uid, gid).
When save_host() persists this to the mon store via json.dumps(), the tuple is serialized as a JSON array since JSON has no tuple type.
On mgr failover or restart, cache.load() deserializes the data with json.loads(), which returns a Python list instead of a tuple.
The comparison in _write_client_files(): match = old_files[path] == (digest, mode, uid, gid) then compares a list (from JSON) against a tuple (freshly built), which always evaluates to False.
This causes every client file to be rewritten on every mgr failover or restart.
Code Fixes:
1. src/pybind/mgr/cephadm/inventory.py:
convert the deserialized lists back to tuples when loading last_client_files
Shweta Bhosale [Wed, 18 Feb 2026 14:29:58 +0000 (19:59 +0530)]
mgr/nfs: 1. Removed the option to enable and disable cluster wide qos, it will be enabled by default
2. Removed the cluster_enable_qos field from the cluster-level block as it was causing confusion for the user.
3. Instead of using cluster use global while showing cluster level qos values in export qos get
Shweta Bhosale [Thu, 6 Nov 2025 13:04:19 +0000 (18:34 +0530)]
mgr/cephadm: support nfs cluster level qos
Added below CEPH_NODES_LIST block in ganesha.conf and enable_cluster_qos in cluster level QoS block
CEPH_NODES_LIST {
Ceph_Nodes = 192.168.100.100, 192.168.100.101, 192.168.100.102;
}
Fixes: https://tracker.ceph.com/issues/69861 Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
mgr/cephadm: Changes to add NFS cluster qos inter node communication port in spec
mgr/nfs: Addressed review comments for cluster level qos support
mgr/nfs: add enable_cluster_qos = true while enabling qos
Shweta Bhosale [Wed, 19 Mar 2025 11:16:10 +0000 (16:46 +0530)]
mgr/nfs: When cluster level qos is disabled and export still has qos parameters, then allow nfs export apply command if file has same qos block which is already set
projects / ceph-ci.git / log