David Galloway [Fri, 10 Jul 2026 20:12:01 +0000 (16:12 -0400)]
script/buildcontainer-setup: fix package install on debian trixie
Debian removed software-properties-common from the archive in trixie,
so the flat apt-get install list fails there. The package was only
needed to provide add-apt-repository for llvm.sh, which installs
clang-19 from apt.llvm.org. Trixie ships clang-19 natively, so install
it from the distro instead; run-make.sh's prepare() then finds clang-19
and skips llvm.sh entirely. Ubuntu and older Debian releases still have
software-properties-common and keep the previous behavior.
Fixes: https://tracker.ceph.com/issues/78111 Signed-off-by: David Galloway <david.galloway@ibm.com>
(cherry picked from commit b5697987ae86f51d7f16a1409ce9f2b542f3de52)
ceph-volume: skip internal raid mirror LVs in inventory
ceph-volume inventory started including all LVM mapper devices after c06bee965f1. On hosts with raid mirrored system volumes, that pulls in
hidden legs like var_rmeta_0 which have no /dev/vg/lv node and makes
cephadm's ceph-volume inventory call fail.
Skip those internal LVs in get_devices() and avoid rewriting the device
path to a missing lv_path in Device._parse().
Collect FCM stats locally from NVMe drives (vendor log page 0xCA)
and expose them via node-proxy, the cephadm agent, and
`ceph orch hardware status --category fcm`.
Introduce a node backend that aggregates Redfish data with
node-local collectors, since FCM metrics are not available
from the BMC.
Abhishek Desai [Tue, 26 May 2026 07:48:40 +0000 (13:18 +0530)]
mgr/dashboard : Support wildcard sans and zonegroup hostnames
fixes : https://tracker.ceph.com/issues/76795 Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>
(cherry picked from commit eee6a15dcd845914595dbb1d76470bd4b73947c1)
Aashish Sharma [Thu, 11 Jun 2026 09:19:04 +0000 (14:49 +0530)]
mgr/dashboard: fix zone creation in rgw service creation form
The zone creation request from the rgw service creation form was missing
the tier_type, sync_from and sync_from_all properties as a result the
zone creation was failing. This PR tends to fix this issue.
Afreen Misbah [Tue, 23 Jun 2026 13:30:22 +0000 (19:00 +0530)]
mgr/dashboard: fix bind address regression from CherryPy isolation
The CherryPy isolation refactor (PR #67227) accidentally changed the
dashboard bind address from wildcard (*:8443) to mon_ip:8443. The
get_mgr_ip() replacement was originally only for URI generation, but
the refactor passed the mutated address to CherryPyMgr.mount() as the
actual socket bind address.
This breaks the management gateway when its VIP is not on the same
interface as mon_ip, as the dashboard becomes unreachable on other
interfaces.
Preserve the original wildcard address for binding and only use
get_mgr_ip() for the advertised URI. Add regression test to prevent
future confusion between bind_addr and server_addr.
Kefu Chai [Sat, 13 Jun 2026 01:50:09 +0000 (09:50 +0800)]
python-common/cryptotools: stop using the removed X509Req API
pyOpenSSL deprecated OpenSSL.crypto.X509Req in 24.2.0 (2024-07-20) and
removed it in 26.3.0 (2026-06-12). as we don't pin pyopenssl, CI picked
up the new release, and create_self_signed_cert() started failing with:
AttributeError: module 'OpenSSL.crypto' has no attribute 'X509Req'
this took down run-tox-mgr, run-tox-mgr-dashboard-py3 and the mypy check.
we only used X509Req to build a subject name and then copied it into the
X509 cert. so drop it, and set the subject on the cert directly. the
resulting cert stays the same: subject from dname, issuer set to the same
subject, self-signed.
Alex Ainscow [Tue, 9 Jun 2026 14:45:25 +0000 (15:45 +0100)]
osd/ECTransaction: fix truncate+write planning for EC shard sizes
There are multiple problems fixed here, which are caused by operations
which perform a truncate-then-write in a single transaction.
NOTE: The only known scenario for these operations are CLS-sparsify operations.
I recommend backporting these changes to Tentacle, however, as they are
regressions in the RADOS API which can lead to data corruption.
PROBLEM: Cache not invalidated correctly:
If projected_size >= orig_size, the invalidate_cache flag is not set in the plan
This means there is potentially data in the RMW extent cache, which
may cause data corruption in subsequent writes (although not the write
being made). No actual use case for this operation is known, so it may not be as serious as
it sounds.
FIX: Invalidate cache on any truncate or "delete_first"
PROBLEM: Parity writes permitted with truncates:
Currently parity delta writes do not support operations which may have
truncates. However, if the object ended up the same size as pre-truncate, a
parity delta write may have been attemtped. This may lead to data corruption.
FIX: Block parity writes in this case.
NOTE: It is not worth the development effort to support PDW in this scenario, as
performance benefit would be minimal overall.
PROBLEM: RMW Reads can occur beyond first truncate point.
Such reads reflect the pre-truncate data (which is incorrect) and be
preserved, even if the invalidate cache flag is set (since the cache is
invalidated before the reads). This can lead to similar corruption as
the earlier "cache not invalidated"
FIX: trim the read to the lower truncate size.
PROBLEM:
When performing a truncate, the parity shards may need to be updated to
reflect truncates on other shards. These truncate writes in the plan were
overriding, rather than adding to, other writes in the op. The result was
a potentially truncated coding shard. This leads to assertions on reads.
FIX: Replace = with insert()
PROBLEM: Some shards not set to correct size on truncate-then-write
If projected_size is smaller or equal to the original size, then the
code which attempts to correctly size a shard will not run. However if
an operation performs a truncate then a partial write and does not
write to a particular shard AND the shard ends up smaller, then this
can lead to an incorrectly sized shard. This leads to assertion on reads.
FIX: Execute the shard-resize code on all truncates.
AI assistance was mainly used to write unit test. However, I cannot rule out a
contribution to the simple fixes found in this commit, so out of caution,
I place the Assisted-by tag.
Fixes: https://tracker.ceph.com/issues/77276 Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com> Assisted-by: IBM-Bob:ClaudeSonnet/GPT
(cherry picked from commit 51d8c5c489ba3e664209fb3316f8d6e03e257e28)
kmip tests are failing again on ubuntu 24 because PyKMIP doesn't support
python 3.12. we'll be removing ubuntu 22 from main, so can't just pin the
test to that distro in the meantime
we're expecing the nvmeof team to add python 3.12 support in our
ceph/PyKMIP fork, and can reenable kmip testing once that happens
Fixes: https://tracker.ceph.com/issues/76995 Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d27261d0c2641e92365cf590684567c58bbfb905) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 17 Jun 2026 18:33:45 +0000 (14:33 -0400)]
qa/rgw: Remove 'force-branch' from s3tests configs
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 5bc34f8ed84aa2f205ccfe04fff9e3ec5ced8727) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Mon, 29 Sep 2025 21:11:09 +0000 (17:11 -0400)]
qa/rgw: Run s3-tests from within the Ceph repo
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit d0c88cb53c6848e9bcd7c4ea90f21f1aa6912242) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Thu, 11 Jun 2026 22:46:45 +0000 (18:46 -0400)]
test/rgw: Include s3-tests in Ceph repo
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 868d20182a08685265e0b5831bad703fdcf3189b) Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Kefu Chai [Tue, 23 Jun 2026 07:43:28 +0000 (15:43 +0800)]
mgr/dashboard: skip the table when an nvmeof cli result has no columns
The dashboard leaves prettytable unpinned. prettytable commit 2574492 ("Apply
some Pylint rules (PLR)", #436) rewrote _stringify_row()'s row_height as
`max(_get_size(c)[1] for c in row)`, which raises ValueError("max() iterable
argument is empty") on a row with no cells. The change is undocumented and
shipped in 3.18.0; get_string() trips on it when a table has a row but no
columns.
AnnotatedDataTextOutputFormatter builds such a table for an empty result, or
one whose only field is status or error_message, so NvmeofCLICommand.call()
returns -EINVAL and the command fails. This broke run-tox-mgr-dashboard-py3
once the tox virtualenv picked up prettytable 3.18.0.
Return an empty string when there are no columns instead of formatting a
degenerate table.
rgw: restore constant-time GCM tag comparison in ISA-L path
a8ed43bfc05 replaced ct_memeq with memcmp in the ISA-L GCM accelerator,
making tag verification and the key-cache compare non-constant-time.
Restore ct_memeq for both; the OpenSSL and EVP paths already compare in
constant time.
Matthew Heler [Fri, 5 Jun 2026 15:48:32 +0000 (10:48 -0500)]
rgw: fix AES-256-GCM key/IV reuse on multipart part re-upload
Re-uploading the same part number in a GCM multipart upload encrypted the new
data under the same key and IV as the first upload, since the IV is
part_number||chunk_index and the part key came from the part number alone. GCM
requires a unique IV per key; reusing one to encrypt different data weakens its
confidentiality and integrity guarantees.
Generate a random 16-byte salt on each UploadPart and fold it into the part key,
HMAC(ObjectKey, BE32(part) || salt), so every upload gets a fresh key. The salt
rides RGWUploadPartInfo, and complete stores the selected part's salt in
RGW_ATTR_CRYPT_PART_NUMS, which now holds (part, salt) pairs. GET reads it back
to re-derive the key, and an empty salt reproduces the old derivation so unsalted
parts still decrypt.
rgw/restore: take the hash mod HASH_PRIME when picking a shard
choose_oid fed ceph_str_hash_linux straight into % max_objs, and the
low bits of that hash are weak enough that similar object names keep
landing on the same shard. LC takes the hash mod HASH_PRIME first for
exactly this reason, do the same here.
Matthew N. Heler [Mon, 18 May 2026 01:57:01 +0000 (20:57 -0500)]
mon: add monitor RocksDB backup and restore
Implements an opt-in backup mechanism for the monitor using
rocksdb::BackupEngine. Backups run on a schedule when
mon_backup_interval is set, or are triggered manually via
`ceph tell mon.* backup`. Cleanup keeps the last N, hourly,
and daily snapshots, with a free-space guard. Off by default.
Restore is offline: stop the mon and run
ceph-mon --restore-backup <dir> --yes-i-really-mean-it
optionally with --backup-version (BackupEngine logical version,
as shown by --list-backups). The mon keyring is stashed alongside
the RocksDB backup so a wiped mon_data is recovered end-to-end,
and kv_backend is stamped back when missing.
Co-authored-by: Daniel Poelzleithner <poelzleithner@b1-systems.de> Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
(cherry picked from commit 3a9ae41e2a8fd614d67e3dac39d28ddf5dd6ca4a)
mgr/DaemonServer: Aggregate and globally sort OSDs for ok-to-upgrade
The 'ok-to-upgrade' command output sorting did not scale accurately
when target CRUSH buckets contained multiple child buckets (e.g., a
chassis containing multiple hosts). OSDs were previously sorted
individually per child bucket and appended sequentially. This created
fragmented, per-host sort segments rather than a globally sorted list
for the parent bucket.
Changes:
1. Fix the issue above by aggregating all child OSDs into a single vector prior
to executing a single, global sort operation based on PG counts. Additionally,
optimize memory efficiency and future-proof the logic by reserving continuous
vector blocks to avoid dynamic heap reallocations.
2. Add integration tests with chassis and rack based CRUSH hierarchies which
verifies the ok-to-upgrade functionality. In addition, the tests crucially
verify the order of OSDs returned is according to the ascending order of
acting PG count. Additionally, make minor fix-ups to lines that determine the
length of a list in JSON response by removing the redundant "| bc".
Patrick Donnelly [Fri, 18 Nov 2022 19:13:01 +0000 (14:13 -0500)]
umbrella: doc: add releases links to toc
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 8cf9ad62949516666ad0f2c0bb7726ef68e4d666) Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Conflicts:
doc/index.rst: index changes
Patrick Donnelly [Wed, 10 Jun 2026 18:30:59 +0000 (14:30 -0400)]
Merge PR #66726 into main
* refs/pull/66726/head:
doc: Update documentation to reflect new functionality
test: Add integration tests for EC Omap operations and recovery
osd: Hook up omap operations in EC pools
osd: Allow for recovery of OMAP header and entries in EC pools
doc: Write design document to explain the reasoning behind implementing this feature
osd: Introduce functions required for EC OMAP support
osd: Add ECOmapJournal class and relocate OmapUpdateType enum class
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com> Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Ronen Friedman [Wed, 3 Jun 2026 05:40:25 +0000 (05:40 +0000)]
crimson/osd: move get_max_object_size() to store level
is_offset_and_length_valid() called get_sharded_store() locally to
obtain the store-specific max_object_size. On alien cores (where
smp::count > store_shard_nums), the local store is inactive and the
call hits assert(shard_store.get_status() == true).
As the max object size is a store-specific property and not a
store-shard one, there is no reason to acquire the
store shard to obtain it. Instead -
a get_max_object_size() method is added to the Store interface.
Sun Yuechi [Wed, 10 Jun 2026 00:13:53 +0000 (08:13 +0800)]
cmake: disable Catch2 tests when Catch2 is unavailable
debhelper on noble passes -DFETCHCONTENT_FULLY_DISCONNECTED=ON, so CPM
cannot fetch Catch2 and silently skips it, leaving no Catch2 targets
behind and breaking the generate step. Fall back to WITH_CATCH2=OFF
with a warning instead.
dheart [Tue, 9 Jun 2026 13:27:14 +0000 (21:27 +0800)]
os/bluestore: prevent reallocation and corruption when shared_blob key is missing/undecodable
When the shared_blob key is missing or fails to decode,
it is necessary to scan the blob's pextents directly as the sole authoritative source
to verify allocated blocks and prevent double-allocation.
Emmanuel Ameh [Tue, 9 Jun 2026 12:40:03 +0000 (13:40 +0100)]
doc/man: Remove stale EOL release names from deprecation notices
ceph.rst: "osd create" deprecation notice cited "the Luminous release"
(2017, EOL 2020). Update to a plain deprecation statement directing
users to the replacement command (osd new).
rbd.rst: cephx_require_signatures option deprecation cited "the Bobtail
release" (2013, EOL 2015) as context for why the option is deprecated.
Remove the EOL release name; retain the deprecation warning. Fix the
companion nocephx_require_signatures notice for consistency ("in a
future release" instead of "in the future").
Matty Williams [Mon, 18 May 2026 09:09:32 +0000 (10:09 +0100)]
osd: Hook up omap operations in EC pools
Add pool flag to determine if omap operations are supported in a pool.
- Currently disabled in EC pools (will later be enabled for Fast EC pools)
Require all osds to have umbrella or later release version to enable pool flag.
Change recovery reads to use journal updates.
Clear the journal for a new epoch.
Set omap_complete accurately before recovery.
Encode omap updates and add entry to journal.
Decode omap updates, apply updates to object store, then remove from journal.
Change omap reads in PrimaryLogPG to use PGBackend functions, including omap updates from journal.
Assisted-by: Bob
Used for debugging and copying patterns (e.g. implementing REPLACE type to match MODIFY).
Fixes: https://tracker.ceph.com/issues/74188 Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Matty Williams [Tue, 12 May 2026 15:11:17 +0000 (16:11 +0100)]
osd: Allow for recovery of OMAP header and entries in EC pools
Add omap fields to read_request_t, read_result_t, ECSubRead and ECSubReadReply.
Read and write omap header and entries if !omap_complete.
Require omap_complete to finish recovery.
Fixes: https://tracker.ceph.com/issues/74244 Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Matty Williams [Fri, 12 Dec 2025 11:21:10 +0000 (11:21 +0000)]
osd: Introduce functions required for EC OMAP support
Introduced a "supports_omap" pool flag which is always enabled for Replicated pools and currently always disabled for EC pools.
Introduced wrappers around omap read operations in PGBackend to include updates from the journal in EC pools with optimisations enabled.
Introduced a function for encoding an EC_OMAP operation in the ObjectModDesc::Visitor class and a function for committing an operation in the Trimmer struct.
Signed-off-by: Matty Williams <Matty.Williams@ibm.com>
Reviewed-by: Joseph Mundackal <jmundackal@bloomberg.net> Reviewed-by: Anthony D Atri <anthony.datri@gmail.com> Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Adam Kupczyk [Tue, 1 Jul 2025 13:47:14 +0000 (13:47 +0000)]
os/bluestore: Add new onode recovery method
Added read_allocation_from_onodes_mt function
(originally copied from read_allocation_from_onodes).
Added Decoder_AllocationsAndStatFS class
(originally copied from ExtentDecoderpartial).
There are significant differences from originals:
- shared blobs are not scanned at all
- to not account allocations more than once,
collisions are detected on SimpleBitmap level;
only the first onode referencing shared blob will mark allocation
- Blobs are not preserved
- instead we remember only if blob or spanning blob was compressed
The underlying logic is make recovery faster and prepare for
multithread refactor.
projects / ceph.git / log