Adam Kupczyk [Tue, 1 Jul 2025 13:47:14 +0000 (13:47 +0000)]
os/bluestore: Add new onode recovery method
Added read_allocation_from_onodes_mt function
(originally copied from read_allocation_from_onodes).
Added Decoder_AllocationsAndStatFS class
(originally copied from ExtentDecoderpartial).
There are significant differences from originals:
- shared blobs are not scanned at all
- to not account allocations more than once,
collisions are detected on SimpleBitmap level;
only the first onode referencing shared blob will mark allocation
- Blobs are not preserved
- instead we remember only if blob or spanning blob was compressed
The underlying logic is make recovery faster and prepare for
multithread refactor.
Adam Kupczyk [Fri, 4 Jul 2025 16:28:16 +0000 (16:28 +0000)]
os/bluestore: Rework on decoding
Refactored ExtentDecoder.
Introduced decode_create_blob method to it.
Converted bluestore_blob_t::decode and Blob::decode methods into templates.
Created clear example path how to specialize these and other decoders.
Adam Kupczyk [Mon, 18 May 2026 16:33:45 +0000 (16:33 +0000)]
objectstore/test_kv: Unittest for util_divide_key_range
Extensive tests for quality of KeyValueDB::util_divide_key_range.
Tests speed and correctness of split.
Has 2 control modes:
1) on Jenkins (detected by JENKINS_HOME) run with reduced scope
2) passing env VERBOSE=1 gives more details
Adam Kupczyk [Fri, 15 May 2026 16:07:07 +0000 (16:07 +0000)]
kv/KeyValueDB: New utility function util_divide_key_range
Significant reshuffle. Cleaned loops.
Points scanned on db were [size]->[key]. Now it is [key]->[size],
which is better since keys are unique by design, but calculation
of size can be a victim to RocksDB estimation precision.
Adam Kupczyk [Tue, 1 Jul 2025 11:30:59 +0000 (11:30 +0000)]
kv/KeyValueDB: New utility function util_divide_key_range
New function splits provided range into smaller chunks.
Declared in KeyValueDB, but implemented only for RocksDBStore.
Useful for splitting large datasets for multiple threads to
iterate in parallel.
Afreen Misbah [Fri, 27 Mar 2026 16:06:38 +0000 (21:36 +0530)]
mgr/dashboard: Add gray10 theme base color to all pages
- applies #f4f4f4 - $background to all pages as base page
- earlier the base color of page was white
- also updates tabs/navs/tables css to adapt
- some fixes of spacings in alerts tabs, nvmeof
Afreen Misbah [Thu, 26 Mar 2026 13:25:18 +0000 (18:55 +0530)]
mgr/dashboard: Remove tooltip and popover defaults
Fixes https://tracker.ceph.com/issues/75410
These defaults are not required as carbon adds blackish color to tooltips and moving forward we want to align to CDS.
If anything breaks then add / fix in the used component
Keep parsed command data alive while running hooks to avoid a
stack-use-after-return in Formatter::create().
Return -EAGAIN from PGCommand when the OSDMap is not ready.
Crimson OSD was missing the PG admin/tell hooks that classic OSD exposes, and it
did not accept the legacy `rados_pg_command()` / `ceph pg <pgid> <cmd>` JSON form
(e.g. `{"prefix":"pg","pgid":"1.0","cmd":"query",...}`), so `ceph pg <pgid> query`
failed.
Adds a `pg` old-form wrapper hook that exists to advertise that exists
to advertise the classic `pgid` + `cmd` + optional `arg` signature. The
runtime dispatch rewrites this to the real subcommand.
This updates parse_cmd to rewrite `prefix=pg` requests to the requested
subcommand and remap the generic `arg` field to the concrete parameter
names (`offset` for `list_unfound`, `mulcmd` for `mark_unfound_lost`)
so validation/parsing is unambiguous.
Add a standalone concept page for the OSDMap require_osd_release field,
the upgrade-gate counterpart to require_min_compat_client. Cover:
- how to set it and how to check it;
- the full set of pre-commit guards the monitor runs, rendered as a
table with each guard's error text and bypass status;
- which commands and features become available as the flag is raised,
per release;
- the OSD boot window that refuses OSDs more than two releases ahead
of the flag;
- the OSD_UPGRADE_FINISHED health warning that prompts admins to set
the flag after an upgrade;
- the initial value on new clusters and the two mon_debug_* knobs
that override it for testing.
Also cross-link the new page from the related-flags table on
require-min-compat-client.rst, and from the rados operations index.
Add a standalone concept page for the OSDMap require_min_compat_client
field, covering: how to set and check it, the non-monotonic lowering
behavior (with the features-in-use floor derived from
OSDMap::get_min_compat_client()), and the operator commands it gates.
Include tables for the floor-pinning features and the flag-gated
commands, so operators can reason about transitions without reading
OSDMonitor.cc.
Cross-reference to the CephFS per-filesystem required_client_features
mechanism, which is the MDSMap-side equivalent for client-protocol
features. Add an anchor on the existing CephFS Required Client Features
section so the cross-reference resolves.
Link the new page from the rados operations index.
doc: document ceph nvmeof CLI subcommands for target configuration
Replaces verbose podman run container commands with native ceph nvmeof
CLI subcommands. The nvmeof-cli container approach is preserved as an
alternative in a note block, with a clarification that its option names
differ from the ceph nvmeof CLI.
doc/scripts: use raw string for regex in gen_state_diagram.py
Python 3.12 emits SyntaxWarning for invalid escape sequences in ordinary
string literals. The re.search() call on line 162 was the only pattern
in the file passed as a non-raw string, causing:
doc/scripts/gen_state_diagram.py:162: SyntaxWarning: invalid escape
sequence '\s'
i = re.search("return\s+transit<\s*(\w*)\s*>()", line)
Add the r"" prefix to match the other re.search / re.finditer / re.sub
call sites in the same file. No behavior change; \s was already being
interpreted as a regex whitespace class because Python leaves unknown
escapes untouched, but this will become a SyntaxError in a future
Python release.
Add unit tests to cover the raw prepare help text for --osd-fsid,
assert generate_uuid is used when no osd_fsid is supplied and
assert an externally provided osd_fsid is passed through to
create_id without generating a new UUID.
ceph-volume: add --osd-fsid support to raw mode prepare
The LVM mode already supports --osd-fsid to allow external tools
(e.g., Kubernetes operators) to pre-register an OSD ID+UUID via
"ceph osd new" and then pass both to ceph-volume, ensuring the
operator retains full control of the OSD ID lifecycle and can
reliably clean up on prepare failure (no orphan OSDs).
The raw mode was missing this support: prepare() unconditionally
called system.generate_uuid(), ignoring any --osd-fsid value.
When an operator pre-registered osd.N with uuid_A and then ran
"ceph-volume raw prepare --osd-id N --dmcrypt", ceph-volume
generated uuid_B internally and called "ceph osd new uuid_B N",
which failed with EINVAL because the ID was already registered
with a different UUID.
This commit:
- Adds --osd-fsid argument to the raw mode argument parser
(devices/raw/common.py), consistent with the LVM mode.
- Changes raw.prepare() to honor an externally provided osd_fsid,
falling back to generate_uuid() only when none is given
(objectstore/raw.py), consistent with the LVM mode.
Ronen Friedman [Mon, 9 Mar 2026 17:23:18 +0000 (17:23 +0000)]
crimson/osd: use a unified super-block for devices
This commit refactors the on-hardware super-block structure
used by the seastore to a unified format that
can accommodate all three device types (HDD, ZBD, RBM).
All devices now have a 60 bytes header at address 0,
similar to the existing BlueStore layout. A 23-byte magic
string ("CRIMSON_DEVICE") is placed at the beginning of
the header, followed by 37 bytes of null padding (to
match the existing 60 bytes of the super-block), and
then the DENC-encoded device_superblock_t structure starting
at offset 60.
A unified device_config_t is now used for all device types.
The per-shard data structure is also unified, now including a union
of all relevant fields for each device type.
We are also adding a check for the super-block magic value in the
RBMDevice::read_rbm_superblock() method, similar to the existing check
in SegmentManager::read_segment_manager_superblock().
Presigned URLs using SigV2 do not contain x-amz-credential causing the
log record field Authentication Type to be incorrectly set to '-'.
This has been fixed to check for the presence of the x-amz-expires and Expires
parameters instead.
John Mulligan [Thu, 23 Apr 2026 21:37:28 +0000 (17:37 -0400)]
CODEOWNERS: add an smb group for various smb related files
Add a new smb group that covers parts of orch that manage smb as well as
the cephfs proxy. This will help automatically notify smb focused devs
on PRs.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
rgw: read_obj_policy() consults s3:prefix when deciding between 403/404
when read_obj_policy() gets ENOENT, it only returns 404 NoSuchKey if the
requester has s3:ListBucket permission. however, policy that allows
s3:ListBucket may be conditional on the s3:prefix to restrict listings
to certain paths/object names. add the requested object name to the iam
environment as s3:prefix to match aws behavior here
* refs/pull/65656/head:
client: do not allow zero‑length reads
src/test: test zero-length async-fsync read using ceph_ll_nonblocking_readv_writev
src/test: test zero-length async-fsync read using ll_preadv_pwritev
Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Christopher Hoffman <choffman@redhat.com>
Generalize prepare_rewrite_publish_to_prior() into stage_visibility_handoff().
* introduce should_use_no_conflict_publish
* Replace is_rewrite_transaction() checks with should_use_no_conflict_publish(),
so adding new no-conflict users becomes straightforward.
* Stop committing metadata (commit_state + sync_checksum) during prepare_record()
(pre-commit). While it is correct for rewrite, doing it pre-commit doesn't buy
us anything today because readers are still blocked until the publish finishes.
Moving metadata commit to the after commit phase would also make future
non-rewrite users easier to support.
This is a prep step for expanding no-conflict publish coverage.
projects / ceph.git / log