crimson/osd:  add warning log for stray pgmeta-like objects

Add a warning log when a stray pgmeta-like object is found during
PG removal so that related error conditions can be detected earlier.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: handle duplicate keys in LogNode::remove_entry

Previously, LogNode::remove_entry returned early when a log_key was
found, assuming uniqueness. However, duplicate keys can exist in the
node if an older entry was previously removed.
This commit also adds a unit test to verify this scenario.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: reload head if modified

This commit also fixes the test case to verify that
the head is correctly allocated and updated
during omap_set_keys operations involving multiple keys.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore, osd/PGLog: handle omap_iterate retry to avoid duplicate entries

Seastore omap_iterate may retry internally on conflicts, which can
cause PGLog to process the same entries multiple times when entries
are handled directly in the iteration callback.

Introduce a conflict hook in omap_iterate so callers can reset
iteration state on retry. PGLog now buffers entries during iteration and
applies process_entry() only after a successful pass, clearing the buffer
on retry to avoid duplicates.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: ensure data integrity with deep copy in omap_get_value

Previously, omap_get_value could return a bufferlist pointing to
memory without guaranteed lifetime. This patch introduces LogNode::copy_t
to distinguish between DEEP and SHALLOW copies.

- Default get_value to DEEP copy for external safety.
- Use SHALLOW copy in internal paths (e.g., remove_kv) to maintain performance.
- Refactor LogManager::omap_get_value to simplify coroutine flow.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: ensure extent is loaded if missing from cache

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: correct UPPER_BOUND behavior to start looking up items exceeding the target

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: fix crash by checking for nullopt in search input

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: return correct result for non-logkey entries

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: use do_with to avoid stack-use-after-return

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

osd/PGLog: set dirty_to to max to avoid partial writes

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: add test for log_manager

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: support for large kv pair in LogNode

Each log_key_t contains a chunk_idx field to manage values
that span multiple LogNodes when the value size exceeds the
maximum capacity of a single LogNode.
See detailed description in log_manager.h.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: optimize handling of batched requests

During 4KB random write workloads, SeaStore  receives
batched dup_* entries in both omap_set_keys.
This change enables efficient batch processing of these
requests to reduce overhead.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: introduce omap_rm_keys interface in omap_manager

Deletion of pg_log_entry_t entries is performed by omap_rm_keys using a set.
For example, omap_rm_keys might be called with a set containing
pg_log_entry_t entries ranging from 0011.0001 to 0011.0010.
In this case, calling omap_rm_key individually for each entry is inefficient,
because each call triggers a traversal of the entire list.

To avoid this, omap_rm_keys with a set is introduced in omap_manager
to handle removal request more efficiently.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: adjust select_log_omap_root to return a single root

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: invoke managers depending types based if branch

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: remove duplicate keys for non-log entries

When writing a non-log key, remove any existing duplicate key
before inserting the new KV pair. With this change, full list
traversal is no longer required during remove_kv.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: make _fastinfo overwritable to minimize space overhead

This commit forces _fastinfo to be stored at the last position of a LogNode.
By doing so, _fastinfo can be overwritten by the next pg_log_entry.

Since _fastinfo has a fixed key with varying contents and is included in
every write transaction, placing it at the tail enables efficient overwrites.

As a result, this change reduces LogNode allocation and deallocation,
thereby lowering space overhead. Moreover, garbage collection for obsolete
key-value pairs is unnecessary due to overwrite semantics.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: add log_manager to handle pgmeta_oid's key-value pairs separately

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: remove log_root to maintain a single root for OMAP and LOG

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

Merge pull request #67716 from rhcs-dashboard/fix-namespace-issue

mgr/dashboard: fix add host API path for namespace mapping in NVMe/TCP

Reviewed-by: Puja Shahu <pshahu@redhat.com>
Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67769 from rhcs-dashboard/fix-nvmeof-subsystem-create-firefox-next

mgr/dashboard: fix-nvmeof-subsystem-create-firefox-next

Reviewed-by: Puja Shahu <pshahu@redhat.com>
Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67810 from gbregman/main

mgr/dashboard: Add "connection get_io_statistics" to NVMeoF CLI

Merge pull request #67742 from VallariAg/wip-nvmeof-upgrade-fix

qa: Add nvmeof upgrade from v20.2.0 and tentacle

Merge pull request #67812 from bluikko/wip-doc-cephadm-spelling2

doc/cephadm: Fix more spelling errors

Merge pull request #67783 from phlogistonjohn/jjm-bwc-configure-args

script/build-with-container: add CONFIGURE_ARGS env var to configure step

mgr/dashboard: Add "connection get_io_statistics" to NVMeoF CLI

Fixes: https://tracker.ceph.com/issues/74964
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #67122 from Naveenaidu/wip-naveen-fix-RemoteReservationRevoked-race-condition

osd/PeeringState: handle race condition of RemoteReservationRevoked event for Backfilling state

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #67754 from idryomov/wip-kclient-features-update

include/ceph_features: note more kernel versions

Reviewed-by: Miki Patel <miki.patel132@gmail.com>

Merge pull request #67244 from bill-scales/issue73891

osd: Avoid pwlc spanning intervals

Reviewed-by: Samuel Just <sjust@ibm.com>
Reviewed-by: Radoslaw Zarzynski<rzarzyns@redhat.com>
Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #62227 from atta/fix-lua-mkdtemp-permissions

rgw: fix luarocks directory permissions

Merge pull request #67436 from adk3798/cephadm-fix-mon-crush-loc-text

qa/cephadm: fix test_set_mon_crush_locations test

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #67776 from bluikko/wip-doc-radosgw-spelling

doc/radosgw: Fix spelling errors

doc/cephadm: Fix more spelling errors

And other such minor obvious issues, including a spelling error
introduced in the previous commit 2565579caa1a118e9032283b55e969f9badcd6b6

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #67619 from tchaikov/wip-cmake-cephfs-journal-tool-without-liburing

cmake: link uring::uring to Boost::boost and fix cephfs tools linkage

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>

Merge pull request #67708 from rhcs-dashboard/fix-miscellaneous

mgr/dashboard: show System metadata used

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #66450 from rkachach/fix_issue_74015

mgr/cephadm: Fix mgmt-gateway default port in get_port_start()

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #67530 from rkachach/fix_ipv6_with_brackets

cephadm: fix EndPoint to handle bracketed IPv6 addresses

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #67690 from rhcs-dashboard/75322-Updating-empty-state-message-on-performance-chart

mgr/dashboard: Fixing message when prometheus is disabled in performa…

Reviewed-by: Afreen Misbah <afreen@ibm.com>

doc/radosgw: Fix spelling errors

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #67421 from ifed01/wip-ifed-raise-min-comp-size-hdd

options/global: raise bluestore_compression_min_blob_size_hdd to 64K

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #67695 from rkachach/fix_issue_74860

qa: fixing cephadm mgmt-gateway test to remove openssl dependency

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

Merge pull request #67433 from ifed01/wip-ifed-fix-min-blob-size-tracking

os/bluestore: track compression_*blob_size* parameters for online update

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

mgr/cephadm: Fix mgmt-gateway default port in get_port_start()

The mgmt-gateway port was already defaulted to 443 in most places, but
get_port_start() did not apply this default. Since the output of
get_port_start() is used both to configure the daemon ports which are
later used to to open them in firewalld, this inconsistency meant the
HTTPS port was not opened when firewalld service was active.

This change makes get_port_start() also default to port 443, ensuring
the daemon is configured correctly and the corresponding firewalld port
is opened as expected.

Fixes: https://tracker.ceph.com/issues/74015
Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/dashboard: show miscellaneous data used

Fixes: https://tracker.ceph.com/issues/75320
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #64975 from liu-chunmei/shard_osd_final

osd shard changes upon restart

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #65912 from MattyWilliams22/wip-dont-delete-objects

test: Add option to not delete created objects

qa: Add nvmeof upgrade from v20.2.0

Also update install task for nvmeof/upgrade test

Fixes: https://tracker.ceph.com/issues/75453
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

mgr/dashboard: Fixing message when prometheus is disabled in performance charts
Fixes: https://tracker.ceph.com/issues/75322
Signed-off-by: Devika Babrekar <devika.babrekar@ibm.com>

Merge pull request #67734 from rhcs-dashboard/fix-namespace-size

mgr/dashboard: Namespace size should accept GiB in NVMe/TCP

Reviewed-by: Puja Shahu <pshahu@redhat.com>
Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67782 from rkachach/fix_issue_75492

mgr/nvmeof: Adding missing CLICommand field to nvmeof mgr module

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #66580 from kamoltat/wip-ksirivad-fix-72994

mon [stretch-mode]: Allow a max bucket weight diff threshold
Reviewed-by: Ronen Friedman <rfriedma@ibm.com>

Merge PR #67780 into main

* refs/pull/67780/head:
Revert "Merge PR #67630 into main"

Reviewed-by: Shraddha Agrawal <shraddhaag@ibm.com>

mgr/nvmeof: Adding missing CLICommand file to nvmeof mgr module

Fixes: https://tracker.ceph.com/issues/75492
Signed-off-by: Redouane Kachach <rkachach@ibm.com>

script/build-with-container: add CONFIGURE_ARGS env var to configure step

Add a new optional CONFIGURE_ARGS environment variable to the configure
step so that there's a mechanism to pass custom cmake options that
aren't handled elsewhere in the run-make.sh script.

Because configure is a rather fundamental build step it's probably
preferable to set this via an env file so that it persists across
rebuilds. Using an environment var here also avoids both needing to
change run-make.sh or add another CLI option to BWC which already has
too many.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Revert "Merge PR #67630 into main"

This reverts commit 3a5e4524aa56de4c26400ccf994baa6ba8e16d9e, reversing
changes made to d334ff531c563bb7d0e37777f606322ec91b7453.

To everyone's surprise, skipping a workflow does not make it less
required. Well done Github!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #67275 from ifed01/wip-ifed-fix-bluefs-expand-test

qa/standalone: fix/improve bluefs tests

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #67609 from ifed01/wip-ifed-bluefs-stats-reset

os/bluestore: add 'bluefs stats reset' admin socket command.

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #67770 from bluikko/wip-doc-cephadm-spelling

doc/cephadm: Fix spelling errors

Merge pull request #67718 from rhcs-dashboard/fix-subsystem-create-layout-issue

mgr/dashboard: Footer actions shift upward instead of staying pinned at modal bottom in NVMe/TCP subsystem create wizard

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Devika Babrekar <devika.babrekar@ibm.com>

doc/cephadm: Fix spelling errors

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #65405 from rhcs-dashboard/notification-store-events

mgr/dashboard: Add restore events in notification screen

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge pull request #67760 from gbregman/main

mgr/dashboard: Add secure and verify-host-name to "listener add" on NVMeoF CLI

Merge pull request #67647 from rhcs-dashboard/fix-75317-main

mgr/dashboard: update onboarding screen as per design

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67713 from rhcs-dashboard/fix-nvmeof-initiator-add-visibility

mgr/dashboard: Initiator add shows success but host is not added/displayed in Subsystem Initiators table

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: pujaoshahu <pshahu@redhat.com>

mgr/dashboard: fix-nvmeof-subsystem-create-firefox-next

Fixes: https://tracker.ceph.com/issues/75434
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>

mgr/dashboard: rename expand-cluster to add-storage

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: update onboarding screen as per design

Fixes: https://tracker.ceph.com/issues/75317
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

qa/suites/crimson-rados: add fio test case for osd shard number changes upon restart for 3 osd

Signed-off-by: Chunmei Liu <chunmei.liu@ibm.com>

doc/dev/seastore.rst: add design implementation for osd shards change

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/common/options: add seastore_require_partition_count_match_reactor_count in crimson.yaml.in

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/osd/osd_admin: add osd command to dump store shards info

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/os/seastore: support other devices

Signed-off-by: Chunmei Liu <chunmei.liu@ibm.com>

test/crimson/seastore: using store_index = 0 for the tests

Signed-off-by: Chunmei Liu <chunmei.liu@ibm.com>

crimson/tools: fixing tools according to osd shards number change modification

Signed-off-by: Chunmei Liu <chunmei.liu@ibm.com>

crimson/os/seastore: make register_metrics works for
 multiple store shards on one reactor

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/osd: replace store call by with_store call in case need remote store calling.

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/osd/shard_services: get multiple store shards for per local state, and use store index to create pg mapping

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/osd/pg_map: add pg mapping policy for osd shards number is different with store shards number

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/os/futurized_store: support cross core store calling

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/os/alienstore: support multiple store shards on each reactor

Signed-off-by: Chunmei Liu <chunmei.liu@ibm.com>

crimson/os/cyanstore: create multiple store shards on each reactor

note: src/stop.sh should wait enought time before kill the crimson-osd
in case cyanstore can't write meta data to disk.

Signed-off-by: chunmei liu <chunmei.liu@ibm.com>

crimson/os/seastore: create multiple device shards and store shards on each reactor.

Signed-off-by: Chunmei Liu <chunmei.liu@ibm.com>

Merge pull request #67396 from Rotemrs/lua-background-vm-fix

rgw/lua: create fresh VM for each background script execution

mgr/dashboard: Add secure and verify-host-name to "listener add" on NVMeoF CLI.
Also add missing "manual" field in "listener list".

Fixes: https://tracker.ceph.com/issues/75447
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #67638 from nbalacha/wip-nbalacha-75306

rgw/lua: fix a crash when D4N is enabled

Merge pull request #67660 from kshtsk/wip-keystone-2025.2

qa/tasks/keystone: upgrade keystone to 2025.2

include/ceph_features: note more kernel versions

Despite both MONNAMES and MONENC being pre-argonaut feature bits and
the kernel client implicitly assuming argonaut since 5.0, its monmap
decoding routine didn't handle MONNAMES and MONENC until 5.11 (when it
became necessary as part of msgr2 support).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #67712 from afreen23/landing-page-fixes

mgr/dashboard: Fix scrubbing state

Reviewed-by: Devika Babrekar <devika.babrekar@ibm.com>

Merge pull request #67714 from afreen23/overview-breaking

mgr/dashboard: Fix breaking overview page

Reviewed-by: Devika Babrekar <devika.babrekar@ibm.com>

Merge pull request #66245 from athanatos/wip-sjust-seastore-conflict

crimson/seatore: rework lba_manager to use LBACursor rather than LBAMapping

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #67739 from dang/wip-dang-posix-readme

Update the POSIXDriver readme to current state

doc: Update the POSIXDriver readme to current state

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #67451 from Ericmzhang/wip-mon-colocate

mon: Health warning for colocated monitors

Merge pull request #67641 from Hezko/revive-nvme-module

introduce nvme module again

Merge pull request #65626 from samarahu/wip-d4n-remove-bucket

rgw/d4n: Implement bucket check_empty and remove methods

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

qa: Add "auto_pool_create" to nvmeof_initiator

While deploying gateways with "ceph orch apply nvmeof",
--pool can be optional now. If not passed, a pool with
name ".nvmeof" would automatically be created.

In nvmeof task, "auto_pool_create: True" would skip --pool
in "ceph orch apply nvmeof".

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

mgr/nvmeof: add missing CLICommand to the module

Fixed AttributeError: type object 'NVMeoF' has no attribute 'CLICommand'

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #67659 from kamoltat/wip-ksirivad-fix-70320

qa: make test_progress atomically capture OSD marked in/out events
Reviewed-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>