crimson/os/seastore: ensure extent is loaded if missing from cache

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: correct UPPER_BOUND behavior to start looking up items exceeding the target

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: fix crash by checking for nullopt in search input

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: return correct result for non-logkey entries

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: use do_with to avoid stack-use-after-return

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

osd/PGLog: set dirty_to to max to avoid partial writes

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: add test for log_manager

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: support for large kv pair in LogNode

Each log_key_t contains a chunk_idx field to manage values
that span multiple LogNodes when the value size exceeds the
maximum capacity of a single LogNode.
See detailed description in log_manager.h.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: optimize handling of batched requests

During 4KB random write workloads, SeaStore  receives
batched dup_* entries in both omap_set_keys.
This change enables efficient batch processing of these
requests to reduce overhead.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: introduce omap_rm_keys interface in omap_manager

Deletion of pg_log_entry_t entries is performed by omap_rm_keys using a set.
For example, omap_rm_keys might be called with a set containing
pg_log_entry_t entries ranging from 0011.0001 to 0011.0010.
In this case, calling omap_rm_key individually for each entry is inefficient,
because each call triggers a traversal of the entire list.

To avoid this, omap_rm_keys with a set is introduced in omap_manager
to handle removal request more efficiently.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: adjust select_log_omap_root to return a single root

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: invoke managers depending types based if branch

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: remove duplicate keys for non-log entries

When writing a non-log key, remove any existing duplicate key
before inserting the new KV pair. With this change, full list
traversal is no longer required during remove_kv.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: make _fastinfo overwritable to minimize space overhead

This commit forces _fastinfo to be stored at the last position of a LogNode.
By doing so, _fastinfo can be overwritten by the next pg_log_entry.

Since _fastinfo has a fixed key with varying contents and is included in
every write transaction, placing it at the tail enables efficient overwrites.

As a result, this change reduces LogNode allocation and deallocation,
thereby lowering space overhead. Moreover, garbage collection for obsolete
key-value pairs is unnecessary due to overwrite semantics.

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: add log_manager to handle pgmeta_oid's key-value pairs separately

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore: remove log_root to maintain a single root for OMAP and LOG

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

Merge pull request #66544 from NitzanMordhai/wip-nitzan-encoder-test-backward-incompability-checks

test/encoding/readable: Add backward incompat checks

Merge pull request #67287 from nbalacha/wip-nbalacha-71390

rgw/configstore:  don't reinitialize the rados client in RadosRealmWatcher

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #66455 from cfanz/wip-cfanz-fix-rgw-counter-overflow

rgw: fix overflow of outstanding counter in SimpleThrottler

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67306 from cbodley/wip-rgw-rest-client-strftime

rgw/multisite: use libfmt to format Date header

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #67112 from rhcs-dashboard/cephfs-module-enable

mgr/dashboard: add CephFS Mirroring enablement page

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #66484 from samarahu/d4n-remove-erase-from-update

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #67348 from VallariAg/wip-nvmeof-udisks-disable

qa: Fix coredumps caused by udisks

Merge pull request #67370 from kotreshhr/qa-mirror-flake8-fix

qa/test_mirroring: Fix flake8 errors

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/test_mirroring: Fix flake8 errors

Introduced-by: c1e827247bd20e8a1851bc2d7a9861c12d033ef0
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #67231 from guits/ceph-volume-inventory-ls-all

ceph-volume: include LVM mapper devices in get_devices()

Merge pull request #67110 from FredNass/patch-2

doc/radosgw: rgw_lua_max_memory_per_state defaults to 128K (not 500K)

Merge pull request #66967 from rhcs-dashboard/gateway-add-modal

mgr/dashboard: gateway-add-modal

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67305 from kotreshhr/qa-mirror

qa: Add retry logic to remove most sleeps in mirroring tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #66912 from idryomov/wip-74394

osd/PrimaryLogPG: encode an empty data_bl for empty sparse reads

Merge pull request #66894 from tchaikov/wip-ec-isa-fix-cache-collision

erasure-code/isa: fix cache collision causing buffer overflow

Merge pull request #66376 from NitzanMordhai/wip-nitzan-self-test-influx-set-hostname

qa/tasks/mgr: test_module_selftest set influx hostname to avoid warnings

Merge pull request #62067 from ljflores/wip-tracker-67179

osd: add pg-upmap-primary to clean_pg_upmaps

Merge pull request #67353 from idryomov/wip-daemonwatchdog-unbound

qa/tasks/daemonwatchdog: fix unbound variable in bark_reason message

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #67351 from idryomov/wip-74712

qa: krbd_rxbounce.sh: do more reads to generate more errors

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #65318 from tchaikov/wip-mgr-progress-cleanup

pybind/mgr/progress: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/tasks/daemonwatchdog: fix unbound variable in bark_reason message

Introduced in commit 783f0e3a9903 ("qa: Adding a new class for the
daemonwatchdog to monitor").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd_rxbounce.sh: do more reads to generate more errors

On faster hardware having each thread do 1024 reads isn't always
sufficient for the "two orders of magnitude" threshold that is used in
the test.

Fixes: https://tracker.ceph.com/issues/74712
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #67218 from jamiepryde/jerasure-unit-test-big-endian-fix

test/erasure-code: Use memory comparisons that are safe on big endian arch

qa: Fix coredumps caused by udisks

Also redo commands to install nvme 2.13.
Fixes: https://tracker.ceph.com/issues/74922
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

mgr/dashboard: add CephFS Mirroring enablement page

Fixes: https://tracker.ceph.com/issues/74633
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

Merge pull request #67240 from guits/cv-raw-list-perf

ceph-volume: single lvs call to speed up exclude_lvm_osd_devices

Merge pull request #66646 from ujjawal-ibm/ceph-volume-format-device

Add NVMe preformat support to skip redundant mkfs discard

Merge PR #66183 into main

* refs/pull/66183/head:

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Shachar Sharon <ssharon@redhat.com>

Merge pull request #66606 from shraddhaag/add-logs

PeeringState::needs_recovery(): log first missing item

Merge pull request #66352 from tchaikov/debian-invoke-rc.d

debian: remove invoke-rc.d calls from postrm scripts

Merge pull request #66591 from NitzanMordhai/wip-nitzan-pickup-object-corpus-20.2.0-380-gdbcbbd3f281

ceph-object-corpus: pick up 20.2.0-380-gdbcbbd3f281

Merge pull request #66553 from smanjara/wip-fix-object-sync-enoent

rgw: retry ENOENT if bucket reshard races with index operation

Merge pull request #66721 from swjtu-zhanglei/main-wip

rgw: refact by encapsulating io_context_pool to enforce lazy initialization

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #66567 from mertsunacoglu/wip-prerequest-auth

RGW: Change prerequest hook to run after authorization process

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #66509 from morphes1995/rgw_qos_fix

rgw: ops large budget caused by rate limiter was enabled and max_ops …

Reviewed-by: Mark Kogan <mkogan@ibm.com>

qa: Fix test_cephfs_mirror_remote_snap_corrupt_fails_synced_snapshot

Fixes: https://tracker.ceph.com/issues/74896
Signed-off-by: Kotresh HR <khiremat@redhat.com>

qa: Add retry logic to remove most sleeps in mirroring tests

The mirroring tests contain lot of sleeps adding it up to ~1hr.
This patch adds a retry logic and removes most of them. This
is cleaner and saves considerable time in test time for mirroring.

Fixes: https://tracker.ceph.com/issues/74878
Signed-off-by: Kotresh HR <khiremat@redhat.com>

ceph-volume: skip redundant NVMe mkfs discards

- Avoid redundant discard during mkfs when discard is disabled
- Reduces mkfs time on large NVMe devices by skipping long running discard operations

Fixes: https://tracker.ceph.com/issues/74908
Signed-off-by: Ujjawal Anand <ujjawal.anand@ibm.com>

Merge pull request #67264 from guits/fix-74818

ceph-volume: pass osd_type value to ceph-volume

Merge pull request #66594 from cbodley/wip-qa-rgw-account-migration

qa/rgw: add test_account_migration workunit

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>

Merge PR #66657 into main

* refs/pull/66657/head:
developer: Add name to maps

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #67309 into main

* refs/pull/67309/head:
doc: correct spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #67078 into main

* refs/pull/67078/head:
doc: Fixes a spelling error on the RATIO for PG scaling recommendation.

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #66526 from gardran/wip-gardran-show-sharding

os/bluestore: show RocksDB sharding information

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge PR #67320 into main

* refs/pull/67320/head:
.github/workflows: use main branch for config diff

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Naveen Naidu <naveen.naidu@ibm.com>

Merge pull request #66821 from rhcs-dashboard/cephfs-mirroring-filesystem

mgr/dashboard: Cephfs mirroring Filesystem Selection

Reviewed-by: Naman Munet <nmunet@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #67044 from Hezko/nvme-submodule-to-1-6-5

mgr/dashboard: bump nvmeof submodule to 1.6.5

Merge pull request #66929 from mkogan1/wip-t74245

ceph-volume: fix build on RHEL 10/python 3.12

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

mgr/dashboard: use gray10 theme colors

- sidenav, header , table - white
- background - gray

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: add-gateway-nodes

Fixes: https://tracker.ceph.com/issues/74335
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>
:wq

Merge pull request #66906 from rhcs-dashboard/gateway-resources

mgr/dashboard: NVme-gateway-resource

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Naman Munet <nmunet@redhat.com>

Merge PR #67316 into main

* refs/pull/67316/head:
doc/dev/release-checklist: add release to redmine-upkeep trigger
.github/workflows/redmine-upkeep: run for release branches

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

.github/workflows: use main branch for config diff

This script does not exist for the "head" reference of backport PRs.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

qa/rgw: add test_account_migration workunit

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67307 from leonidc/fix-tests-after-beacon-diff

 qa: fixing the qa test in mon/misc.sh after adding beacon-diff feature

doc/dev/release-checklist: add release to redmine-upkeep trigger

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

.github/workflows/redmine-upkeep: run for release branches

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #67011 into main

* refs/pull/67011/head:
qa/multisite: use boto3's ClientError in place of assert_raises from tools.py.
qa/multisite: test fixes
qa/multisite: boto3 in tests.py
qa/multisite: zone files use boto3 resource api
qa/multisite: switch to boto3 in multisite test libraries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: correct spelling

I saw a typo so I fixed it

Signed-off-by: Amnay <amnay.m@gmail.com>

libcephfs_proxy: log more information about the negotiation

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: implement client support for batched readdir

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: create internal ceph_dir_result structure

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: add the number of supported operations to negotiation

The v0 negotiation structure has been modified to hold the total number
of operations and callbacks supported by the peer. The changes are done in
a way that it's completely transparent and harmless for a peer expecting
the previous definition.

This will be useful to quickly check if the daemon supports some
operation, or the client supports some callback before sending them.

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: implement daemon support for batched readdir

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: define protocol for batched readdir

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: cleanup unneded variable

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

libcephfs_proxy: fix include dependency

Signed-off-by: Xavi Hernandez <xhernandez@gmail.com>

 qa: fixing the qa test in mon/misc.sh after adding beacon-diff feature

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

mgr/dashboard: NVme-gateway-resource

Fixes: https://tracker.ceph.com/issues/74334
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>

rgw/multisite: use libfmt to format Date header

as noted in https://tracker.ceph.com/issues/63973, std::strftime() is
not thread-safe and may produce garbage values. use libfmt instead

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67290 from shraddhaag/wip-shraddhaag-cephadm-crimson-tests

cephadm, ceph-volume: add tests for crimson OSD support

Merge pull request #66255 from rkachach/fix_issue_73853

mgr/cephadm: don't remove TLS certs if svc still has daemons on host

Reviewed-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

mgr/dashboard: Cephfs Mirroring - Filesystem Selection

Fixes: https://tracker.ceph.com/issues/74280
Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge PR #65538 into main

* refs/pull/65538/head:

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Shachar Sharon <ssharon@redhat.com>

Merge pull request #67182 from Kh9705/add-s3select-limit-examples

doc/radosgw/s3select: added SQL LIMIT operator examples

Merge pull request #67104 from rhcs-dashboard/header-page-component

mgr/dashboard: add page header component

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge pull request #67059 from bluikko/wip-src-script-backport-fixes

ceph-backport.sh: clean function argument use and update GitHub help link

Merge pull request #67202 from tchaikov/wip-ec-isa-cleanup

erasure-code/isa: eliminate redundant map lookups

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #67128 from afreen23/subsystem-step-2

mgr/dashboard: Step two subsystem creation form

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: pujaoshahu <pshahu@redhat.com>

mgr/dashboard: bump nvmeof submodule to 1.6.5

update proto files and gateway submodule
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>
tomer

Merge pull request #67174 from bluikko/wip-fix-74721-trailing-space

mgr/dashboard: remove trailing space from directory name

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: pujaoshahu <pshahu@redhat.com>

d4n/policy: Remove erase call from update

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

rgw: do not re-init Rados in RadosRealmWatcher

The RadosRealmWatcher::watch_start() function incorrectly
reinitialized a rados client that had already been initialized during
the creation of the ConfigStore. This caused the original Objecter
to be lost, which was subsequently reported as a memory leak by Valgrind.

Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>

Merge pull request #66092 from kshtsk/wip-fix-lua-kernel-check

qa/cephfs: lua to respect missing kernel in yaml