Merge PR #55616 into main

* refs/pull/55616/head:
PendingReleaseNotes: add note for replay completion warning
qa: test to verify `MDS_ESTIMATED_REPLAY_TIME` warning
doc: add a note for `MDS_ESTIMATED_REPLAY_TIME` MDS warning
mds: emit warning for estinated replay time

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #60640 into main

* refs/pull/60640/head:
qa: use a larger timeout for kernel_untar_build workunit

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #60988 from thuvh/feat/kafka_multiple_brokers

rgw: allow bucket notification send message to kafka with multiple brokers

Merge pull request #61144 from cyx1231st/wip-seastore-improve-backref

crimson/os/seastore/cache: improve committing allocations by backref entries

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Zhang Song <zhangsong02@qianxin.com>

Merge PR #60653 into main

* refs/pull/60653/head:
mds: do not process client metrics message with fast dispatch

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #61143 from kotreshhr/qa-nfsganesha-fio-test

qa: Add async io test to nfs ganesha using fio

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55145 from mchangir/mgr-snap_schedule-handle-volume-delete

mgr/snap_schedule: handle volume delete

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw: allow bucket notification send message to kafka with multiple brokers

- add new parameter to define list of broker
- update document
- change nose to nose-py3
- add test case for multiple brokers (happy case)

Signed-off-by: Hoai-Thu Vuong <thuvh87@gmail.com>

Merge pull request #61177 from Skalador/wip-doc-typo-hardware-controllers

doc/start: Fixes a typo in controllers section of hardware recommendations

doc: Fixes a typo in controllers section of hardware recommendations

Signed-off-by: Kevin Niederwanger <k.niederwanger@gmail.com>

Merge pull request #59116 from rzarzynski/wip-constbl-compressor

compressor: switch out of get_current_ptr() for the sake of const-stricter bufferlist

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #60808 from ronen-fr/wip-rf-reservStat2

osd/scrub: show reservation status in 'pg dump' output

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd/scrub: show reservation status in 'pg dump' output

Whenever a PG is selected for scrubbing, and is waiting for
remote reservations, the 'pg dump' output will include the
following text (under the 'SCRUB_SCHEDULING' column):
Reserving. Waiting Ns for OSD.k (n/m)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os/seastore/cache: move committing existing blocks to prepare_record()

So the 2 steps of remapping are atomically committed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore/journal: hide RBM specific finish_commit()

By introducing a callback upon completing submission.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: minor cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60821 from Matan-B/wip-matanb-crimson-seastore-deafult

crimson: switch to SeaStore as default object store

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61155 from idryomov/wip-69324

rbd: handle --{group,image}-namespace in "rbd group image {add,rm}"

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #61150 from idryomov/wip-69319

rbd: open images in read-only mode for "rbd mirror pool status --verbose"

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #61129 from idryomov/wip-68998

librbd: avoid data corruption on flatten when object map is inconsistent

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #61125 from ljflores/wip-release-note

Merge PR #61149 into main

* refs/pull/61149/head:
script/ceph-backport: allow hyphen in project during ceph fork deduction

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #60777 from VallariAg/wip-nvmeof-prometheus-rbd-image-reused

monitoring: Add prometheus alert NVMeoFMultipleNamespacesOfRBDImage

Reviewed-by: Afreen Misbah <afreen@ibm.com>

rbd: drop --pool option from "rbd group image {add,rm}"

It stopped working with removal of get_special_pool_group_names() in
commit 3e8624f157a1 ("rbd: add support for namespaces") over six years
ago.  Given how much time has passed, stop accepting this option.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61146 from zdover23/wip-doc-2024-12-19-radosgw-uadk-accel

doc/radosgw: edit uadk-accel.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rbd: handle --{group,image}-namespace in "rbd group image {add,rm}"

Currently only passing the namespace as part of the group or image spec
works.  If --group-namespace or --image-namespace options are used, the
namespace isn't picked up.

Fixes: https://tracker.ceph.com/issues/69324
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

PendingReleaseNotes: add note about tracker #69012

We merged a fix for v19.2.1 that helps alleviate
the worst of this problem (https://tracker.ceph.com/issues/68657),
but it still comes up on occasion. This release note addresses the
remaining issues tracked in https://tracker.ceph.com/issues/69012.

Signed-off-by: Laura Flores <lflores@ibm.com>

rbd: open images in read-only mode for "rbd mirror pool status --verbose"

This is cleaner and makes the command run a bit faster because watches
won't be established.

Fixes: https://tracker.ceph.com/issues/69319
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

script/ceph-backport: allow hyphen in project during ceph fork deduction

This will allow the fork deduction code to recognize project names
that include a hyphen, such as:

    git@github.com:USERNAME/ceph-fork.git

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #61113 from leonidc/wip-leonidc_no_listener_fix

nvmeofgw*: fix no-listeners

Merge pull request #61107 from cbodley/wip-69241

rgw/posix: std::ignore return value of write()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59703 from rhcs-dashboard/usermanagement-form

mgr/dashboard: carbonize user management form

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #61130 from baum/wip-baum-20241218-00

src/nvmeof/NVMeofGwMonitorClient: remove MDS client, not needed

doc/radosgw: edit uadk-accel.rst

Edit the sections of doc/radosgw/uadk-accel.rst that concern the
automatic and manual building of UADK.

This is one in a series of uadk-accel.rst-related changes that includes
the following PRs:
https://github.com/ceph/ceph/pull/60953
https://github.com/ceph/ceph/pull/61128

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61128 from zdover23/wip-doc-2024-12-18-radosgw-uadk-accel

doc/radosgw: edit uadk-accel.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

vstart: use crimson_osd_objectstore

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/dev/crimson: update SeaStore as default backend

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

vstart.sh: set alien cores with bluestore only

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: make use of crimson_osd_objectstore

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/os/seastore: make the updates to backref_entry_mset be consistent with extents

Generally, make alloc/free be consistent with add/commit/retire extents
in Cache.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

Merge pull request #60788 from oshrey16/tests-rgw-ps-list-topics-op

tests/rgw/notifications: Add tests for RGWPSListTopicsOp::execute()

qa: Add async io test to nfs ganesha using fio

Fixes: https://tracker.ceph.com/issues/69311
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #60431 from VallariAg/fix-nvmeof-ns-test

qa: fix nvmeof scalability and namespace test

Merge PR #61018 into main

* refs/pull/61018/head:
msg/async/Event: make the pipe byte static

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw/notifications: Add tests for RGWPSListTopicsOp::execute()

Tests:
Add comprehensive test cases to verify the behavior of `RGWPSListTopicsOp::execute()` under various scenarios:

    Migration case: Validate correct handling when `support_all_zones` is enabled, with v1 in a new state after migration and v2 topics present.
    v2 notification case: Ensure proper retrieval when v2 notifications are supported.
    v1 notification case: Verify fallback behavior when v2 notifications are unavailable.

Enhancements:
    Update `delete_all_topics` to handle v1 responses with the `result` key.

fixes: https://tracker.ceph.com/issues/68756
Signed-off-by: Oshrey Avraham <oshrey16@gmail.com>

Merge pull request #60771 from YaZasnyal/fix/complete-abort-mp

rgw/abortmp: Race condition on AbortMultipartUpload

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60676 from awojno-bloomberg/rgw-admin-json

rgw-admin: Add --format option for bucket sync status

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #60970 from clwluvw/redirect-qs

rgw: append query string to redirect URL if present

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60579 from clwluvw/empty-multidelete

rgw: return MalformedXML for empty objects list in DeleteObjects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #60217 into main

* refs/pull/60217/head:
CMakeLists.txt: do not compile {Objecter,Striper}.cc twice

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #61091 from leonidc/wip-leonidc_publish_rebalance_index

nvmeofgw* : fix publishing rebalance index

test/librbd: add TestInternal.FlattenInconsistentObjectMap

Inject an object map with all possible inconsistencies before
flattening to ensure that something similar to commit 40af4f87b64f
("librbd: flatten operation should use object map") doesn't reappear
in a different form.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: avoid data corruption on flatten when object map is inconsistent

By making flatten skip copyup in case the object is marked
OBJECT_EXISTS or OBJECT_EXISTS_CLEAN, commit 40af4f87b64f ("librbd:
flatten operation should use object map") introduced a critical
regression.  If the object map becomes inconsistent (e.g. because
flatten gets interrupted by killing "rbd flatten" process or a client
running on the clone crashes after updating the object map but before
writing to the image), the following attempt to flatten would corrupt
the clone if the copyup is actually still needed.

By design, it's impossible to tell whether the object is "known to
exist" based on the object map -- only telling whether the object is
"known to NOT exist" is possible (i.e. only OBJECT_NONEXISTENT state
is reliable).  Negating OBJECT_NONEXISTENT tells that the object "may
exist", not that the object is "known to exist".  This is reflected in
the name of object_may_exist() helper that was introduced together with
the object map implementation.  Something like object_may_not_exist()
simply can't be constructed given the rest of librbd.

This effectively reverts commits 4c86bccf07b8 ("librbd: add
object_may_not_exist helper") and 40af4f87b64f ("librbd: flatten
operation should use object map").

Fixes: https://tracker.ceph.com/issues/68998
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

monitoring: Add prometheus alert NVMeoFMultipleNamespacesOfRBDImage

NVMeoFMultipleNamespacesOfRBDImage alerts the user if a RBD image
is used for multiple namespaces. This is important alerts for cases
where namespaces are created on same image for different gateway group.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

src/nvmeof/NVMeofGwMonitorClient: remove MDS client, not needed

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

doc/radosgw: edit uadk-accel.rst

Incorporate Anthony D'Atri's suggested changes from
https://github.com/ceph/ceph/pull/60953 into doc/radosgw/uadk-accel.rst.

Two questions from that PR remain unclear to me: one is about whether
IOMMU should be disabled for performance on AMD EPYC systems, and the
other is about UADK. The note about UADK will be rewritten in improved
English in a near-future PR and any remaining technical questions that
involve it can be discussed in that PR.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60953 from zdover23/wip-doc-2024-12-04-radosgw-uadk-accel

doc/radosgw: edit uadk-accel.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

crimson/os/seastore: move backref_entry_t to a separate header

For reuse.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: rework backref_batch_update()

Leverage the fact that seq won't go backwards.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: introduce named factory methods for alloc/retire

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/seastore_types: use is_backref_mapped_type()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: don't mark allocations as L_ADDR_NULL

Which is used for deallocations.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: misc related cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: drop unused backref_entry_t::seq

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60976 from rhcs-dashboard/fix-69144-main

mgr/dashboard: Fix Latency chart data units in rgw overview page

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #61115 from ronen-fr/wip-rf-tst-p-replicated

qa/standalone/scrub: fix TEST_periodic_scrub_replicated

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #60863 from ronen-fr/wip-rf-deeperrors

osd/scrub: remove the 'has_deep_errors' scheduling flag

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #61122 from ceph/revert-61024-wip_ops-replayer_jfw-signedness-warning

Revert "ops_replayer: fix signed/unsigned warning and add parameter range checking."

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #56583 from pritha-srivastava/wip-rgw-del-bucket-tag

rgw: fix to correctly store updated attrs in backend store after erasing an attr/attrs for delete ops on a bucket.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60940 from rhcs-dashboard/rgw-user-accounts-api

mgr/dashboard: RGW user accounts CRUD api's

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Revert "ops_replayer: fix signed/unsigned warning and add parameter range checking."

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61024 from chardan/wip_ops-replayer_jfw-signedness-warning

ops_replayer: fix signed/unsigned warning and add parameter range checking.

Merge pull request #60942 from AliMasarweh/wip-alimasa-bucket-logging-filter

RGW/logging: add filtering for bucket logging

Reviewed-by: yuvalif<ylifshit@redhat.com>

Merge pull request #60697 from AliMasarweh/wip-alimasa-bucket-logging-flush

RGW/logging: add rest API to flush logging bucket

Reviewed-by: yuvalif<ylifshit@redhat.com>

qa/standalone/scrub: fix TEST_periodic_scrub_replicated

A bogus change introduced as part of PR#54363 (commit
fbb7d73) changed multiple 'scrub' commands to 'scheduled-scrub'.
In this one instance - that was wrong.

Fixes: https://tracker.ceph.com/issues/69276
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #59936 from anoopcs9/client-chown-statx-fixes

client: Handle empty pathnames for `ceph_chownat()` and `ceph_statxat()`

RGW\logging: convert s3_filter to key_filter

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

RGW/logging: add filtering for bucket logging

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

RGW/logging: add rest API to flush logging bucket

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

nvmeofgw*: fix no-listeners

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

nvmeofgw* : fix publishing rebalance index

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

Merge pull request #60798 from rhcs-dashboard/edit-configuration-at-runtime

mgr/dashboard: Administration > Configuration > rgw_dns_name and rgw_dns_s3website_name not updatable via dashboard

Reviewed-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Fix Latency chart data units in rgw overview page

Issue: The Latency chart in the rgw overview page shows incorrect data
unit as the unit that we are passing as an input to the dashboard-area
chart component is `ms` whereas the data that we get from the metrics is
in seconds. Due to this if we pass a value like 0.725s to
the dashboard chart component it will show the value in the chart as
0.725ms whereas it should be 725ms.

Fix: Pass the value in ms as expected to the dashboard area chart
component

Fixes: https://tracker.ceph.com/issues/69144
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #60807 from baum/nvmeof_service_ports

🧹cephadm/nvmeof: fix ports when default values are overridden

mgr/dashboard: RGW user accounts CRUD api's

Fixes: https://tracker.ceph.com/issues/69110
PR Changes:
Added CRUD & managing quota endpoints for RGW user accounts management with the unit-tests

Signed-off-by: Naman Munet <namanmunet@li-ff83bccc-26af-11b2-a85c-a4b04bfb1003.ibm.com>

Merge pull request #61106 from ivancich/wip-fix-hadoop-java

qa/rgw: force Hadoop to run under Java 1.8

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #59736 into main

* refs/pull/59736/head:
deb: Add cephfs.pc to libcephfs-dev
rpm: Add cephfs.pc to libcephfs-devel
cephfs: Add a pkgconfig file for libcephfs

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

cephadm/nvmeof: fix ports when default values are overridden

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

Merge pull request #60806 from baum/nvmeof_addr_map

cephadm/nvmeof: support per-node gateway addresses

rgw/posix: std::ignore return value of write()

/ceph/src/rgw/driver/posix/notify.h: In member function 'void file::listing::Inotify::signal_shutdown()':
/ceph/src/rgw/driver/posix/notify.h:215:19: error: ignoring return value of 'ssize_t write(int, const void*, size_t)' declared with attribute 'warn_unused_result' [-Werror=unused-result]
  215 |       (void) write(efd, &msg, sizeof(uint64_t));
      |              ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors

Fixes: https://tracker.ceph.com/issues/69241
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61031 from phlogistonjohn/jjm-rm-assigned-only

test/fio: remove assign-only variable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61030 from phlogistonjohn/jjm-remove-unused-var

test/cls_log: remove unused variable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Fix signed/unsigned warning and add parameter range checking.

Signed-off-by: Jesse F. Williamson <1643380+chardan@users.noreply.github.com>

qa/rgw: force Hadoop to run under Java 1.8

The Hadoop test installs Java 1.8 but then just runs the default
version. This makes sure it will run the version it installed.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #61005 from athanatos/sjust/wip-crimson-io-3

crimson: rework request pipeline to allow for per-object processing

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #56756 from clwluvw/head-bucket-stats

rgw: allow disabling bucket stats on head bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45384 from ifed01/wip-ifed-fragmentation-command

tool/ceph-bluestore-tool: fix wrong keyword for 'free-fragmentation' …

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

cephadm/nvmeof: support per-node gateway addresses

Added gateway and discovery address maps to the service specification.
These maps store per-node service addresses. The address is first searched
in the map, then in the spec address configuration. If neither is defined,
the host IP is used as a fallback.

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

Merge pull request #60429 from VVoidV/lbr-fix-formatter

<common> fix formatter buffer out-of-bounds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60499 from clwluvw/bucket-layout-output

rgw: fix json output for bucket layout

Reviewed-by: Casey Bodley <cbodley@redhat.com>