Merge pull request #56874 from smanjara/wip-shilpa-multisite-tests

qa/multisite: stabilize multisite testing

multisite/qa: resolve rebase conflicts

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

multisite/qa: disable sync error injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: disable two-zonegroup and three-zone configs.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/rgw/multisite: enable notification_v2 feature

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1e7fc66683e4dea7294f0ec95f47b22f48b33b7e)

qa/multisite: disable sync policy, endpoint and data_sync_init tests until fixed.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/rgw/multisite: add two-zone configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 14204442ac2eaaaa3f40a76403da49f0f4ed92c1)

qa/rgw/multisite: double cluster nodes and osd count

> WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand.

give each cluster 2 nodes instead of 1, and 6 osds instead of 3

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9a80cf4bf473c95c8d0b18fcdce88be20f74f973)

qa/rgw/multisite: reduce polling intervals from 20s to 5s

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit bc084c0ac64477abdbf54d96bc4981edf41e1343)

rgw/multisite: reconfigure checkpoint delay to 90s

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #57725 from Matan-B/wip-matanb-crimson-load-exist

crimson/osd/object_context_loader: Fix obc cache existence usage

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #57879 into main

* refs/pull/57879/head:
qa: add common conf links
qa: disable mon_warn_on_pool_no_app in fs suite

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57877 into main

* refs/pull/57877/head:
qa: correct json lookup for new `lock path` output

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57813 into main

* refs/pull/57813/head:
mds: avoid use of swap to initialize

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57887 from zdover23/wip-doc-2024-06-05-rados-config-network-config-ref

doc/rados: add options to network config ref

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #57641 from cbodley/wip-rgw-delete-objects-throttle

rgw: RGWDeleteMultiObj uses spawn_throttle for concurrency

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57702 from Suyashd999/fix-uam2

rados/cls_fifo_legacy: 'oid' used after it was moved

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

crimson/osd/object_context_loader: get_or_load to support atomicity

make use of try_lock in order to support atomicity when called in
ObjectContext::_with_lock()

Co-authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: make lock() atomic if doesn't need wait

Otherwise, promotion cannot be atomic with the 1st locker.

Identified by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/tri_mutex: minor cleanup to be consistent

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: add waiter_t::waiter_name

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: add debug logs

to be used only for testing

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: with_head_obc to log `existed`

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: cleanup `loaded`

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57696 from kotreshhr/fix_headcheck_first_damage

tools/first-damage: Make CEPH_NOSNAP int64

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

tools/first-damage: Make CEPH_NOSNAP int64

Make CEPH_NOSNAP int64 and pack 8 bytes when updating the omap
value.

Signed-off-by: Kotresh HR <khiremat@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56275 from smanjara/wip-set-attr-mtime

rgw/multisite: allow PutACL replication.

Merge pull request #57900 from zdover23/wip-doc-2024-06-06-start-intro-to-index

doc/start: s/intro.rst/index.rst/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/multisite: add object set acl test

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

doc/start: s/intro.rst/index.rst/

Change the filename "doc/start/intro.rst" to "doc/start/index.rst" so
that Sphinx finds the root filename for the "/start" directory in the
default location.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/multisite: allow modified object acls to be replicated

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #57015 from liangmingyuanneo/wip-bluefs-max-alloc-size

bluefs: bluefs alloc unit should only be shrink

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56428 from tchaikov/wip-crush-no-vla

crush: use std::vector instead of variable length arrays

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56389 from ronen-fr/wip-rf-qtimeout

osd/scrub: remove detection & handling of reservation timeouts from the code

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57886 from zdover23/wip-doc-2024-06-05-dev-perf-counters

doc/dev: origin of Labeled Perf Counters

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57812 from tchaikov/wip-drop-setuptools

ceph.spec.in: remove setuptools dependency

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: add options to network config ref

Add the following options to
doc/rados/configuration/network-config-ref.rst:

- public_network_interface
- cluster_network_interface

These additions were made in response to a request from Blaine Gardner.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/dev: origin of Labeled Perf Counters

Note that Labeled Perf Counters were introduced in Reef.

Fixes: https://github.com/ceph/ceph/pull/57753#discussion_r1626483732
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57868 from mcv21/ma_npage_typo

doc: correct typo (ma npage -> man page)

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge PR #57673 into main

* refs/pull/57673/head:
qa: use tell interface for command that may fail
mds: dump AsyncHandler ss to stderr if present
mds: unconditionally dump message in formatter
mds: use appropriate abbrev. for variable name
mds: dump formatter even for errors
common/admin_socket: create type for finisher callback

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

qa: use tell interface for command that may fail

The asok interface will mangle stdout if the command actually fails.

The reason `flush path` is done via the asok interface is because the tell/asok
interfaces were unified after these tests were written and `flush path` was
only available via the asok interface.

Fixes: https://tracker.ceph.com/issues/66184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump AsyncHandler ss to stderr if present

As it's being used for error output.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: unconditionally dump message in formatter

It's unpleasant to test for existence in json. Just dump an empty string if not present.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use appropriate abbrev. for variable name

Existing convention is that "css" is short for CachedStackStringStream while
"ss" is stringstream.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump formatter even for errors

The admin_socker framework only dumps the formatter by default if the command
succeeds.

Fixes: a4dc88140071f9cdfe5654402067434728fee469
Fixes: https://tracker.ceph.com/issues/66184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/admin_socket: create type for finisher callback

And change second argument to std::string_view to have more flexible
conversions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add common conf links

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: disable mon_warn_on_pool_no_app in fs suite

Fixes: https://tracker.ceph.com/issues/65976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57553 into main

* refs/pull/57553/head:
mds: try to choose a new batch head in request_clientup()
Revert "mds: find a new head for the batch ops when the head is dead"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: correct json lookup for new `lock path` output

Fixes: https://tracker.ceph.com/issues/66355
Fixes: 3552fc5a9ea17c173a18be41fa15fbbae8d77edf
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57811 from tchaikov/wip-debian-mgr-rgw

debian: package mgr/rgw in ceph-mgr-modules-core

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57867 from zdover23/wip-doc-2024-06-04-start-get-involved

doc/start: s/http/https/ in links

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57594 from connorfawcett/gcov

cmake: Add gcov linker options when coverage option is set

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc: correct typo

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

doc/start: s/http/https/ in links

Replace "http" with "https" in doc/start/get-involved.rst.

This commit is, in a way, a repeat of
https://github.com/ceph/ceph/pull/57213/
(1c5383b91bd7dbfa9670c6485fcc5ff28b79f40d), which targeted the Reef
branch instead of the main branch. When this commit has been merged and
backported, I will close https://github.com/ceph/ceph/pull/57213/.

I am listing Casey Cain here as the co-author, but he is in fact the
true author of this change.

Co-authored-by: Casey Cain <ccain@linuxfoundation.org>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57626 from AliMasarweh/wip-alimasa-bn-admin-cli

RGW\bucket notification tests: code dedup for admin commands

Merge pull request #57739 from rhcs-dashboard/fix-rgw-monitoring-doc

doc/monitoring: update rgw metrics names

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #54768 from rhcs-dashboard/exporter-unit-tests

exporter: enhance unit-tests coverage for exporter

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

RGW\bucket notification tests: code dedup for admin commands

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #57134 from rhcs-dashboard/smb-service

mgr/dashboard: add smb service management support

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56718 from ceph/shallow-submodules

do_cmake.sh: Use shallow submodule clones

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #57830 from guits/fix-ceph-volume-entrypoint

ceph-volume: fix regression

mds: avoid use of swap to initialize

These should just init with properly forwarded arguments.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57538 from ivancich/wip-shrinky-dink

rgw: add shard reduction ability to dynamic resharding

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #57839 from zdover23/wip-doc-2024-06-03-beginners-guide-edit

doc/start: Edit Beginner's Guide

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: Edit Beginner's Guide

Make some improvements to the basic text of the Beginner's Guide.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57550 from AliMasarweh/wip-alimasa-bn-end-point

RGW\bucket notification tests: end-point agnostic + refactoring

Reviewed-by: yuvalif<ylifshit@redhat.com>

ceph-volume: fix regression

This fixes a regression introduced by: 24f8e5c61b19deab7397b0237f8376c6c03a5dcb

`iter_entry_points` from `pkg_resources` takes one argument whereas
`entry_points` from `importlib.metadata` does not.

The call to `entry_points(group=group)` makes ceph-volume fail.

Fixes: https://tracker.ceph.com/issues/66328
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #57824 from zdover23/wip-doc-2024-06-03-rados-troubleshooting-mon

doc/rados: add stop monitor command

Merge pull request #57477 from vshankar/wip-66029

qa: enable debug logs for fs:cephadm:multivolume subsuite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: add stop monitor command

Add the command for stopping a monitor to the procedure that explains
how to inject a monmap into a monitor.

Zac of the future: cf. 05 Aug 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57820 from zdover23/wip-doc-2024-06-02-start-beginners-guide

doc/start: Add Beginner's Guide

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/start: Add Beginner's Guide

Add a Beginner's Guide to docs.ceph.com.

Signed-off-by: Zac Dover <zac.dover@proton.me>

ceph.spec.in: remove setuptools dependency

in 844b66de, we stopped using pkg_resources for import packaging.
and the exact reason why we introduced pkg_resources was for using
the packaging python module, see cf608920.

so, let's partially revert cf608920.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #57804 from zdover23/wip-doc-2024-05-31-cephfs-mantle-vstart-warning

doc/cephfs: edit vstart warning text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

debian: package mgr/rgw in ceph-mgr-modules-core

in 110db72e, we added the rgw mgr module to ceph-mgr-modules-core
rpm package. but we didn't add this module to the corresponding
debian package.

rgw mgr module provides a simple interface to deploy RGW multisite
setup. so it would be nice to have it in ceph's debian packages as
well.

despite that rgw is not part of the core features, since this module
is already in ceph-mgr-modules-core rpm package, and it is relatively
small and does not pulling extra dependencies, let's added to the
debian packge with the same name. we can revisit this decision and
extract it out in a following up change if it is necessary in future.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rgw: track initiator of reshard queue entries

The logic for managing the reshard queue (log) can vary depending on
whether the entry was added by an admin or by dynamic resharding. For
example, if it's a reshard reduction, dynamic resharding won't
overwrite the queue entry so as not to disrupt the reduction wait
period. On the other hand, and admin should be able to overwrite the
entry at will.

So we now track the initiator of each entry on the queue. This adds
another field to that at rest data structure, and it updates the logic
to make use of it.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: provide testing support to dynamic resharding with reduction

Adds a config option rgw_reshard_debug_interval that will allow us to
make the resharding algorithms run on a faster schedule by allowing
one day to be simulated by a set number of seconds.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add shard reduction ability to dynamic resharding

Previously, dynamic resharding could only *increase* the number of
bucket index shards for a given bucket. This adds the ability to also
*reduce* the number of shards.

So in addition the existing 100,000 entries (current default value)
per shard trigger for an increase, there's a new trigger of 10,000
entries per shard for a decrease.

However, for buckets with object-counts that go up and down regularly,
we don't want to keep resharding up and down to chase the number of
objects. So for shard reduction to take place there's also a time
delay (default 5 days). Once the entry on the reshard queue (log) is
added for reduction, processing will not result in a reshard reduction
within this delay period as the queue is processed. Only when the
reshard entry is processed after this delay can it perform the shard
reduction.

However, if at any point between the time the shard reduction entry is
added to the queue and after the delay, if the entry is processed and
there are *not* few enough entries to trigger a shard reduction, the
entry on the reshard queue entry will be discarded.

So using the defaults, this effectively means the bucket must have few
enough objects for a shard reduction for 5 consecutive days before the
reshard will take place.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/cephfs: edit vstart warning text

Improve the English in the vstart warning in doc/cephfs/mantle.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57774 from linuxbox2/wip-rgwlc-66287

rgwlc: print new filters and newer-noncurrent elements in radosgw-adm…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWDeleteMultiObj adds error messages

and consistently returns ERR_MALFORMED_XML for xml errors

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWDeleteMultiObj uses constexpr variable

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWDeleteMultiObj adds per-object DoutPrefix

add object names to the DoutPrefixProvider used by
handle_individual_object() to make it easier to trace through concurrent
calls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: avoid gotos in RGWDeleteMultiObj::execute

use the send_response() hook to send error responses after execute()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWDeleteMultiObj uses spawn_throttle for concurrency

adapt RGWDeleteMultiObj to use ceph::async::spawn_throttle for the
handle_individual_object() coroutines it spawns

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57377 from dang/wip-dang-obj-state

RGW: Remove RGWObjState and get_obj_state()/put_obj_state() from the SAL API

Merge pull request #57697 from petrutlucian94/sv_wait

rbd-wnbd: wait for the disk cleanup to complete

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rbd-wnbd: wait for the disk cleanup to complete

The WNBD disk removal workflow is asynchronous, which is why we'll
need to wait for the cleanup to complete when stopping the service.

The "disconnect_all_mappings" function is moved to
RbdMappingDispatcher::stop, allowing us to access the mapping list
more easily and reject new mappings after a stop has been requested.

While at it, we'll log service stop requests.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #57650 from tchaikov/wip-use-importlib

ceph-volume: use importlib from stdlib on Python 3.8 and up

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57785 from rhcs-dashboard/ssl-file-fix

mgr/dashboard: read cert from in-memory file

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

mgr/dashboard: read cert from in-memory file

Fixes: https://tracker.ceph.com/issues/66303
Signed-off-by: Nizamudeen A <nia@redhat.com>

cls/rgw: adding an entry to reshard queue has O_CREAT option

Adds the ability to prevent overwriting a reshard queue (log) entry
for a given bucket with a newer entry. This adds a flag to the op, so
it will either CREATE or make no changes. If an entry already exists
when this flag is set, -EEXIST will be returned.

This is a preparatory step to adding shard reduction to dynamic
resharding.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

RGW\bucket notification tests: end-point agnostic + refactoring

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #57790 from zdover23/wip-doc-2024-05-30-cephfs-mantle-front-matter

doc/cephfs: edit front matter in mantle.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit front matter in mantle.rst

Improve the structure and grammar of the front matter in the
doc/cephfs/mantle.rst file.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57711 from cyx1231st/wip-seastore-fix-incorrent-reclaimable-segments

crimson/os/seastore/async_cleaner: fix incorrect get_num_rolls()

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57612 from Svelar/asan_rgw_kms

rgw_kms: fix Realloc and free cct

rgwlc: print new filters and newer-noncurrent elements in radosgw-admin lc get

Fixes: https://tracker.ceph.com/issues/66287
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cmake: Add gcov --coverage flags when coverage option is set

Signed-off-by: Connor Fawcett <connorfa@uk.ibm.com>
Fixes: https://tracker.ceph.com/issues/66158