doc/start: format squash procedure

Add letters to substeps in a procedure so that readers won't get too
confused by having numbered steps nested inside numbered steps.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #48130 into main

* refs/pull/48130/head:
qa: add killpoint testing for dirfrags
qa: stringify arguments to setfattr
qa: move some configs to cluster-conf
qa: restore default for config to split exports
qa/tasks/ceph_test_case: rollback configs using `config reset`
qa/cephfs: set confs using cluster-conf
qa/tasks/ceph: provide configuration for setting configs via mon
mds: optimize MDBalancer code path config access
mds: add killpoints for directory fragmentation

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57912 from ceph/wip-lusov-qdb-exclude-or-cancel

mds: QuiesceDbRequest: update the internal encoding of ops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57923 from zdover23/wip-doc-2024-06-07-global-yaml-in-inline-literals

doc/src: add inline literals (``) to variables

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57755 from rhcs-dashboard/fix-66267-main

mgr/dashboard: provide option to enable pool based mirroring mode while creating a pool

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #57882 into main

* refs/pull/57882/head:
mds: add debug message when conf changes are processed

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #57919 into main

* refs/pull/57919/head:
script/ceph-backport: lookup "Release" custom_field by id

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57894 from AliMasarweh/wip-alimasa-bn-http-teuthology

RGW|BN: make http tests run on a seperate task

Reviewed-by: yuvalif<ylifshit@redhat.com>

Merge pull request #57746 from ceph/nodejs-lts-update

mgr/dashboard: update node js version

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: provide option to enable pool based mirroring mode while
creating a pool

Fixes: https://tracker.ceph.com/issues/66267
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

doc/src: add inline literals (``) to variables

Add inline-literal RST markup (``) (double backticks) around variable
names in src/common/options/global.yaml.in.

This follows https://github.com/ceph/ceph/pull/57887.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56874 from smanjara/wip-shilpa-multisite-tests

qa/multisite: stabilize multisite testing

multisite/qa: resolve rebase conflicts

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

multisite/qa: disable sync error injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: disable two-zonegroup and three-zone configs.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/rgw/multisite: enable notification_v2 feature

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1e7fc66683e4dea7294f0ec95f47b22f48b33b7e)

qa/multisite: disable sync policy, endpoint and data_sync_init tests until fixed.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/rgw/multisite: add two-zone configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 14204442ac2eaaaa3f40a76403da49f0f4ed92c1)

qa/rgw/multisite: double cluster nodes and osd count

> WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand.

give each cluster 2 nodes instead of 1, and 6 osds instead of 3

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9a80cf4bf473c95c8d0b18fcdce88be20f74f973)

qa/rgw/multisite: reduce polling intervals from 20s to 5s

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit bc084c0ac64477abdbf54d96bc4981edf41e1343)

rgw/multisite: reconfigure checkpoint delay to 90s

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #57725 from Matan-B/wip-matanb-crimson-load-exist

crimson/osd/object_context_loader: Fix obc cache existence usage

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

script/ceph-backport: lookup "Release" custom_field by id

Right now the script assumes the Release custom_field will be ordered first in
the array from the REST API.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add killpoint testing for dirfrags

Fixes: https://tracker.ceph.com/issues/7320
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

qa: stringify arguments to setfattr

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: move some configs to cluster-conf

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: restore default for config to split exports

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/ceph_test_case: rollback configs using `config reset`

This is faster and catches config changes made outside this interface.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: set confs using cluster-conf

So that these configs can be easily overriden by tests.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/ceph: provide configuration for setting configs via mon

These configs may be set using:

ceph:
  cluster-config:
    entity:
      foo: bar

same as the current:

ceph:
  config:
    entity:
      foo: bar

The configs will be set in parallel using the `ceph config set` command.

The main benefit here is to avoid using the ceph.conf to set configs which
cannot be overriden using subsequent `ceph config` command. The only way to
override is to change the ceph.conf in the test (yuck) or the admin socket
(which gets reset when the daemon restarts).

Finally, we can now exploit the `ceph config reset` command will let us
trivially rollback config changes after a test completes. That is exposed
as the `ctx.config_epoch` variable.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: optimize MDBalancer code path config access

This change is necessary because the new way of setting config is to use the
ceph config command or the asok interface rather than the old way which
involved editing the ceph.conf and restarting the daemons to reflect the
changes. Have updated the code to support runtime config changes.

Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add killpoints for directory fragmentation

Defined a set of killpoints for testing directory fragmentation during
cluster recovery after failure.

Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57879 into main

* refs/pull/57879/head:
qa: add common conf links
qa: disable mon_warn_on_pool_no_app in fs suite

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57877 into main

* refs/pull/57877/head:
qa: correct json lookup for new `lock path` output

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57813 into main

* refs/pull/57813/head:
mds: avoid use of swap to initialize

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57887 from zdover23/wip-doc-2024-06-05-rados-config-network-config-ref

doc/rados: add options to network config ref

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #57641 from cbodley/wip-rgw-delete-objects-throttle

rgw: RGWDeleteMultiObj uses spawn_throttle for concurrency

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57702 from Suyashd999/fix-uam2

rados/cls_fifo_legacy: 'oid' used after it was moved

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

crimson/osd/object_context_loader: get_or_load to support atomicity

make use of try_lock in order to support atomicity when called in
ObjectContext::_with_lock()

Co-authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: make lock() atomic if doesn't need wait

Otherwise, promotion cannot be atomic with the 1st locker.

Identified by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/tri_mutex: minor cleanup to be consistent

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: add waiter_t::waiter_name

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: add debug logs

to be used only for testing

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: with_head_obc to log `existed`

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: cleanup `loaded`

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57696 from kotreshhr/fix_headcheck_first_damage

tools/first-damage: Make CEPH_NOSNAP int64

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: QuiesceDbRequest: update the internal encoding of ops

Excluding the last root from a set will automatically mark it as QS_CANCELED.
Hence, it makes more sense if `exclude` and `cancel` share the same op code,
rather than `exclude` and `release`.

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/66383

RGW|BN: make http tests run on a seperate task

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

tools/first-damage: Make CEPH_NOSNAP int64

Make CEPH_NOSNAP int64 and pack 8 bytes when updating the omap
value.

Signed-off-by: Kotresh HR <khiremat@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56275 from smanjara/wip-set-attr-mtime

rgw/multisite: allow PutACL replication.

Merge pull request #57900 from zdover23/wip-doc-2024-06-06-start-intro-to-index

doc/start: s/intro.rst/index.rst/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/multisite: add object set acl test

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

doc/start: s/intro.rst/index.rst/

Change the filename "doc/start/intro.rst" to "doc/start/index.rst" so
that Sphinx finds the root filename for the "/start" directory in the
default location.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/multisite: allow modified object acls to be replicated

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #57015 from liangmingyuanneo/wip-bluefs-max-alloc-size

bluefs: bluefs alloc unit should only be shrink

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56428 from tchaikov/wip-crush-no-vla

crush: use std::vector instead of variable length arrays

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56389 from ronen-fr/wip-rf-qtimeout

osd/scrub: remove detection & handling of reservation timeouts from the code

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57886 from zdover23/wip-doc-2024-06-05-dev-perf-counters

doc/dev: origin of Labeled Perf Counters

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57812 from tchaikov/wip-drop-setuptools

ceph.spec.in: remove setuptools dependency

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: add options to network config ref

Add the following options to
doc/rados/configuration/network-config-ref.rst:

- public_network_interface
- cluster_network_interface

These additions were made in response to a request from Blaine Gardner.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/dev: origin of Labeled Perf Counters

Note that Labeled Perf Counters were introduced in Reef.

Fixes: https://github.com/ceph/ceph/pull/57753#discussion_r1626483732
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57868 from mcv21/ma_npage_typo

doc: correct typo (ma npage -> man page)

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge PR #57673 into main

* refs/pull/57673/head:
qa: use tell interface for command that may fail
mds: dump AsyncHandler ss to stderr if present
mds: unconditionally dump message in formatter
mds: use appropriate abbrev. for variable name
mds: dump formatter even for errors
common/admin_socket: create type for finisher callback

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

mds: add debug message when conf changes are processed

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use tell interface for command that may fail

The asok interface will mangle stdout if the command actually fails.

The reason `flush path` is done via the asok interface is because the tell/asok
interfaces were unified after these tests were written and `flush path` was
only available via the asok interface.

Fixes: https://tracker.ceph.com/issues/66184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump AsyncHandler ss to stderr if present

As it's being used for error output.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: unconditionally dump message in formatter

It's unpleasant to test for existence in json. Just dump an empty string if not present.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use appropriate abbrev. for variable name

Existing convention is that "css" is short for CachedStackStringStream while
"ss" is stringstream.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump formatter even for errors

The admin_socker framework only dumps the formatter by default if the command
succeeds.

Fixes: a4dc88140071f9cdfe5654402067434728fee469
Fixes: https://tracker.ceph.com/issues/66184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/admin_socket: create type for finisher callback

And change second argument to std::string_view to have more flexible
conversions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add common conf links

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: disable mon_warn_on_pool_no_app in fs suite

Fixes: https://tracker.ceph.com/issues/65976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57553 into main

* refs/pull/57553/head:
mds: try to choose a new batch head in request_clientup()
Revert "mds: find a new head for the batch ops when the head is dead"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: correct json lookup for new `lock path` output

Fixes: https://tracker.ceph.com/issues/66355
Fixes: 3552fc5a9ea17c173a18be41fa15fbbae8d77edf
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57811 from tchaikov/wip-debian-mgr-rgw

debian: package mgr/rgw in ceph-mgr-modules-core

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57867 from zdover23/wip-doc-2024-06-04-start-get-involved

doc/start: s/http/https/ in links

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57594 from connorfawcett/gcov

cmake: Add gcov linker options when coverage option is set

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc: correct typo

Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>

doc/start: s/http/https/ in links

Replace "http" with "https" in doc/start/get-involved.rst.

This commit is, in a way, a repeat of
https://github.com/ceph/ceph/pull/57213/
(1c5383b91bd7dbfa9670c6485fcc5ff28b79f40d), which targeted the Reef
branch instead of the main branch. When this commit has been merged and
backported, I will close https://github.com/ceph/ceph/pull/57213/.

I am listing Casey Cain here as the co-author, but he is in fact the
true author of this change.

Co-authored-by: Casey Cain <ccain@linuxfoundation.org>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57626 from AliMasarweh/wip-alimasa-bn-admin-cli

RGW\bucket notification tests: code dedup for admin commands

Merge pull request #57739 from rhcs-dashboard/fix-rgw-monitoring-doc

doc/monitoring: update rgw metrics names

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #54768 from rhcs-dashboard/exporter-unit-tests

exporter: enhance unit-tests coverage for exporter

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

RGW\bucket notification tests: code dedup for admin commands

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #57134 from rhcs-dashboard/smb-service

mgr/dashboard: add smb service management support

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56718 from ceph/shallow-submodules

do_cmake.sh: Use shallow submodule clones

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #57830 from guits/fix-ceph-volume-entrypoint

ceph-volume: fix regression

mds: avoid use of swap to initialize

These should just init with properly forwarded arguments.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57538 from ivancich/wip-shrinky-dink

rgw: add shard reduction ability to dynamic resharding

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #57839 from zdover23/wip-doc-2024-06-03-beginners-guide-edit

doc/start: Edit Beginner's Guide

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/start: Edit Beginner's Guide

Make some improvements to the basic text of the Beginner's Guide.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57550 from AliMasarweh/wip-alimasa-bn-end-point

RGW\bucket notification tests: end-point agnostic + refactoring

Reviewed-by: yuvalif<ylifshit@redhat.com>

ceph-volume: fix regression

This fixes a regression introduced by: 24f8e5c61b19deab7397b0237f8376c6c03a5dcb

`iter_entry_points` from `pkg_resources` takes one argument whereas
`entry_points` from `importlib.metadata` does not.

The call to `entry_points(group=group)` makes ceph-volume fail.

Fixes: https://tracker.ceph.com/issues/66328
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #57824 from zdover23/wip-doc-2024-06-03-rados-troubleshooting-mon

doc/rados: add stop monitor command

Merge pull request #57477 from vshankar/wip-66029

qa: enable debug logs for fs:cephadm:multivolume subsuite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: add stop monitor command

Add the command for stopping a monitor to the procedure that explains
how to inject a monmap into a monitor.

Zac of the future: cf. 05 Aug 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57820 from zdover23/wip-doc-2024-06-02-start-beginners-guide

doc/start: Add Beginner's Guide

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/start: Add Beginner's Guide

Add a Beginner's Guide to docs.ceph.com.

Signed-off-by: Zac Dover <zac.dover@proton.me>

ceph.spec.in: remove setuptools dependency

in 844b66de, we stopped using pkg_resources for import packaging.
and the exact reason why we introduced pkg_resources was for using
the packaging python module, see cf608920.

so, let's partially revert cf608920.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #57804 from zdover23/wip-doc-2024-05-31-cephfs-mantle-vstart-warning

doc/cephfs: edit vstart warning text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

debian: package mgr/rgw in ceph-mgr-modules-core

in 110db72e, we added the rgw mgr module to ceph-mgr-modules-core
rpm package. but we didn't add this module to the corresponding
debian package.

rgw mgr module provides a simple interface to deploy RGW multisite
setup. so it would be nice to have it in ceph's debian packages as
well.

despite that rgw is not part of the core features, since this module
is already in ceph-mgr-modules-core rpm package, and it is relatively
small and does not pulling extra dependencies, let's added to the
debian packge with the same name. we can revisit this decision and
extract it out in a following up change if it is necessary in future.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>