test/ librbd: clean up unused 'TEST_COOKIE'

Clean them up because
```
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/librbd/exclusive_lock/test_mock_PreAcquireRequest.cc:47:26: error: unused variable 'TEST_COOKIE' [-Werror,-Wunused-const-variable]
static const std::string TEST_COOKIE("auto 123");
                         ^
1 error generated.
```

Ref: https://jenkins.ceph.com/job/ceph-pull-requests-arm64/59242/consoleFull#-5120798632a811ea2-3e7b-466b-84b4-d13df7e35809

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #57957 from zdover23/wip-doc-2024-06-11-rados-troubleshooting-mon

doc/rados: improve leader/peon monitor explanation

doc/rados: improve leader/peon monitor explanation

Add an explanation of leader-peon conditions that obtain when the
cluster is in the "HEALTH_OK" state. Previously, the text discussed
these two monitor states only in the context of a health detail entry.

This improvement to the documentation was suggested on the [ceph-users]
email list by Joel Davidow. This email, an absolute model of user
engagement with an upstream project, can be reviewed here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KF67F5TXFSSTPXV7EKL6JKLA5KZQDLDQ/

I will list Joel Davidow here as the co-author for the sake of more
expediently getting this change into the documentation, but though he is
listed as the co-author, he is the true author.

Co-authored-by: Joel Davidow <jdavidow@nso.edu>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56744 from benhanokh/ncb_async_discard_fix

os/BlueStore: NCB fix for leaked space when bdev_async_discard is ena…

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Joshua Baergen <jbaergen@digitalocean.com>

Merge pull request #53477 from jianwei1216/fix_msgr_worker_cpu_high

src/msg: fix high CPU consumption of msgr worker thread

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #51171 from amathuria/wip-amat-scrub-cost-related-changes

osd/scrub: Change scrub cost to average object size

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #57593 from Matan-B/wip-matanb-crimson-default-cpu-cores

Crimson: Support basic deployments

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57313 from xxhdx1985126/wip-65832

crimson/osd/ops_executor: calculation of clone_overlap shouldn't consider snap contexts

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57279 from xxhdx1985126/wip-65806

crimson/osd/pg: trigger wait_for_active_blocker on replica osds when the activate event is committed

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57947 from zdover23/wip-doc-2024-06-10-start-rm-intro

doc/start: remove "intro.rst"

<Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/start: remove "intro.rst"

Remove "start/intro.rst", which has been renamed "start/index.rst" in
order to follow the conventions followed elsewhere in the documentation.

Follows https://github.com/ceph/ceph/pull/57900.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57908 from xxhdx1985126/wip-66374

crimson/osd/osd_operations: correct connection pipelines for osd operations

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57692 from xxhdx1985126/wip-crimson-unnecessary-move

crimson/osd/pg_shard_manager: remove the unnecessary "std::move" call

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57561 from guojidan/rollback

crimson/osd/ops_executer: LIST_SNAPS only on CEPH_SNAPDIR

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57416 from xxhdx1985126/wip-65972

crimson/osd/pg: hold PGs' references to the last minute of snap trim events executions

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57455 from xxhdx1985126/wip-66002

crimson/osd/replicated_backend: no need to set_rollback_to for repops

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57386 from xxhdx1985126/wip-65894

crimson/osd/recovery_backend: change recovery waiters' promises into optional ones

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57276 from xxhdx1985126/wip-65804

crimson/osd/pg_backend: DONOT modify OSDOp::indata when handling CEPH_OSD_OP_CHECKSUM

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57251 from xxhdx1985126/wip-65773

crimson/osd/pg_backend: only create new onode when it doesn't exist

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57204 from rzarzynski/wip-crimson-osd-read-clamp

crimson/osd: clamp reads to object size and bring full read trimming

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57069 from xxhdx1985126/wip-65632

crimson/common/operation: detach blockers from blocking events when they are destroyed

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57788 from cyx1231st/wip-seastore-report-disk

crimson/os/seastore: implement disk and writer level stats reporting

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57939 from zdover23/wip-doc-2024-06-09-glossary-swift

doc/glossary.rst: add "OpenStack Swift" and "Swift"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/glossary.rst: add "OpenStack Swift" and "Swift"

Add "OpenStack Swift" and "Swift" entries to the glossary.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57940 from zdover23/wip-doc-2024-06-090-start-documenting-ceph-squash

doc/start: format squash procedure

doc/start: format squash procedure

Add letters to substeps in a procedure so that readers won't get too
confused by having numbered steps nested inside numbered steps.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #48130 into main

* refs/pull/48130/head:
qa: add killpoint testing for dirfrags
qa: stringify arguments to setfattr
qa: move some configs to cluster-conf
qa: restore default for config to split exports
qa/tasks/ceph_test_case: rollback configs using `config reset`
qa/cephfs: set confs using cluster-conf
qa/tasks/ceph: provide configuration for setting configs via mon
mds: optimize MDBalancer code path config access
mds: add killpoints for directory fragmentation

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57912 from ceph/wip-lusov-qdb-exclude-or-cancel

mds: QuiesceDbRequest: update the internal encoding of ops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57923 from zdover23/wip-doc-2024-06-07-global-yaml-in-inline-literals

doc/src: add inline literals (``) to variables

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57755 from rhcs-dashboard/fix-66267-main

mgr/dashboard: provide option to enable pool based mirroring mode while creating a pool

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #57882 into main

* refs/pull/57882/head:
mds: add debug message when conf changes are processed

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #57919 into main

* refs/pull/57919/head:
script/ceph-backport: lookup "Release" custom_field by id

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57894 from AliMasarweh/wip-alimasa-bn-http-teuthology

RGW|BN: make http tests run on a seperate task

Reviewed-by: yuvalif<ylifshit@redhat.com>

Merge pull request #57746 from ceph/nodejs-lts-update

mgr/dashboard: update node js version

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: provide option to enable pool based mirroring mode while
creating a pool

Fixes: https://tracker.ceph.com/issues/66267
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

doc/src: add inline literals (``) to variables

Add inline-literal RST markup (``) (double backticks) around variable
names in src/common/options/global.yaml.in.

This follows https://github.com/ceph/ceph/pull/57887.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #56874 from smanjara/wip-shilpa-multisite-tests

qa/multisite: stabilize multisite testing

multisite/qa: resolve rebase conflicts

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

multisite/qa: disable sync error injection

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: disable two-zonegroup and three-zone configs.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/rgw/multisite: enable notification_v2 feature

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1e7fc66683e4dea7294f0ec95f47b22f48b33b7e)

qa/multisite: disable sync policy, endpoint and data_sync_init tests until fixed.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/rgw/multisite: add two-zone configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 14204442ac2eaaaa3f40a76403da49f0f4ed92c1)

qa/rgw/multisite: double cluster nodes and osd count

> WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand.

give each cluster 2 nodes instead of 1, and 6 osds instead of 3

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9a80cf4bf473c95c8d0b18fcdce88be20f74f973)

qa/rgw/multisite: reduce polling intervals from 20s to 5s

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit bc084c0ac64477abdbf54d96bc4981edf41e1343)

rgw/multisite: reconfigure checkpoint delay to 90s

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #57725 from Matan-B/wip-matanb-crimson-load-exist

crimson/osd/object_context_loader: Fix obc cache existence usage

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

script/ceph-backport: lookup "Release" custom_field by id

Right now the script assumes the Release custom_field will be ordered first in
the array from the REST API.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add killpoint testing for dirfrags

Fixes: https://tracker.ceph.com/issues/7320
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

qa: stringify arguments to setfattr

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: move some configs to cluster-conf

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: restore default for config to split exports

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/ceph_test_case: rollback configs using `config reset`

This is faster and catches config changes made outside this interface.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: set confs using cluster-conf

So that these configs can be easily overriden by tests.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/ceph: provide configuration for setting configs via mon

These configs may be set using:

ceph:
  cluster-config:
    entity:
      foo: bar

same as the current:

ceph:
  config:
    entity:
      foo: bar

The configs will be set in parallel using the `ceph config set` command.

The main benefit here is to avoid using the ceph.conf to set configs which
cannot be overriden using subsequent `ceph config` command. The only way to
override is to change the ceph.conf in the test (yuck) or the admin socket
(which gets reset when the daemon restarts).

Finally, we can now exploit the `ceph config reset` command will let us
trivially rollback config changes after a test completes. That is exposed
as the `ctx.config_epoch` variable.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: optimize MDBalancer code path config access

This change is necessary because the new way of setting config is to use the
ceph config command or the asok interface rather than the old way which
involved editing the ceph.conf and restarting the daemons to reflect the
changes. Have updated the code to support runtime config changes.

Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add killpoints for directory fragmentation

Defined a set of killpoints for testing directory fragmentation during
cluster recovery after failure.

Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57879 into main

* refs/pull/57879/head:
qa: add common conf links
qa: disable mon_warn_on_pool_no_app in fs suite

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57877 into main

* refs/pull/57877/head:
qa: correct json lookup for new `lock path` output

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57813 into main

* refs/pull/57813/head:
mds: avoid use of swap to initialize

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57887 from zdover23/wip-doc-2024-06-05-rados-config-network-config-ref

doc/rados: add options to network config ref

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #57641 from cbodley/wip-rgw-delete-objects-throttle

rgw: RGWDeleteMultiObj uses spawn_throttle for concurrency

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57702 from Suyashd999/fix-uam2

rados/cls_fifo_legacy: 'oid' used after it was moved

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

crimson/osd/object_context_loader: get_or_load to support atomicity

make use of try_lock in order to support atomicity when called in
ObjectContext::_with_lock()

Co-authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: make lock() atomic if doesn't need wait

Otherwise, promotion cannot be atomic with the 1st locker.

Identified by: Matan Breizman <mbreizma@redhat.com>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/tri_mutex: minor cleanup to be consistent

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: add waiter_t::waiter_name

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/tri_mutex: add debug logs

to be used only for testing

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: with_head_obc to log `existed`

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_context_loader: cleanup `loaded`

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #57696 from kotreshhr/fix_headcheck_first_damage

tools/first-damage: Make CEPH_NOSNAP int64

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: QuiesceDbRequest: update the internal encoding of ops

Excluding the last root from a set will automatically mark it as QS_CANCELED.
Hence, it makes more sense if `exclude` and `cancel` share the same op code,
rather than `exclude` and `release`.

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/66383

RGW|BN: make http tests run on a seperate task

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

tools/first-damage: Make CEPH_NOSNAP int64

Make CEPH_NOSNAP int64 and pack 8 bytes when updating the omap
value.

Signed-off-by: Kotresh HR <khiremat@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

crimson/osd/osd_operations: correct connection pipelines for osd
operations

Fixes: https://tracker.ceph.com/issues/66374
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #56275 from smanjara/wip-set-attr-mtime

rgw/multisite: allow PutACL replication.

Merge pull request #57900 from zdover23/wip-doc-2024-06-06-start-intro-to-index

doc/start: s/intro.rst/index.rst/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/multisite: add object set acl test

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

doc/start: s/intro.rst/index.rst/

Change the filename "doc/start/intro.rst" to "doc/start/index.rst" so
that Sphinx finds the root filename for the "/start" directory in the
default location.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/multisite: allow modified object acls to be replicated

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #57015 from liangmingyuanneo/wip-bluefs-max-alloc-size

bluefs: bluefs alloc unit should only be shrink

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #56428 from tchaikov/wip-crush-no-vla

crush: use std::vector instead of variable length arrays

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56389 from ronen-fr/wip-rf-qtimeout

osd/scrub: remove detection & handling of reservation timeouts from the code

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57886 from zdover23/wip-doc-2024-06-05-dev-perf-counters

doc/dev: origin of Labeled Perf Counters

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57812 from tchaikov/wip-drop-setuptools

ceph.spec.in: remove setuptools dependency

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: add options to network config ref

Add the following options to
doc/rados/configuration/network-config-ref.rst:

- public_network_interface
- cluster_network_interface

These additions were made in response to a request from Blaine Gardner.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/dev: origin of Labeled Perf Counters

Note that Labeled Perf Counters were introduced in Reef.

Fixes: https://github.com/ceph/ceph/pull/57753#discussion_r1626483732
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57868 from mcv21/ma_npage_typo

doc: correct typo (ma npage -> man page)

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge PR #57673 into main

* refs/pull/57673/head:
qa: use tell interface for command that may fail
mds: dump AsyncHandler ss to stderr if present
mds: unconditionally dump message in formatter
mds: use appropriate abbrev. for variable name
mds: dump formatter even for errors
common/admin_socket: create type for finisher callback

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

mds: add debug message when conf changes are processed

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use tell interface for command that may fail

The asok interface will mangle stdout if the command actually fails.

The reason `flush path` is done via the asok interface is because the tell/asok
interfaces were unified after these tests were written and `flush path` was
only available via the asok interface.

Fixes: https://tracker.ceph.com/issues/66184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump AsyncHandler ss to stderr if present

As it's being used for error output.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: unconditionally dump message in formatter

It's unpleasant to test for existence in json. Just dump an empty string if not present.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use appropriate abbrev. for variable name

Existing convention is that "css" is short for CachedStackStringStream while
"ss" is stringstream.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: dump formatter even for errors

The admin_socker framework only dumps the formatter by default if the command
succeeds.

Fixes: a4dc88140071f9cdfe5654402067434728fee469
Fixes: https://tracker.ceph.com/issues/66184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/admin_socket: create type for finisher callback

And change second argument to std::string_view to have more flexible
conversions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add common conf links

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: disable mon_warn_on_pool_no_app in fs suite

Fixes: https://tracker.ceph.com/issues/65976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57553 into main

* refs/pull/57553/head:
mds: try to choose a new batch head in request_clientup()
Revert "mds: find a new head for the batch ops when the head is dead"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: correct json lookup for new `lock path` output

Fixes: https://tracker.ceph.com/issues/66355
Fixes: 3552fc5a9ea17c173a18be41fa15fbbae8d77edf
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>