osd: improve debugs on EC paths that turned problematic in testing

We want to have better coverage espcially in the matter of:
  * subchunking,
  * want_to_read handling,
  * result bl trimming.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

erasure-code: hack the subchunk handling to honor legacy

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: decode_concat() always honors want_to_read

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa: don't pass --max_attr_len to ceph_test_rados unconditionally

Fixes: https://tracker.ceph.com/issues/66321
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 5c821233d097eb6eb4287bbd1d0b6d01638e5f90)

qa: test-erasure-eio.sh honors the EC partial read support

This is supposed to fix:

```
2024-05-15T01:19:55.945 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size:  rados_get td/test-erasure-
eio pool-jerasure obj-size-81362-1-10 fail
2024-05-15T01:19:55.946 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get:  local dir=td/test-erasure-eio
2024-05-15T01:19:55.946 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get:  local poolname=pool-jerasure
2024-05-15T01:19:55.946 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get:  local objname=obj-size-81362-1-10
2024-05-15T01:19:55.946 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get:  local expect=fail
2024-05-15T01:19:55.946 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get:  '[' fail = fail ']'
2024-05-15T01:19:55.946 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get:  rados --pool pool-jerasure get obj-size-
81362-1-10 td/test-erasure-eio/COPY
2024-05-15T01:19:56.175 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get:  return
2024-05-15T01:19:56.175 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size:  return 1
2024-05-15T01:19:56.175 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1:  return 1
2024-05-15T01:19:56.175 INFO:tasks.workunit.client.0.smithi190.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run:  return 1
```

(https://pulpito.ceph.com/rzarzynski-2024-05-14_22:09:16-rados-wip-osd-ec-partial-reads-distro-default-smithi/7706517/)

The failed scenario was exercising a behavior that got truly
changed by introduction of partial reads. Before, regardless
of read size, OSD was always reading and checking for errors
entire stripe.

In this test first 4 KB has been read from an EC pool with
m=2 k=1 while errors had been injected to shards 1 and 2.
Handling the first 4 KB doesn't really require the damaged
shards but, because of the full-stripe alignment, EIO was
returned. This is not anymore.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>

osd: fix trimming of multistripe partial reads

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/ceph_test_rados: fix seeking over virtual, in-memory objects

`ceph_test_rados` maintains a database of in-memory pseudo-objects
that are used to dynamically 1) generate content for writing into
RADOS objects and 2) regenerate it later when comparing buffers of
completed RADOS reads.

Content of these objects is ephemeral; to preserve space we store
only starting parameters for a pseudo-random number generator.

However, this memory optimization requires strict management of
calls to the PRGN. Unfortunately, adding partial reads support has
unveiled a bug in this aspect: `ObjectDesc::iterator::seek()` was
missing a call to underlying `RandGenerator::iterator_impl::seek()`,
resulting in mismatches on read buffer validation.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/ceph_test_rados: bring non-0 offset reads support.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd, tests: add unittesting for ReadPipeline::get_min_want_to_read_shards()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: don't realign EC reads to stripe in ReadPipeline::send_all_remaining_reads()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: minimize alignment requirements of ReadPipeline

Before it was operating on stripe-aligned requests, now chunk-
alignment is enough.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: drop the is_systematic from ErasureCodeInterface

This method has introduced by the intial EC partial read
effort. However, at the moment there is no implementation
of the `ErasureCodeInterface` that returns `false`.
In other words: all out EC algorithms are systematic
(for explanation of this term please refer to Wikipedia:
https://en.wikipedia.org/wiki/Systematic_code).

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: glue read scheduling with decoding in terms of want_to_read

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: EC's ReadCompleters align to chunk size

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: generalize the EC partial read optimization

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: refactor get_min_want_to_read_shards() of EC support

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch ec_extents_t from pair to named struct

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: switch ec_align_t from tuple to named struct

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: introduce osd_ec_partial_reads configurable

The rationale is to have a valve allowing to bypass
the optimization at run-time.
After staging period it will be fine to drop it.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: shuffle ECUtil::stripe_info_t::aligned_offset_len_to_chunk to .cc

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: avoid visual confusion of var names in EC ReadPipeline

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: EC Partial Stripe Reads (Retry of #23138 and #52746)

This commit is a further ressurection of the EC partial reads
concept; this time of the Mark Nelson's work sent as PR #52746.
The modifications in this commit are mostly about settling
Mark's work on top of the recent rework of `ECBackend` which
had shared the EC codebase with the crimson-osd.

At the original description says, Mark's work is based on earlier
attempt from Xiaofei Cui.

Therefore credits go to:
  * Mark Nelson (Clyso),
  * Xiaofei Cui (cuixiaofei@sangfor.com.cn).

The original commit description is preserved below:

> This is a re-implementation of PR #23138 rebased on main with a couple of nitpicky changes to make the code a little more clear (to me at least).  Credit goes to Xiaofei Cui [cuixiaofei@sangfor.com.cn](mailto:cuixiaofei@sangfor.com.cn) for the original implementation.
>
> Looking at the original PR's review, it does not appear that we can use the same technique as in 468ad4b.  We don't have the ReadOp yet.  I'm not sure if @gregsforytwo's idea to query the plugin works, but it's clear we are not doing the efficient thing from the get-go here.
>
> The performance and efficiency benefits for small random reads appears to be quite substantial, especially for large stripe widths.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: introduce aliases around EC reads

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57824 from zdover23/wip-doc-2024-06-03-rados-troubleshooting-mon

doc/rados: add stop monitor command

Merge pull request #57477 from vshankar/wip-66029

qa: enable debug logs for fs:cephadm:multivolume subsuite

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: add stop monitor command

Add the command for stopping a monitor to the procedure that explains
how to inject a monmap into a monitor.

Zac of the future: cf. 05 Aug 2023.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57820 from zdover23/wip-doc-2024-06-02-start-beginners-guide

doc/start: Add Beginner's Guide

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/start: Add Beginner's Guide

Add a Beginner's Guide to docs.ceph.com.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57804 from zdover23/wip-doc-2024-05-31-cephfs-mantle-vstart-warning

doc/cephfs: edit vstart warning text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit vstart warning text

Improve the English in the vstart warning in doc/cephfs/mantle.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57774 from linuxbox2/wip-rgwlc-66287

rgwlc: print new filters and newer-noncurrent elements in radosgw-adm…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57377 from dang/wip-dang-obj-state

RGW: Remove RGWObjState and get_obj_state()/put_obj_state() from the SAL API

Merge pull request #57697 from petrutlucian94/sv_wait

rbd-wnbd: wait for the disk cleanup to complete

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rbd-wnbd: wait for the disk cleanup to complete

The WNBD disk removal workflow is asynchronous, which is why we'll
need to wait for the cleanup to complete when stopping the service.

The "disconnect_all_mappings" function is moved to
RbdMappingDispatcher::stop, allowing us to access the mapping list
more easily and reject new mappings after a stop has been requested.

While at it, we'll log service stop requests.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #57650 from tchaikov/wip-use-importlib

ceph-volume: use importlib from stdlib on Python 3.8 and up

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57785 from rhcs-dashboard/ssl-file-fix

mgr/dashboard: read cert from in-memory file

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

mgr/dashboard: read cert from in-memory file

Fixes: https://tracker.ceph.com/issues/66303
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57790 from zdover23/wip-doc-2024-05-30-cephfs-mantle-front-matter

doc/cephfs: edit front matter in mantle.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit front matter in mantle.rst

Improve the structure and grammar of the front matter in the
doc/cephfs/mantle.rst file.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57711 from cyx1231st/wip-seastore-fix-incorrent-reclaimable-segments

crimson/os/seastore/async_cleaner: fix incorrect get_num_rolls()

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57612 from Svelar/asan_rgw_kms

rgw_kms: fix Realloc and free cct

rgwlc: print new filters and newer-noncurrent elements in radosgw-admin lc get

Fixes: https://tracker.ceph.com/issues/66287
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #51591 from rhcs-dashboard/add-helpers-pool-compression

mgr/dashboard: add helpers for compression in pool form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #57748 from yuvalif/wip-yuval-test-http

test/rgw/notification: verify network access to http server

Reviewed-By: Ali Masarwe <ali.masarwa@ibm.com>

Merge PR #57354 into main

* refs/pull/57354/head:
mds: relax divergent backtrace scrub failures for replicated ancestor inodes

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #55758 into main

* refs/pull/55758/head:
doc: update 'journal reset' command with --yes-i-really-really-mean-it
qa: fix cephfs-journal-tool command options and make fs inactive
cephfs-journal-tool: Add warning messages during 'journal reset' and prevent execution on active fs

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57291 into main

* refs/pull/57291/head:
pybind/mgr/mirroring: Fix KeyError: 'directory_count' in daemon status

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57694 from zhscn/fix-avl-alloc

crimson/os/seastore/rbm/avlallocator: return enough regions when request size is greater than max_alloc_size

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57753 from zdover23/wip-doc-2024-05-29-doc-dev-perf-counters

doc/dev: add note about intro of perf counters

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57729 from idryomov/wip-66232

qa/suites/rbd: override extra_system_packages directly on install task

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #57688 into main

* refs/pull/57688/head:
tools/first-damage: Don't skip stray directory object

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57579 into main

* refs/pull/57579/head:
mds/quiesce: disable quiesce root debug parameters by default
mds/quiesce-agt: never send a synchronous ack
mds/quiesce-agt: add test for a rapid async ack
mds/quiesce: always abort fragmenting asynchronously to prevent reentrancy
mds/quiesce: overdrive an export if it hasn't frozen the tree yet
mds/quiesce: quiesce_inode should not hold on to remote auth pins
qa/cephfs: check that a completed quiesce doesn't hold remote auth pins
mds: add `--lifetime` parameter to the `lock path` asok command
mds/quiesce: accept a regular file as the quiesce root
mds: command_quiesce_path: rename `--wait` to `--await` for consistency
mds: command_quiesce_path: do not block the asok thread and return an adequate rc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev: add note about intro of perf counters

Add a note to the "perf counter" section of doc/dev/perf_counters.rst
that explains that this feature was introduced in the Reef release of
Ceph. This note will prevent us from accidentally backporting
perf-counter-related PRs to Quincy.

Signed-off-by: Zac Dover <zac.dover@proton.me>

RGW: Rename rgw::sal::Object::get_obj_size() to get_size()

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

RGW: Remove get_obj_state()/set_obj_state from SAL

RGWObjState is the state for the StoreObject class.  It has historically
been accessible via get_obj_state()/set_obj_state(), but the double
pointer nature of this access has caused multiple bugs, and the
RGWObjState itself is an implementation detail that doesn't need to be
exposed.

Instead, add a load_obj_state() that loads the state from the store, and
use proper getters/setters for the data.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #57462 from rhcs-dashboard/rgw-policy-group

mgr/dashboard: rgw policy group management api

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #57655 from AliMasarweh/wip-alimasa-bn-correct-reload

RGW|Bucket notification: reload realm correctly

Reviewed-by: yuvalif<ylifshit@redhat.com>

Merge pull request #57732 from rishabh-d-dave/doc-dev-teuth

doc/developer_guide: update doc about installing teuthology

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

test/rgw/notification: verify network access to http server

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

doc/developer_guide: update doc about installing teuthology

There are 2 more ways to install teuthology. Approach with boostrap
script is easier and more convenient while other approach is more
elaborate but manual, document both of them. Don't delete the currently
documented approach because it lets users install teuthology
conveniently in a custom virtual environment. So, keep all three.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57070 from rhcs-dashboard/fix-read-only-landing-page

mgr/dashboard: fix readonly landingpage

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57720 from zdover23/wip-doc-2024-05-27-cephfs-fs-volumes

doc/cephfs: s/subvolumegroups/subvolume groups

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57638 from yuvalif/wip-yuval-66033

test/rgw/notifications: allow http server to buffer more requests

Reviewed-By: Casey Bodley <cbodley@ibm.com>

mgr/dashboard: manage flow and pipes apis

Fixes: https://tracker.ceph.com/issues/66238
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: rgw policy group management api

Fixes: https://tracker.ceph.com/issues/66238
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57700 from tchaikov/wip-mgr-prometheus-packaging

mgr/prometheus: s/pkg_resources.packaging/packaging/

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

qa/suites/rbd: override extra_system_packages directly on install task

[1] and [2] added support for applying extra_system_packages overrides
directly on install task, but at the same time broke our long standing
workaround where we sneaked extra_system_packages directive in through
an override on ceph task.  This is likely getting addressed in [3], but
it's better to not rely on this odd feature in the first place.

[1] https://github.com/ceph/teuthology/pull/1941
[2] https://github.com/ceph/teuthology/pull/1943
[3] https://github.com/ceph/teuthology/pull/1942

Fixes: https://tracker.ceph.com/issues/66232
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #57671 from idryomov/wip-labeler-qa

.github: expand tests label to all files under qa

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #57628 from joscollin/wip-B66173-doc-per-client-metrics

doc: document metrics exported by CephFS

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: s/subvolumegroups/subvolume groups

Use the term "subvolume groups" instead of "subvolumegroups" where the
term appears in plain English. The string "subvolumegroups" is correct
in commands, and remains unchanged.

Also add formatting to command output, to make clearer that the output
is indeed output.

Fixes: https://github.com/ceph/ceph/pull/57574#discussion_r1606988766
Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: add helpers for compression in pool form

Fixes: https://tracker.ceph.com/issues/61297
Signed-off-by: avanthakkar <avanjohn@gmail.com>
Adding helpers for compression mode, algorithm, min/max blob size and
compression ratio which is set to 0.875 as default.

doc: document metrics exported by CephFS

Fixes: https://tracker.ceph.com/issues/66173
Signed-off-by: Jos Collin <jcollin@redhat.com>

crimson/os/seastore/async_cleaner: fix incorrect get_num_rolls()

The number of journal segments should not be based on the committed
journal_head. Otherwise, if a new journal segment is just opened and the
committed journal_head hasn't been updated, the result will be wrong.

This causes ceph_assert(get_segments_reclaimable() == 0) in
SegmentCleaner::get_next_reclaim_segment().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57584 from rhcs-dashboard/dashboard-pendingreleasenotes

doc: Update pendingreleasenotes for dashboard

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

mds/quiesce: disable quiesce root debug parameters by default

Fixes: https://tracker.ceph.com/issues/66225
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-agt: never send a synchronous ack

Defer to the agent thread to perform all acking.
This avoids race conditions between the updating thread
and the acking thread.

Fixes: https://tracker.ceph.com/issues/66219
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-agt: add test for a rapid async ack

In this scenario, the agent thread is able to run and generate an ack
before the db_update call returns to the caller.

Fixes: https://tracker.ceph.com/issues/66219
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: always abort fragmenting asynchronously to prevent reentrancy

Fixes: https://tracker.ceph.com/issues/66208
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: overdrive an export if it hasn't frozen the tree yet

Just like with the fragmenting, we should abort an ongoing export
if a quiesce is attempted for the directory.

To minimize the stress for the system, we only allow the abort
if the export hasn't yet managed to freeze the tree. If that is the case,
then quiesce will have to wait for the export to finish.

Fixes: https://tracker.ceph.com/issues/66123
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: quiesce_inode should not hold on to remote auth pins

1. avoid taking a remote authpin for the quiesce lock
2. drop remote authpins that were taken because of other locks

We should not be forcing a mustpin when taking quiesce lock.
This creates unnecessary overhead due to the distributed nature
of the quiesce: all ranks will execute quiesce_inode, including
the auth rank, which will authpin the inode.

Auth pinning on the auth rank is important to synchronize quiesce
with operations that are managed by the auth, like fragmenting
and exporting.

If we let a remote quiesce process take a foreign authpin then
it may block freezing on the auth, which will stall quiesce locally.
This wouldn't be a problem if the quiesce that is blocked on the auth
and the quiesce that's holding a remote authpin from the replica side
were unrelated, but in our case it may be the same logical quiesce
that effectively steps on its own toes. This creates an opportunity
for a deadlock.

Fixes: https://tracker.ceph.com/issues/66152
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

qa/cephfs: check that a completed quiesce doesn't hold remote auth pins

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: add `--lifetime` parameter to the `lock path` asok command

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce: accept a regular file as the quiesce root

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: command_quiesce_path: rename `--wait` to `--await` for consistency

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds: command_quiesce_path: do not block the asok thread and return an adequate rc

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

ceph-volume: use importlib from stdlib on Python 3.8 and up

since packaging was apparently removed from pkg_resources, let's use
importlib.metadata when it is available and pkg_resources on older
Python versions.

Refs https://tracker.ceph.com/issues/66201
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #57393 into main

* refs/pull/57393/head:
doc/man: add missing long option switches

Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge pull request #57609 from Svelar/asan_rbd_mirror

test/rbd-mirror: free remote_journaler in PrepareRemoteImageRequest tests

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #56033 from yuvalif/wip-yuval-kafka-cleanup

rgw/kafka: do not destroy the connection on errors

Reviewed-By: Krunal Chheda <kchheda3@bloomberg.net>, Casey Bodley <cbodley@ibm.com>

Merge pull request #57701 from zdover23/wip-doc-2024-05-25-cephfs-fs-volumes

doc/cephfs: disambiguate two sentences

doc/cephfs: disambiguate two sentences

Rewrite two sentences so that they are unambgiuously in the indicative
mood and not, as they previously seemed, in the imperative mood.

Signed-off-by: Zac Dover <zac.dover@proton.me>

test/rbd: free remote_journaler in PrepareRemoteImageRequest tests

When sanitizer is enabled, unittest_rbd_mirror shows

```
=================================================================
==1377627==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2 byte(s) in 2 object(s) allocated from:
    #0 0xaaaac462d088 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rbd_mirror+0x30cd088) (BuildId: 054d3b6699c622daad91c7f70e36616220fbd5ad)
    #1 0xaaaac522f13c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::get_client() /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:148:24
    #2 0xaaaac522e60c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int) /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:120:5
    #3 0xaaaac5232d1c in librbd::util::detail::C_CallbackAdapter<rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>, &(rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int))>::finish(int) /root/ceph/src/librbd/Utils.h:63:5
    #4 0xaaaac4863428 in Context::complete(int) /root/ceph/src/include/Context.h:99:5
    #5 0xaaaac489ac30 in librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()::operator()() const /root/ceph/src/librbd/asio/ContextWQ.h:31:12
    #6 0xaaaac489aaf4 in boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>::operator()() /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:60:5
    #7 0xaaaac489aaac in void boost::asio::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3
    #8 0xaaaac489aa4c in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #9 0xaaaac489a9ec in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>*) /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:111:3
    #10 0xaaaac4899c00 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #11 0xaaaac489c814 in boost::asio::detail::executor_op<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7
    #12 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5
    #13 0xaaaac489f074 in boost::asio::detail::strand_executor_service::run_ready_handlers(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>&) /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.ipp:150:8
    #14 0xaaaac489ed2c in boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.hpp:136:5
    #15 0xaaaac489ebd4 in void boost::asio::asio_handler_invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3
    #16 0xaaaac489e284 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3
    #17 0xaaaac48a3e08 in boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7
    #18 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5
    #19 0xaaaac6766b1c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:493:12
    #20 0xaaaac67660a4 in boost::asio::detail::scheduler::run(boost::system::error_code&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:210:10
    #21 0xaaaac7562d48 in boost::asio::io_context::run() /root/ceph/build/boost/include/boost/asio/impl/io_context.ipp:64:24
    #22 0xaaaac7562bcc in ceph::async::io_context_pool::start(short)::'lambda'()::operator()() const /root/ceph/src/common/async/context_pool.h:69:16
    #23 0xaaaac7562b18 in void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #24 0xaaaac7562960 in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #25 0xaaaac7562804 in std::invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/functional:97:14
    #26 0xaaaac75627dc in auto std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)::operator()<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) const /root/ceph/src/common/Thread.h:79:10
    #27 0xaaaac7562728 in ceph::async::io_context_pool::start(short)::'lambda'() std::__invoke_impl<void, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)&&, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #28 0xaaaac75624ec in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #29 0xaaaac756231c in void std::thread::_Invoker<std::tuple<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13

SUMMARY: AddressSanitizer: 2 byte(s) leaked in 2 allocation(s).
```

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Merge pull request #57693 from tchaikov/cmake-shec_utils

cmake: : link shec_utils against legacy-option-headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/prometheus: s/pkg_resources.packaging/packaging/

instead of relying on the internal implementation of pkg_resources,
let's import packaging directly.

in this change, we also add the dependency to the packaging module
in the packaging recipes.

See also https://github.com/pypa/setuptools/issues/4385

Fixes https://tracker.ceph.com/issues/66201
Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #57686 into main

* refs/pull/57686/head:
script/ceph-backport: update Ceph PR id

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57689 from kotreshhr/fix_unpack_first_damage

tools/first-damage: Fix struct unpack

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/rbm/avlallocator: return enough regions when request size is greater than max_alloc_size

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

cmake: : link shec_utils against legacy-option-headers

in c24a6ffe20, we tried to link all target dependent on legacy option
headers against legacy-option-headers, but we missed some of them.
in our CI, we spotted build failure like:
```
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/erasure-code/ErasureCode.cc:26:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/osd/osd_types.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.h:41:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config_proxy.h:6:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config.h:27:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/config_values.h:59:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/options/legacy_config_opts.h:1:10: fatal error: 'global_legacy_options.h' file not found
         ^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
so in this change, we link the shec_utils to `legacy-option-headers`
as well to fulfill the build dependency.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

tools/first-damage: Fix struct unpack

The first field is a 64 bit integer, the script
was unpacking it as 32 bit integer. Fixed the
same.

Also printing the 'first' value in debug

Signed-off-by: Kotresh HR <khiremat@redhat.com>

tools/first-damage: Don't skip stray directory object

Signed-off-by: Kotresh HR <khiremat@redhat.com>