Merge pull request #62274 from phlogistonjohn/jjm-more-bwc-fixes

build-with-container: two small fixes

Merge pull request #62252 from ronen-fr/wip-rf-fix-62188

librbd/ConfigWatcher: fix a missing 'override'

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #54396 into main

* refs/pull/54396/head:
PendingReleaseNote: add note for pause_purging and pause_cloninig
doc/cephfs: add note for config option pause_purging and pause_cloning
qa/cephfs: add tests for mgr/vol config pause_cloning
qa/cephfs: extend wait for trash empty
qa/cephfs: add tests for config option pause_purging
qa/cephfs: don't strip any whitespace for get_shell_stdout
mgr/vol: add pause/resume mechanism for async jobs

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #62240 into main

* refs/pull/62240/head:
qa/cephfs: require-osd-release should now be tentacle

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #62200 into main

* refs/pull/62200/head:
common/include: Add map type election and fmt::format to interval_map

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #59780 from irq0/wip/osd-asok-messenger-dump

Add Asok Command: Dump Messenger Status (Connections, TCP stats, ..)

Merge PR #62230 into main

* refs/pull/62230/head:
client: avoid multiple calls to path_walk

Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #61791 from MaxKellermann/common_includes

common: add missing includes

Reviewed-By: Adam Emerson <aemerson@ibm.com>

Merge pull request #62160 from cbodley/wip-qa-vault-list

qa/vault: create_secrets() cleanup checks for orphaned keys

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

script: ensure curl is always available in build containers

Ensure that curl is installed in all build containers regardless of
ceph's dependencies or other factors. This allows us to use curl in
any subsequent build steps/scripts.

Fixes: https://tracker.ceph.com/issues/70451
Signed-off-by: John Mulligan <jmulligan@redhat.com>

script/build-with-container: fix incorrect build invocation for tests

The `tests` build target was set to build the "normal" target but it
needs to build the tests.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61945 from kotreshhr/snapshot-referent-inodes

mds: snapshot referent inodes feature

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60014 from rhcs-dashboard/cephadm-e2e-endless-fix-1

mgr/dashboard: fix cephadm dashboard e2e and a combobox issue

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #62245 from afreen23/wip-disable-rgw-test

mgr/dashboard: Disable MFA test

Merge pull request #62249 from anthonyeleven/improvedmi

doc/monitoring: Improve index.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: skip nfs exports check

nfs service creation is not working because of some issue with cephadm
for a while now. disabling it until its resolved

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: save video only on failure

as per
https://docs.cypress.io/app/guides/screenshots-and-videos#Delete-videos-for-specs-without-failing-or-retried-tests.

This would save storage as saving entire video is useless

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix rgw and osd regression in e2e

use aria-label as the selector in tests.

data-testid is not supported through this carbon cds-checkbox
input so we can't use it

regression started after started after #61319 and
https://github.com/ceph/ceph/pull/61478/files#diff-7bd2a1f48bd2a7bcebb744eb04d84706087b6b3c2651c4b1ad085b4b6e03fc6d
where multi-select is removed

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: undefined while creating rbd image

this only happens on a fresh cluster but consistently reproducible in a
test environment. also cephfs also throws some error occasionally which
is also being handled similarly

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: cypress failure after angular upgrade

after angular upgrade, the tests started failing with
An uncaught error was detected outside of a test:
     TypeError: The following error originated from your test code, not from Cypress.
Cannot read properties of undefined (reading 'value')

which seems to originate from the page-helper.po. So handling the
undefined value.

Example run: https://jenkins.ceph.com/job/ceph-dashboard-pull-requests/17541/consoleFull#204663575173a8703b-5adb-41c5-84a0-8cf4e065ba3d

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fixing cephadm dashboard e2e failures

the wizard component changes caused issue to the e2e, so fixing it.
It was uncaught before because of the test were unable to run due to
some lab issues.

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix osd list search with id

The osd list search with id is broken after the pagination. So if you
search for any id it won't retrieve anything.

PS: One more broken item is the searching with any other cell name but
that's a known issue with server side pagination.

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: reorder test runs

I saw cypress starting the grafana tests even before the cluster is
setup through the expansion tests. so running grafana separately

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: add custom items to combo box

previously we were able to add custom items to our select-badges like
custom labels for hosts. but it got dropped unintentionally due to the
carbon. fixing it here

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

PendingReleaseNote: add note for pause_purging and pause_cloninig

Added release notes for mgr/vol config option "pause_purging and
"pause_cloning".

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/cephfs: add note for config option pause_purging and pause_cloning

Update documentation for add information about mgr/vol config options
"pause_purging" and "pause_cloning".

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add tests for mgr/vol config pause_cloning

mgr/vol config option pause_cloning allows pausing of cloner threads.
Add tests for this.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: extend wait for trash empty

Trash directory for a volume is not created by default. If
_wait_for_trash_empty() in test_volumes.py encounters absence of trash
directory, return true.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add tests for config option pause_purging

Setting MGR config option mgr/volumes/pause_purging to true halts
all ongoing purges and allows no new purging to begin until this option
is changed to false. Add tests for this.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: don't strip any whitespace for get_shell_stdout

Whitespace is not removed from the end of the stdout returned by the
method get_ceph_cmd_stdout(). Follow the same policy here since it is
better to not do so (this whitespace can be useful, when copying Ceph
auth keyrings from stdout to a file) and also for sake of uniformity of
interfaces.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/vol: add pause/resume mechanism for async jobs

Add mechansim that allows pausing/resuming of the entire async job
machinery that queues, launches and picks next async job; both async
jobs, clones as well as purges.

And then add mgr/vol config option pause_purging and pause_cloning so
that both of these async jobs can be paused and resumed individually.

Fixes: https://tracker.ceph.com/issues/61903
Fixes: https://tracker.ceph.com/issues/68630
Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vault: create_secrets() cleanup checks for orphaned keys

the vault task creates some keys on startup for use in s3-tests. on
cleanup, check that rgw has removed any temporary bucket keys that
were created during testing. fail with an assertion if the listed keys
differ

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59516 from sajibreadd/wip-67700

OSD: Split osd_recovery_sleep into settings applied to degraded or clean PGs

Merge pull request #62242 from rhcs-dashboard/pin-xmlsec

mgr/dashboard: pin lxml to fix run-dashboard-tox-make-check failure

librbd/ConfigWatcher: fix a missing 'override'

dropped in PR#62188

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #62188 into main

* refs/pull/62188/head:
librbd/ConfigWatcher: replace obsolete get_tracked_conf_keys()
rbd/throttler: replace obsolete get_tracked_conf_keys()
mds: replace obsolete get_tracked_conf_keys()
nvmeof: replace obsolete get_tracked_conf_keys()

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #62136 into main

* refs/pull/62136/head:
qa: add test case for remove xattr
cephfs-shell: Add the option to remove xattr

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #62062 into main

* refs/pull/62062/head:
qa/cephfs: avoid unnecessary importing in test_volumes.py

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

doc/monitoring: Improve index.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #61544 from rkachach/fix_automated_cert_rotation_69581

Introducing certmgr: a unified certificate management system for Cephadm

Reviewed-by: Adam King <adking@redhat.com>

mgr/dashboard: Disable MFA test

Fixes https://tracker.ceph.com/issues/46735

Disabling the test due to flakiness

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #61536 from Matan-B/wip-matanb-backfill-scan

crimson/osd/recovery_backend: scan_for_backfill to use obc_manager

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62226 from ronen-fr/wip-rf-bitset-fmt

common: fmt support for bitset_set & mini_flat_map

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: pin lxml to fix run-dashboard-tox-make-check failure

xmlsec had an upgrade yesterday night and python3-saml might need to
adapt its library accordingly I suppose. Testing a fix by pinning lxml

Another approach is being tried out separately
https://github.com/ceph/ceph/pull/62239, but that is failing with some
other errors.

Fixes: https://tracker.ceph.com/issues/70411
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/cephfs: require-osd-release should now be tentacle

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #62217 from aainscow/flat_maps_fix

common:  Remove unnecessary assert checking unsigned number is 0 or greater in bitset_set.h

common/include: Add map type election and fmt::format to interval_map

Add capability to choose the type of map used in interval_map.

The default will be std::map and so should not change any existing
implementation.

This behaviour is already available in interval_set. The intention
is to enable a more efficient map library which suits a particular
application.

The optimized EC code, which is coming soon, will use a boost::flat_map
here since it deals with large numbers of very small (usually single-entry)
interval_maps.

Also adds fmt::format support and refactors print functions in both
set and map.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61828 from clwluvw/datasync-useracl

rgw: prevent data sync from replicating to buckets not owned by the user

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

client: avoid multiple calls to path_walk

When we wrap a dname after a path walk, we cannot do another path walk.
Originally this method was setup to permit that but with (upcoming) encryption
it cannot work. The dname from the path_walk is in the encrypted namespace so
we cannot do another lookup in a second path_walk.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #60714 from aclamk/wip-aclamk-bluefs-unittest-async-compaction

os/bluestore: New unit test for BlueFS async compaction

Merge pull request #61810 from Jayaprakash-ibm/wip-bluestore-onode-test-enhancement

os/bluestore : Fill onode test instances with sample data

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #61900 from aclamk/wip-aclamk-bs-fix-get-unused-mask

os/bluestore: implemented bluestore_blob_t::get_unused_mask

Reviewed-by: Chunmei Liu <chunmei.liu@ibm.com>

common: Remove unnecessary assert checking unsigned number is 0 or greater.

Minor compiler warning caused by unsigned comparison with zero. Clearly
assert is not necessary and so was removed. Wrapping scenarios are
policed using remaining assert code.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61829 from rkachach/fix_issue_69971

mgr/cephadm: Add virtual_ip as a param for mgmt-gateway apply cmd

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #62085 from afreen23/wip-sso

mgr/dashboard: Improve sso role mapping

Merge pull request #60530 from aclamk/wip-aclamk-mute-deferred-test

os/bluestore: Disable invoking unittest_deferred

Merge pull request #62214 from Matan-B/wip-matanb-revert-60753

Revert "os/bluestore: Fix problem with deferred writes replay"

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #61476 from piyushagarwal1411/pool

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in pool.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61474 from piyushagarwal1411/osd

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in osd.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61463 from piyushagarwal1411/host

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in host.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61210 from piyushagarwal1411/cephfs

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in cephfs.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

test: test the fmt support for bitset_set & mini_flat_map

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: fmt support for bitset_set & mini_flat_map

Note: both classes have begin()/end() methods, which
required an explicit opt-out of fmt/range default
handling.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #62126 from vshankar/wip-70287

mds: include deleted file dentry in snapdiff readdir result

mgr/cepahdm: adding a message when certificates check is OK

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: addressing reviewer comments

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: fixing certificate get_status_description output

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding new debug flag for testing purposes only

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: fixing docs and defaults for some options

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: improving individual certificates checks

Some refactoring to improve the individual certificates checking and
use the same code for both cases: certs with and without keys are

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: added some logic to avoid loading unknown certs/keys

Those keys can be already present in the monstore because they were
used/generated by previous versions of cert_store (new certmgr). Since
they are not in use anymore we should not load them.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: reverting usage of Org Name 'Ceph' in self-signed certs

This could cause issues on upgrades as it imply chaning the root CA CN

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding new comands to remvoe keys/certificates

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: adding code to store self-signed rgw certificate

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

doc/cepahadm: adding new file for certmgr documentation

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding cephadm mgr-module to the gitleaks file

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adjusting grafana custom code to handle certificates

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: fixing cert user_made and pre_remove actions of mgmt-gw

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: fixing oauth_proxy pre_remove action

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding UT and adjusting existing unit-tests

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: renaming cert-store cmds to certmgr, adding new cmds

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adding certificates validationg and management logic

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: refactor cert_store to use the same code for cert/key

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Revert "os/bluestore: Add debug hooks"

This reverts commit cbfc24dd7e9b3d13b76c340b60d8d56f13f75ae6.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Deferred writes unit test, replicates corruption"

This reverts commit b2086ef0c96362b6bab4666e635cf26e3f01b40d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Modify _deferred_replay"

This reverts commit c10a794f72613ffdccdd64e24840dfce93174f30.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Split _deferred_replay into 1) apply IO and 2) remove keys"

This reverts commit f2b33bc9719fa5f7bafbbd0a4e591ae350fc85af.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "test/unittest_deferred: debug code"

This reverts commit fff713965ecc5ce4ae80a299d1975cb23b31c3e9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "more debug code"

This reverts commit e1e840628d3842e6664d2f1305a821abf6b631d2.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "fixup"

This reverts commit 5935a0f295a5762bedbe8d64e7d94b88ac81c2cb.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61963 from myoungwon/wip-qa-seastore-rbm-test

crimson/os/seastore: add QA test for seastore with RBM under rados_simple

Merge pull request #61783 from adk3798/cephadm-nvmeof-one-daemon-per-node

mgr/cephadm: block deploying nvmeof daemons of different groups on same host

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61887 from ShwetaBhosale1/fix_issue_70060_port_4721_is_not_opened_while_deploying_agent

mgr/cephadm: Open port 4721 while deployling cephadm agent

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #62189 from idryomov/wip-66419

qa/workunits/rbd: wait for resize to be applied in rbd-nbd

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #62013 from Matan-B/wip-matanb-crimson-coroutine-fixes

crimson: revert coroutine workarounds

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60403 from trociny/wip-68206

rgw: trigger resharding of versioned buckets sooner

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill pass start by value

See: https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rcoro-reference-parameters

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill to use seastar::lambda

We could also use here seastar::coroutine::parallel_for_each. However,
an `interruptible` overload must be used. Instead, use the coroutine
which wrapper is simpler this time.

```
kernel callstack:
    #0 0x56f6e7a in seastar::lw_shared_ptr<std::map<hobject_t, eversion_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, eversion_t> > > >::operator->() const /home/matan/ceph/src/seastar/include/seastar/core/shared_ptr.hh:347
    #1 0x56f6e7a in operator() /home/matan/ceph/src/crimson/osd/recovery_backend.cc:245
    #2 0x5286c62 in std::__n4861::coroutine_handle<crimson::internal::promise_base<crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>, void, void> >::resume() const /opt/rh/gcc-toolset-13/root/usr/include/c++/13/coroutine:24

SUMMARY: AddressSanitizer: stack-use-after-return /home/matan/ceph/src/seastar/include/seastar/core/shared_ptr.hh:347 in seastar::lw_shared_ptr<std::map<hobject_t, eversion_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, eversion_t> >

```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/crimson/test_crimson_coroutine: add test_coroutine_loops

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: _handle_pull_response to use obc_manager

Signed-off-by: Matan Breizman <mbreizma@redhat.com>