qa/tasks/radosbench: make use of concurrency flag

This affects mainly radosbench-high-concurrency.yaml
The concurrency flag was only used in non-write scenarios to
populate the initial data.
We should use this flag with `write` mode as well.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61945 from kotreshhr/snapshot-referent-inodes

mds: snapshot referent inodes feature

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #60014 from rhcs-dashboard/cephadm-e2e-endless-fix-1

mgr/dashboard: fix cephadm dashboard e2e and a combobox issue

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #62245 from afreen23/wip-disable-rgw-test

mgr/dashboard: Disable MFA test

Merge pull request #62249 from anthonyeleven/improvedmi

doc/monitoring: Improve index.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: skip nfs exports check

nfs service creation is not working because of some issue with cephadm
for a while now. disabling it until its resolved

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: save video only on failure

as per
https://docs.cypress.io/app/guides/screenshots-and-videos#Delete-videos-for-specs-without-failing-or-retried-tests.

This would save storage as saving entire video is useless

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix rgw and osd regression in e2e

use aria-label as the selector in tests.

data-testid is not supported through this carbon cds-checkbox
input so we can't use it

regression started after started after #61319 and
https://github.com/ceph/ceph/pull/61478/files#diff-7bd2a1f48bd2a7bcebb744eb04d84706087b6b3c2651c4b1ad085b4b6e03fc6d
where multi-select is removed

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: undefined while creating rbd image

this only happens on a fresh cluster but consistently reproducible in a
test environment. also cephfs also throws some error occasionally which
is also being handled similarly

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: cypress failure after angular upgrade

after angular upgrade, the tests started failing with
An uncaught error was detected outside of a test:
     TypeError: The following error originated from your test code, not from Cypress.
Cannot read properties of undefined (reading 'value')

which seems to originate from the page-helper.po. So handling the
undefined value.

Example run: https://jenkins.ceph.com/job/ceph-dashboard-pull-requests/17541/consoleFull#204663575173a8703b-5adb-41c5-84a0-8cf4e065ba3d

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fixing cephadm dashboard e2e failures

the wizard component changes caused issue to the e2e, so fixing it.
It was uncaught before because of the test were unable to run due to
some lab issues.

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix osd list search with id

The osd list search with id is broken after the pagination. So if you
search for any id it won't retrieve anything.

PS: One more broken item is the searching with any other cell name but
that's a known issue with server side pagination.

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: reorder test runs

I saw cypress starting the grafana tests even before the cluster is
setup through the expansion tests. so running grafana separately

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: add custom items to combo box

previously we were able to add custom items to our select-badges like
custom labels for hosts. but it got dropped unintentionally due to the
carbon. fixing it here

Fixes: https://tracker.ceph.com/issues/68871
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59516 from sajibreadd/wip-67700

OSD: Split osd_recovery_sleep into settings applied to degraded or clean PGs

Merge pull request #62242 from rhcs-dashboard/pin-xmlsec

mgr/dashboard: pin lxml to fix run-dashboard-tox-make-check failure

Merge PR #62188 into main

* refs/pull/62188/head:
librbd/ConfigWatcher: replace obsolete get_tracked_conf_keys()
rbd/throttler: replace obsolete get_tracked_conf_keys()
mds: replace obsolete get_tracked_conf_keys()
nvmeof: replace obsolete get_tracked_conf_keys()

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #62136 into main

* refs/pull/62136/head:
qa: add test case for remove xattr
cephfs-shell: Add the option to remove xattr

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #62062 into main

* refs/pull/62062/head:
qa/cephfs: avoid unnecessary importing in test_volumes.py

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

doc/monitoring: Improve index.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #61544 from rkachach/fix_automated_cert_rotation_69581

Introducing certmgr: a unified certificate management system for Cephadm

Reviewed-by: Adam King <adking@redhat.com>

mgr/dashboard: Disable MFA test

Fixes https://tracker.ceph.com/issues/46735

Disabling the test due to flakiness

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #61536 from Matan-B/wip-matanb-backfill-scan

crimson/osd/recovery_backend: scan_for_backfill to use obc_manager

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62226 from ronen-fr/wip-rf-bitset-fmt

common: fmt support for bitset_set & mini_flat_map

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: pin lxml to fix run-dashboard-tox-make-check failure

xmlsec had an upgrade yesterday night and python3-saml might need to
adapt its library accordingly I suppose. Testing a fix by pinning lxml

Another approach is being tried out separately
https://github.com/ceph/ceph/pull/62239, but that is failing with some
other errors.

Fixes: https://tracker.ceph.com/issues/70411
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #62217 from aainscow/flat_maps_fix

common:  Remove unnecessary assert checking unsigned number is 0 or greater in bitset_set.h

Merge pull request #61828 from clwluvw/datasync-useracl

rgw: prevent data sync from replicating to buckets not owned by the user

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #60714 from aclamk/wip-aclamk-bluefs-unittest-async-compaction

os/bluestore: New unit test for BlueFS async compaction

Merge pull request #61810 from Jayaprakash-ibm/wip-bluestore-onode-test-enhancement

os/bluestore : Fill onode test instances with sample data

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #61900 from aclamk/wip-aclamk-bs-fix-get-unused-mask

os/bluestore: implemented bluestore_blob_t::get_unused_mask

Reviewed-by: Chunmei Liu <chunmei.liu@ibm.com>

common: Remove unnecessary assert checking unsigned number is 0 or greater.

Minor compiler warning caused by unsigned comparison with zero. Clearly
assert is not necessary and so was removed. Wrapping scenarios are
policed using remaining assert code.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61829 from rkachach/fix_issue_69971

mgr/cephadm: Add virtual_ip as a param for mgmt-gateway apply cmd

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #62085 from afreen23/wip-sso

mgr/dashboard: Improve sso role mapping

Merge pull request #60530 from aclamk/wip-aclamk-mute-deferred-test

os/bluestore: Disable invoking unittest_deferred

Merge pull request #62214 from Matan-B/wip-matanb-revert-60753

Revert "os/bluestore: Fix problem with deferred writes replay"

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #61476 from piyushagarwal1411/pool

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in pool.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61474 from piyushagarwal1411/osd

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in osd.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61463 from piyushagarwal1411/host

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in host.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61210 from piyushagarwal1411/cephfs

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in cephfs.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

test: test the fmt support for bitset_set & mini_flat_map

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: fmt support for bitset_set & mini_flat_map

Note: both classes have begin()/end() methods, which
required an explicit opt-out of fmt/range default
handling.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #62126 from vshankar/wip-70287

mds: include deleted file dentry in snapdiff readdir result

mgr/cepahdm: adding a message when certificates check is OK

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: addressing reviewer comments

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: fixing certificate get_status_description output

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding new debug flag for testing purposes only

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: fixing docs and defaults for some options

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: improving individual certificates checks

Some refactoring to improve the individual certificates checking and
use the same code for both cases: certs with and without keys are

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: added some logic to avoid loading unknown certs/keys

Those keys can be already present in the monstore because they were
used/generated by previous versions of cert_store (new certmgr). Since
they are not in use anymore we should not load them.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: reverting usage of Org Name 'Ceph' in self-signed certs

This could cause issues on upgrades as it imply chaning the root CA CN

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding new comands to remvoe keys/certificates

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: adding code to store self-signed rgw certificate

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

doc/cepahadm: adding new file for certmgr documentation

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding cephadm mgr-module to the gitleaks file

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adjusting grafana custom code to handle certificates

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: fixing cert user_made and pre_remove actions of mgmt-gw

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: fixing oauth_proxy pre_remove action

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding UT and adjusting existing unit-tests

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: renaming cert-store cmds to certmgr, adding new cmds

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adding certificates validationg and management logic

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: refactor cert_store to use the same code for cert/key

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Revert "os/bluestore: Add debug hooks"

This reverts commit cbfc24dd7e9b3d13b76c340b60d8d56f13f75ae6.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Deferred writes unit test, replicates corruption"

This reverts commit b2086ef0c96362b6bab4666e635cf26e3f01b40d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Modify _deferred_replay"

This reverts commit c10a794f72613ffdccdd64e24840dfce93174f30.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Split _deferred_replay into 1) apply IO and 2) remove keys"

This reverts commit f2b33bc9719fa5f7bafbbd0a4e591ae350fc85af.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "test/unittest_deferred: debug code"

This reverts commit fff713965ecc5ce4ae80a299d1975cb23b31c3e9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "more debug code"

This reverts commit e1e840628d3842e6664d2f1305a821abf6b631d2.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "fixup"

This reverts commit 5935a0f295a5762bedbe8d64e7d94b88ac81c2cb.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61963 from myoungwon/wip-qa-seastore-rbm-test

crimson/os/seastore: add QA test for seastore with RBM under rados_simple

Merge pull request #61783 from adk3798/cephadm-nvmeof-one-daemon-per-node

mgr/cephadm: block deploying nvmeof daemons of different groups on same host

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61887 from ShwetaBhosale1/fix_issue_70060_port_4721_is_not_opened_while_deploying_agent

mgr/cephadm: Open port 4721 while deployling cephadm agent

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #62189 from idryomov/wip-66419

qa/workunits/rbd: wait for resize to be applied in rbd-nbd

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #62013 from Matan-B/wip-matanb-crimson-coroutine-fixes

crimson: revert coroutine workarounds

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60403 from trociny/wip-68206

rgw: trigger resharding of versioned buckets sooner

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill pass start by value

See: https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rcoro-reference-parameters

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill to use seastar::lambda

We could also use here seastar::coroutine::parallel_for_each. However,
an `interruptible` overload must be used. Instead, use the coroutine
which wrapper is simpler this time.

```
kernel callstack:
    #0 0x56f6e7a in seastar::lw_shared_ptr<std::map<hobject_t, eversion_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, eversion_t> > > >::operator->() const /home/matan/ceph/src/seastar/include/seastar/core/shared_ptr.hh:347
    #1 0x56f6e7a in operator() /home/matan/ceph/src/crimson/osd/recovery_backend.cc:245
    #2 0x5286c62 in std::__n4861::coroutine_handle<crimson::internal::promise_base<crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>, void, void> >::resume() const /opt/rh/gcc-toolset-13/root/usr/include/c++/13/coroutine:24

SUMMARY: AddressSanitizer: stack-use-after-return /home/matan/ceph/src/seastar/include/seastar/core/shared_ptr.hh:347 in seastar::lw_shared_ptr<std::map<hobject_t, eversion_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, eversion_t> >

```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/crimson/test_crimson_coroutine: add test_coroutine_loops

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: _handle_pull_response to use obc_manager

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/.../replicate_recovery_backend: remove unnecessary check

Already checked in handle_recovery_op.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill to use obc_manager

let obc_loader get the relevant obc and avoid duplicating the
loading logic.

Fixes: https://tracker.ceph.com/issues/69154
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: move scan_for_backfill to coroutine

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: Improve sso role mapping

Fixes https://tracker.ceph.com/issues/70366

- adds support for ISV
- using jmespath expression for fetching roles from payload
- added roles_path parameter in sso enable command as optional argument
- modified roles mapper for sso login

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60753 from aclamk/wip-aclamk-more-deferred-overwrite-fix

os/bluestore: Fix problem with deferred writes replay
This PR had gone through testing, including teuthology, but not in its final form.
But the issue its fixed is urgent.
This PR is merged before final teuthology run.

Merge pull request #62146 from Matan-B/wip-matanb-crimson-wip-69439-v2

crimson: Handle peer replies handling once received

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62118 from rhcs-dashboard/update-proto-files-dashboard

mgr/dashboard: update dashboard nvmeof proto files

Merge pull request #62176 from anthonyeleven/weofferedthereleasesorder

doc/releases: Add ordering comment to releases.yml

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61224 from CookiePLMonster/patch-22

dokan: fix a Y2038 bug by replacing `Int32x32To64` with multiplication

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

qa/workunits/rbd: wait for resize to be applied in rbd-nbd

Implement the same logic as in commit 6f3d0f570f1a ("test/librbd/fsx:
wait for resize to propagate in krbd_resize()").

Fixes: https://tracker.ceph.com/issues/66419
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/ConfigWatcher: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

rbd/throttler: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

mds: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

As per the original code in this specific case: the
vector of keys is expected to be sorted.

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

nvmeof: replace obsolete get_tracked_conf_keys()

.. with get_tracked_keys().

Following https://github.com/ceph/ceph/pull/61394,
all uses of the deprecated interface will be updated,
and that old interface will be removed.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61753 from aainscow/flat_maps

common: Introduce efficient map and set structure useful for EC.

Merge pull request #62074 from ronen-fr/wip-rf-more4-keys

mon,mgr,auth,client: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62133 from ronen-fr/wip-rf-more6-keys

tests: replace obsolete get_tracked_conf_keys()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62173 from guits/fix-70363

ceph-volume: allow zapping partitions on multipath devices

Merge pull request #61669 from badone/wip-remove-openstack.yaml

qa: Remove openstack.yaml file which is no longer relevant

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Laura Flores <lflores@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

mgr/dashboard: update dashboard nvmeof proto files

Fixes: https://tracker.ceph.com/issues/70330
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/releases: Add ordering comment to releases.yml

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #61624 from VallariAg/wip-nvmeof-ceph-status

mon: Add nvmeof group/gateway name in  "ceph -s"