rbd: print the return codes on error

The calls in the mirror pool cli do not always print the error code.
This is now done in order to make debugging easier.

Fixes: https://tracker.ceph.com/issues/70444
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

Merge pull request #61544 from rkachach/fix_automated_cert_rotation_69581

Introducing certmgr: a unified certificate management system for Cephadm

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #61536 from Matan-B/wip-matanb-backfill-scan

crimson/osd/recovery_backend: scan_for_backfill to use obc_manager

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62226 from ronen-fr/wip-rf-bitset-fmt

common: fmt support for bitset_set & mini_flat_map

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62217 from aainscow/flat_maps_fix

common:  Remove unnecessary assert checking unsigned number is 0 or greater in bitset_set.h

Merge pull request #61828 from clwluvw/datasync-useracl

rgw: prevent data sync from replicating to buckets not owned by the user

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #60714 from aclamk/wip-aclamk-bluefs-unittest-async-compaction

os/bluestore: New unit test for BlueFS async compaction

Merge pull request #61810 from Jayaprakash-ibm/wip-bluestore-onode-test-enhancement

os/bluestore : Fill onode test instances with sample data

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #61900 from aclamk/wip-aclamk-bs-fix-get-unused-mask

os/bluestore: implemented bluestore_blob_t::get_unused_mask

Reviewed-by: Chunmei Liu <chunmei.liu@ibm.com>

common: Remove unnecessary assert checking unsigned number is 0 or greater.

Minor compiler warning caused by unsigned comparison with zero. Clearly
assert is not necessary and so was removed. Wrapping scenarios are
policed using remaining assert code.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61829 from rkachach/fix_issue_69971

mgr/cephadm: Add virtual_ip as a param for mgmt-gateway apply cmd

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #62085 from afreen23/wip-sso

mgr/dashboard: Improve sso role mapping

Merge pull request #60530 from aclamk/wip-aclamk-mute-deferred-test

os/bluestore: Disable invoking unittest_deferred

Merge pull request #62214 from Matan-B/wip-matanb-revert-60753

Revert "os/bluestore: Fix problem with deferred writes replay"

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #61476 from piyushagarwal1411/pool

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in pool.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61474 from piyushagarwal1411/osd

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in osd.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61463 from piyushagarwal1411/host

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in host.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #61210 from piyushagarwal1411/cephfs

mgr/dashboard: Changing SimpleGraphPanel to TimeSeries Panel in cephfs.libsonnet to fix inconsistency in Line Graphs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

test: test the fmt support for bitset_set & mini_flat_map

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: fmt support for bitset_set & mini_flat_map

Note: both classes have begin()/end() methods, which
required an explicit opt-out of fmt/range default
handling.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #62126 from vshankar/wip-70287

mds: include deleted file dentry in snapdiff readdir result

mgr/cepahdm: adding a message when certificates check is OK

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: addressing reviewer comments

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: fixing certificate get_status_description output

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding new debug flag for testing purposes only

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: fixing docs and defaults for some options

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: improving individual certificates checks

Some refactoring to improve the individual certificates checking and
use the same code for both cases: certs with and without keys are

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: added some logic to avoid loading unknown certs/keys

Those keys can be already present in the monstore because they were
used/generated by previous versions of cert_store (new certmgr). Since
they are not in use anymore we should not load them.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: reverting usage of Org Name 'Ceph' in self-signed certs

This could cause issues on upgrades as it imply chaning the root CA CN

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding new comands to remvoe keys/certificates

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahadm: adding code to store self-signed rgw certificate

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

doc/cepahadm: adding new file for certmgr documentation

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding cephadm mgr-module to the gitleaks file

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adjusting grafana custom code to handle certificates

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: fixing cert user_made and pre_remove actions of mgmt-gw

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: fixing oauth_proxy pre_remove action

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: adding UT and adjusting existing unit-tests

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: renaming cert-store cmds to certmgr, adding new cmds

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cepahdm: adding certificates validationg and management logic

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: refactor cert_store to use the same code for cert/key

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Revert "os/bluestore: Add debug hooks"

This reverts commit cbfc24dd7e9b3d13b76c340b60d8d56f13f75ae6.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Deferred writes unit test, replicates corruption"

This reverts commit b2086ef0c96362b6bab4666e635cf26e3f01b40d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Modify _deferred_replay"

This reverts commit c10a794f72613ffdccdd64e24840dfce93174f30.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "os/bluestore: Split _deferred_replay into 1) apply IO and 2) remove keys"

This reverts commit f2b33bc9719fa5f7bafbbd0a4e591ae350fc85af.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "test/unittest_deferred: debug code"

This reverts commit fff713965ecc5ce4ae80a299d1975cb23b31c3e9.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "more debug code"

This reverts commit e1e840628d3842e6664d2f1305a821abf6b631d2.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "fixup"

This reverts commit 5935a0f295a5762bedbe8d64e7d94b88ac81c2cb.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61963 from myoungwon/wip-qa-seastore-rbm-test

crimson/os/seastore: add QA test for seastore with RBM under rados_simple

Merge pull request #61783 from adk3798/cephadm-nvmeof-one-daemon-per-node

mgr/cephadm: block deploying nvmeof daemons of different groups on same host

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61887 from ShwetaBhosale1/fix_issue_70060_port_4721_is_not_opened_while_deploying_agent

mgr/cephadm: Open port 4721 while deployling cephadm agent

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #62189 from idryomov/wip-66419

qa/workunits/rbd: wait for resize to be applied in rbd-nbd

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #62013 from Matan-B/wip-matanb-crimson-coroutine-fixes

crimson: revert coroutine workarounds

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60403 from trociny/wip-68206

rgw: trigger resharding of versioned buckets sooner

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill pass start by value

See: https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rcoro-reference-parameters

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill to use seastar::lambda

We could also use here seastar::coroutine::parallel_for_each. However,
an `interruptible` overload must be used. Instead, use the coroutine
which wrapper is simpler this time.

```
kernel callstack:
    #0 0x56f6e7a in seastar::lw_shared_ptr<std::map<hobject_t, eversion_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, eversion_t> > > >::operator->() const /home/matan/ceph/src/seastar/include/seastar/core/shared_ptr.hh:347
    #1 0x56f6e7a in operator() /home/matan/ceph/src/crimson/osd/recovery_backend.cc:245
    #2 0x5286c62 in std::__n4861::coroutine_handle<crimson::internal::promise_base<crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>, void, void> >::resume() const /opt/rh/gcc-toolset-13/root/usr/include/c++/13/coroutine:24

SUMMARY: AddressSanitizer: stack-use-after-return /home/matan/ceph/src/seastar/include/seastar/core/shared_ptr.hh:347 in seastar::lw_shared_ptr<std::map<hobject_t, eversion_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, eversion_t> >

```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/crimson/test_crimson_coroutine: add test_coroutine_loops

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: _handle_pull_response to use obc_manager

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/.../replicate_recovery_backend: remove unnecessary check

Already checked in handle_recovery_op.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: scan_for_backfill to use obc_manager

let obc_loader get the relevant obc and avoid duplicating the
loading logic.

Fixes: https://tracker.ceph.com/issues/69154
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/recovery_backend: move scan_for_backfill to coroutine

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: Improve sso role mapping

Fixes https://tracker.ceph.com/issues/70366

- adds support for ISV
- using jmespath expression for fetching roles from payload
- added roles_path parameter in sso enable command as optional argument
- modified roles mapper for sso login

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60753 from aclamk/wip-aclamk-more-deferred-overwrite-fix

os/bluestore: Fix problem with deferred writes replay
This PR had gone through testing, including teuthology, but not in its final form.
But the issue its fixed is urgent.
This PR is merged before final teuthology run.

Merge pull request #62146 from Matan-B/wip-matanb-crimson-wip-69439-v2

crimson: Handle peer replies handling once received

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62118 from rhcs-dashboard/update-proto-files-dashboard

mgr/dashboard: update dashboard nvmeof proto files

Merge pull request #62176 from anthonyeleven/weofferedthereleasesorder

doc/releases: Add ordering comment to releases.yml

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61224 from CookiePLMonster/patch-22

dokan: fix a Y2038 bug by replacing `Int32x32To64` with multiplication

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

qa/workunits/rbd: wait for resize to be applied in rbd-nbd

Implement the same logic as in commit 6f3d0f570f1a ("test/librbd/fsx:
wait for resize to propagate in krbd_resize()").

Fixes: https://tracker.ceph.com/issues/66419
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61753 from aainscow/flat_maps

common: Introduce efficient map and set structure useful for EC.

Merge pull request #62074 from ronen-fr/wip-rf-more4-keys

mon,mgr,auth,client: replace obsolete get_tracked_conf_keys()

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62133 from ronen-fr/wip-rf-more6-keys

tests: replace obsolete get_tracked_conf_keys()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62173 from guits/fix-70363

ceph-volume: allow zapping partitions on multipath devices

Merge pull request #61669 from badone/wip-remove-openstack.yaml

qa: Remove openstack.yaml file which is no longer relevant

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Laura Flores <lflores@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

mgr/dashboard: update dashboard nvmeof proto files

Fixes: https://tracker.ceph.com/issues/70330
Signed-off-by: Nizamudeen A <nia@redhat.com>

doc/releases: Add ordering comment to releases.yml

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #61624 from VallariAg/wip-nvmeof-ceph-status

mon: Add nvmeof group/gateway name in  "ceph -s"

Merge pull request #62150 from ifed01/wip-ifed-fix-discard-op

blk/kerneldevice: count ops not bytes in discard_op perf counter

Reviewed-by: guyite@bytedance.com

qa/rgw: add test for sync policy user mode

Test policies created through PutBucketReplication API, whether they
respect the uid used in the policies.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

qa/rgw: add non-account (alt) users to multisite tests

Create a user and alt user without account to test PutBucketReplication
scenarios as account is not supported to call that API yet.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

ceph-volume: allow zapping partitions on multipath devices

ceph-volume refuses to zap a device if it is a partition on a multipath
device due to an overly strict condition. This change ensures that only
full mapper devices (excluding partitions) are blocked from being zapped,
allowing partitions on multipath devices to be processed correctly.

Fixes: https://tracker.ceph.com/issues/70363
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #61651 from anthonyeleven/improve-radosgw-layout

doc/radosgw: Improve layout.rst

Merge pull request #62158 from dang/wip-dang-68336

RGW - Temporarily disable List2Inotify1 in the posix bucket cache uni…

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

common: bitset_set and mini_flat_map

The bitset_set is compile-time-fixed size bitmap, which can be accessed using a std::set-like iterator.

The mini_flat_map is similar to a boost::flat_map, except with more restrictions on the size and key, allowing it to make
better use of vectors.

Both have the restriction that the key must be unambiguously castable to/from a small integer. The mini_flat_map has the additional restriction, that the size must be known at construction time.

Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>

Merge pull request #61933 from NitzanMordhai/wip-nitzan-logmonitor-forward-msg-noreply

LogMonitor: set no_reply for forward MLog commands

Merge pull request #61934 from NitzanMordhai/wip-nitzan-remove-tier-cache-tests-rados-suite

suites/rados: cache tier deprecated, no need to keep the tests for it

Merge pull request #61404 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_plugin_support

common/io_exerciser: ceph_test_rados_io_sequence plugin/profile support

Merge pull request #61348 from Naveenaidu/naveen/wip-tracker-65450

qa/tasks/thrashosds-health: whitelist PG_BACKFILL_FULL

crimson/os/seastore/rbm: add do_with() to prevent accessing unavailable reference

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

crimson/os/seastore/rbm: create a device file if it doesn't exist

Signed-off-by: Myoungwon Oh <ohmyoungwon@gmail.com>

qa/suite/crimson-rados: add seastore test with RBM

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

os/bluestore: implemented bluestore_blob_t::get_unused_mask

The function was used only by writer v2, but it was returning 0 always.
Now it properly returns the mask of used and unused blob regions.
Changed returned type unused_t (16 bits) -> uint64_t.

Made write_v2 path now properly mark unused.

+ new unittest bluestore_blob_t.get_unused_mask
+ fixed unittest ExtentMapFixture.rain

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

RGW - Temporarily disable List2Inotify1 in the posix bucket cache unit tests, until we can find why inotify fails on CI machines

Signed-off-by: Daniel Gryniewicz <dang@fprintf.net>

Merge pull request #61659 from Matan-B/wip-matanb-cmake-crimson-boost-asio

crimson/CMakeLists: cleanup BOOST_ASIO_DISABLE_CONCEPTS

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61569 from soumyakoduri/wip-skoduri-tier-config

rgw/cloudtier: Correct option ordering in RGWZoneGroupPlacementTier

Reviewed-by: J. Eric Ivancich (ivancich@redhat.com)

Merge PR #62108 into main

* refs/pull/62108/head:
cmake/cephfs: fix options to enable client and dependencies

Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge PR #62107 into main

* refs/pull/62107/head:
pybind/cephfs: use legacy noexcept for cdefs for cython 3.Y.Z
pybind/cephfs: increment ref before calling out to c++

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62028 from Matan-B/wip-matanb-crimson-mgrc-con

crimson/mgr/client: fix _send_report empty conn usage

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/pg_backend: rep_op_ret_t to not return acked_peers

Cleanup, no longer used.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: Handle peer replies handling once received

Instead of updating last complete on disk and calling complete_write
in PG::submit_transacion (chained to all_completed).

Move the completion handling earlier as soon as all the peers acked.
This essentially means that we move the handling to
ReplicatedBackend::get_rep_op_reply, unless replication size is 1 -
then we would mark completion in ReplicatedBackend::submit_transaction.

Fixes: https://tracker.ceph.com/issues/69439
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/replicated_backend: Introduce pending_shart_t::last_complete

This will be used (next commit) once we'll call complete_write from
ReplicatedBackend and not from PG as we do now.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: Introduce PG peering helpers

This will be used once we'll call complete_write from
ReplicatedBackend and not from PG as we do now.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>