mgr/dashboard: add an option to control the dashboard crypto caller

Add a mgr config option `crypto_caller` that lets a ceph user override
the default behavior of using the remote crypto caller. Supported
values are `internal` and `remote`.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: always use the internal cryptocaller

The cephadm modules needs to use python cryptography module for ssh (via
asyncssh) and thus there's no need to use the remote crypto caller in
cephadm. Configure cephadm to always use the internal cryptocaller.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: catch all failures to read cert

Previously, the internal crypto caller would catch (and convert) some
errors when reading the cert but not all cases. Move the logic to catch
the errors to a common location and do it once consistently.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: create module for selecting crypto caller

Add a module to select a desired crypto caller. Update the callers
to use the crypto caller interface.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: move internal crypto caller to new file

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: add caller module for base class

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: unify and organize all endpoint functions

Lightly reorganize and make the "endpoint" functions in cryptotools.py more
consistent and uniform. Use small functions for input and output
handling so that the handling is done the same way throughout. Pass a
pre-constructed crypto caller via the args to then endpoint functions.
Make generating the private key it's own named function rather than
one single (and only) function with overloaded behavior controlled by
a cli switch.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: use a main function

Use a main function to encapsulate the cli parsing rather than a block
of code in module scope.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: move actual crypto opts into a class

The functions now handle the i/o but allow the crypto function class
to centralize the functions that actually use the crypto libs.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr: fix test case in test_tls.py

Why violate the typing in a test? mypy never noticed this because tests
are not type checked but there seems to be no need to turn a str into
bytes to pass to a function that is typed only as taking str!

Signed-off-by: John Mulligan <jmulligan@redhat.com>

mgr/dashboard: replace direct use of bcrypt in dashboard

Replace a direct usage of bycrypt with our cryptocaller wrapper.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: give the parsers more sensible names

Name the parser objects after their functions and not `foo` and `bar`.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr: Appropriately rename function.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

python-common/cryptotools: Remove ascii and utf-8 references from encode/decode.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

python-common/cryptotools: fix error path in verify tls function

The remote verify_tls function was not raising errors when it should.
Fix the function so that it always returns an object when it succeeds or
fails gracefully. Always parse that function in the crypto caller class.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr: Correct code to ensure cephadm/tests/test_certmgr.py passes.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

python-common/cryptotools: Always encode, Err via stderr and signal the exit.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

python-common: Correct typo in private_key naming field.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

pybind/mgr: update mgr_util to use cryptotools CryptoCaller class

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common: remove unused dir

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: use one single dir for cryptotools

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: create CrytpoCaller interface class

Create a class to act as a common shim between the cryptotools external
functions and the mgr. It provides common conversion mechanisms and
could possibly act as an abstraction in case we decide to make
the external function calls in different ways in the future.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

python-common/cryptotools: use json for structured output

Where possible try to use structured output in JSON for easier parsing
and interaction with the parent process.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

pybind/mgr: Hack around the 'ImportError: PyO3 modules may only be initialized once per interpreter process' issue.

Fixes: https://tracker.ceph.com/issues/64213
Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

Merge pull request #64275 from xxhdx1985126/wip-crimson-backedn-namespace

crimson/osd: put PGBackend/RecoveryBackend/PGRecovery and other related classes into the crimson::osd  namespace

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #64262 from Matan-B/wip-matanb-crimson-abort-msg

crimson: switch to ceph_abort_msg

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #64246 from tchaikov/wip-cmake-drop-with_seastar

cmake: replace WITH_SEASTAR with WITH_CRIMSON

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #63817 from mohit84/objectstore_type

crimson: Use osd_objectstore instead of using crimson_osd_objectstore

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #64067 from VallariAg/wip-nvmeof-keepalive-alert

monitoring: Add alert NVMeoFHostKeepAliveTimeout

Merge pull request #62170 from connorfawcett/wip-online-consistency-checker-0703

erasure-code: Add a new utility which checks the consistency of objects within an EC pool

Merge pull request #64110 from yuvalif/wip-yuval-71402

rgw/notifications: add http request timeout and max inflight

Merge pull request #64328 from yuvalif/wip-yuval-71945

doc/rgw/logging: fix journal record format

Merge pull request #63144 from liu-chunmei/omap_noexist

crimson/osd/pg_backend: needn't check if os.exist

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #64061 from rhcs-dashboard/local-storage-class

mgr/dashboard: Local Storage Class - create and list

Merge pull request #63378 from Matan-B/wip-matanb-readme-build-types

README: add Build Types section

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc/README: add Build Types section

Originated from: https://tracker.ceph.com/issues/71360

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Adds a new command-line utility which can check the consistency of objects within an erasure coded pool.
A new test-only inject tells the EC backend to return both data and parity shards to the client so that they can
be checked for consistency by the new tool.

Supports both optimized and unoptimized EC pools.

Signed-off-by: Connor Fawcett <connorfa@uk.ibm.com>

mgr/dashboard: Local Storage Class - create and list
Fixes: https://tracker.ceph.com/issues/71460
Signed-off-by: Dnyaneshwari Talwekar <dtalwekar@redhat.com>

Merge pull request #64252 from xxhdx1985126/wip-seastore-lba-bottom-up-search

crimson/os/seastore/btree: bottom-up search capability

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #64318 from xxhdx1985126/wip-seastore-omap-leaf-node-rewrite

crimson/os/seastore/omap_manager: fix OMapLeafNode rewrite type error

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #64109 from xxhdx1985126/wip-71794

crimson/os/seastore/lba/btree_lba_manager: never downcast reserved ptrs

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #63779 from zdover23/wip-doc-2025-06-06-mgr-telemetry-63735-followup

doc/mgr: edit telemetry.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61989 from rhcs-dashboard/edit-topic

mgr/dashboard: Add RGW CRUD topic in dashboard

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Naman Munet <nmunet@redhat.com>

Merge pull request #61286 from Jayaprakash-ibm/discard_queue_overutilized

blk:Warning added for discard queue overflow

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #44283 from mkogan1/wip-rgw-radosgw-admin-cert

rgw: allow specifying ssl certificate for radosgw-admin operations

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62949 from joscollin/wip-test-mirroring-fix

qa: setup mount_b in test_mirroring.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #64319 from zdover23/wip-doc-2025-07-03-tr58092-radosgw-config-ref

doc/radosgw: add rgw_enable_lc_threads & rgw_enable_gc_threads

Reviewed-by: Casey Bodley <cbodely@redhat.com>

Merge pull request #62842 from joscollin/wip-B68001-B68446

qa: fix multi-fs tests in test_mds_metrics.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>

doc/rgw/logging: fix journal record format

Fixes: https://tracker.ceph.com/issues/71945
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #64245 from leonidc/delete_gw_fix

nvmeofgw:

Minor cleanup to ensure make check passes

Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>

mon: Integrate discard queue overflow into pg health warnings

Added a health warning mechanism to monitor the discard queue for potential overload
Emits a warning if the accumulated discarded bytes in the queue exceed the configured threshold
Introduced a debugging tool to simulate slow discard operations by adding a configurable delay

Fixes : https://tracker.ceph.com/issues/69082

Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>

common/options: Added bdev_discard_max_bytes and bdev_debug_discard_sleep options

Added a health warning mechanism to monitor the discard queue for potential overload
Emits a warning if the accumulated discarded bytes in the queue exceed the configured threshold
Introduced a debugging tool to simulate slow discard operations by adding a configurable delay

Fixes : https://tracker.ceph.com/issues/69082

Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>

blk:Warning added for discard queue overflow

Added a health warning mechanism to monitor the discard queue for potential overload
Emits a warning if the accumulated discarded bytes in the queue exceed the configured threshold
Introduced a debugging tool to simulate slow discard operations by adding a configurable delay

Fixes : https://tracker.ceph.com/issues/69082

Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>

Merge pull request #64306 from zdover23/wip-doc-2025-07-02-tr23594-rados-ops-user-management

doc/rados/ops: add caps restore command

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: add rgw_enable_lc_threads & rgw_enable_gc_threads

Add rgw_enable_lc_threads and rgw_enable_gc_threads to
doc/radosgw/config-ref.rst.

Fixes: https://tracker.ceph.com/issues/58092
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/omap_manager: fix OMapLeafNode rewrite type error

Fixes: https://tracker.ceph.com/issues/71939
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #62943 from rhcs-dashboard/tieiring-allow-read

mgr/dashboard: Tieiring - allow read through

Merge pull request #64238 from tchaikov/wip-test-ec-fix-stack-use-after-scope

test/erasure-code: fix stack-use-after-scope by replacing initializer_list with array

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #64250 from tchaikov/wip-cmake-boost-1.88

cmake/modules/FindBoost: add support for Boost 1.88.0

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #64228 from phlogistonjohn/jjm-lab-extras

install-deps.sh: enable lab extras on el10 builders

Merge pull request #63359 from zdover23/wip-doc-2025-05-20-restore-63351-document_HTTP_X_AMZ_DELETE_IF_UNMODIFIED_SINCE

doc/radosgw/s3: Document delete-if-unmodified-since

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw/s3: Document delete-if-unmodified-since

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 3381041d190fd48cf9f6d53f679f1737028cccf4)

doc/rados/ops: add caps restore command

Add a command that restores caps to client.admin after they have been
accidentally destroyed or removed.

Fixes: https://tracker.ceph.com/issues/23594
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #64296 from Naveenaidu/wip-naveen-introduce-fix-config-ok-command-logic

.github/workflows/scripts/config-diff-post-comment.js: fix config check ok logic

crimson/os/seastore/btree: only compile debug-purposed methods in debug
builds

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/fixed_kv_btree: remove unused constructors

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

mgr/dashboard: Add RGW CRUD topic in dashboard

Fixes: https://tracker.ceph.com/issues/699654
Signed-off-by: pujaoshahu <pshahu@redhat.com>

crimson: switch to ceph_abort_msg

ceph_abort doesn't print a message. Use ceph_abort_msg instead.
Most of the instances are not printing useful information but some are:
  ceph_abort_msg("seastore device size setting is too small");

```
  ::ceph::__ceph_abort( __FILE__, __LINE__, __CEPH_ASSERT_FUNCTION, "abort() called")

  ::ceph::__ceph_abort( __FILE__, __LINE__, __CEPH_ASSERT_FUNCTION, msg)
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #64215 from Hezko/nvmeof-cli-migrate-prs

mgr/dashboard: NVMeoF CLI migrate prs

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

nvmeofgw: fixing GW delete issues
1.fixing the issue when gw is deleted based on invalid subsystem info
2. in function track_deleting_gws: break from loop only if
  delete was really done
        3. fix published rebalance index - publish ana-group instead of
  index
        4. do not dump gw-id string after gw was removed

Fixes: https://tracker.ceph.com/issues/71896
Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

Merge pull request #64260 from tchaikov/wip-osdc-iwyu

osdc: remove unused rados.h include from error_code.h

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #64087 from NitzanMordhai/wip-nitzan-aio-pool-eio-flag-mutex-left

test/librados/aio_cxx: Fix potential mutex destruction failure on Pool EIOFlag

.github/workflows/scripts/config-diff-post-comment.js: fix config check ok logic

currently, whenever a "config diff tool output" comment is created it
also has the string `/config check ok` string in it. The next time the
test run it see's this text and assumes that the user has commented it.
We fix the logic to makes sure that we ignore such cases.

Signed-off-by: Naveen Naidu <naveennaidu479@gmail.com>

Merge pull request #62239 from rhcs-dashboard/xmlsec-fail-main

mgr/dashboard: use system packages when running tox

Merge pull request #63889 from shreya-subramanian/perf_counters_clean

osd: Added perf counters for full and incremental maps

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #61535 from kinazarov1999/import-stdin-progress

rbd: add --estimated-size option for import from stdin

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61468 from Nordix/fix-69610-sunnat

mgr/prometheus: Make prometheus TLS config work with Rook orchestrator

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #64255 from bluikko/doc-config-ceph-conf-improvements-rados

doc/rados/configuration: Small improvements in ceph-conf.rst

mgr/dashboard: migrate nvmeof pr #1346 to new cli

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/dashboard: migrate nvmeof pr #1335 to new cli

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/dashboard: migrate nvmeof pr #1327 to new cli

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/dashboard: migrate nvmeof pr #1233 to new cli

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/dashboard: migrate nvmeof pr #1277 to new cli

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/dashboard: migrate nvmeof pr #1308 to new cli

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/dashboard: migrate nvmeof pr #1238 to new cli

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

crimson/osd: put PGBackend/RecoveryBackend/PGRecovery and other related
classes into the crimson::osd  namespace

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

mgr/dashboard: catch protobuf error due to mismatch in version

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: unpin grpcio deps

Fixes: https://tracker.ceph.com/issues/70411
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: sitepackages only for py3

Signed-off-by: Nizamudeen A <nia@redhat.com>

Revert "mgr/dashboard: pin lxml to fix run-dashboard-tox-make-check failure"

This reverts commit 1f84505f1232dd8445df1a2a819fa000062d3934.

Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: disable saml2 tests when the dep is not there

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #64239 from tchaikov/wip-doc-mgr-crash

doc/mgr/crash.rst: remove outdated module enabling instructions

Reviewed-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/FixedKVBtree: return the iterator when removing mapping

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/FixedKVBtree: introduce partial iterator

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: fix indention of handle_merge

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: make "check_split" asynchronous

This is a preparation for btree's make_partial_iter

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/linked_tree_node: add get_parent_node for retrieving
accessible parents

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/cache: add maybe_wait_accessible() for lba bottom-up
search

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #64065 from tchaikov/wip-test-mon-election-fix-leaks

test/mon/test_election: fix memory leaks of Owner and ConnectionTracker

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>