Merge pull request #67602 from bluikko/wip-doc-radosgw-adminops-fixes

doc/radosgw: Fix markup in adminops.rst

Merge pull request #67595 from bluikko/wip-doc-remove-markup-from-cli-commands

doc: Remove markup from CLI example commands

Merge pull request #66578 from ceph/dvanders_mds_set_balanced_budget

mds: enable osdc op throttles

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #67444 from rhcs-dashboard/fix-osd-form-alert

mgr/dashboard: Misleading warning when no eligible devices are available for OSD creation

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #66799 from aainscow/missing-list-corruption-fix

osd: Change rmissing map key from version_t to eversion_t

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #67578 from ivancich/wip-install-java-17

rgw/tests: add os-specific java 1.7 install commands to keycloak task

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67523 from shraddhaag/wip-shraddhaag-improve-crimson-docs

doc: improve crimson docs

Merge PR #67608 into main

* refs/pull/67608/head:
Revert "Merge pull request #67167 from Hezko/nvme-module-two"

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #66111 from fa-at-pulsit/wip-73702-rgw-keystone-scope

rgw: Add Keystone scope information to ops logging

Merge pull request #67126 from gardran/wip-gardran-fix-mclock-options

osd: handle all the updated keys in OSD::maybe_override_options_for_qos

Reviewed-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>

Merge pull request #67414 from rhcs-dashboard/add-config-params-cephadm-certmgr

mgr/dashboard : Add cephadm config params to dashboard

Reviewed-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

doc: reformat crimson docs

This commit rearranges crimson docs so the deployment steps are inorder
to how they are supposed to be executed. Also, it removed `crimson-osd`
referneces as that is an internal detail that users don't need to be
aware of.

Revert "Merge pull request #67167 from Hezko/nvme-module-two"

This PR broke the ceph-mgr [1] and was not QA'd more broadly.

[1] https://github.com/ceph/ceph/pull/67167#issuecomment-3984274659

This reverts commit 884bbcfd38d88b08213c3106dc89cd6c956d4b95, reversing
changes made to 472e267f03705145ee306d15f3d566ec4ac3f726.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

mgr/dashboard: Misleading warning when no eligible devices are available for OSD creation

Fixes: https://tracker.ceph.com/issues/75050
Signed-off-by: Naman Munet <naman.munet@ibm.com>

Merge PR #67476 into main

* refs/pull/67476/head:

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #67554 from rkachach/fix_splitting_services_ut

mgr/cephadm: splitting services big UT file into smaller per-service files

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #66832 from nbalacha/wip-nbalacha-lua-74219

Wip rgw/lua: performance improvements

doc/radosgw: Fix markup in adminops.rst

Add blank lines so that markup is not rendered verbatim.

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #67480 from afreen23/subsystem-integration

mgr/dashboard: Allow adding authentication to subsystem flow

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #67158 from sujay-d07/automated-kafka-security-testing

qa: Extend Kafka SASL Security Tests

Merge pull request #67569 from gbregman/main

mgr/dashboard: Add DHCHAP controller key to NVME host commands

doc: Remove markup from CLI example commands

Remove all the remaining stars in the preformatted block CLI example
commands.
Several example CLI commands had stars around placeholder text in
addition to the standard angle brackets. The stars could make sense in
man pages or in text to make the placeholder italic but it will just be
rendered as-is in preformatted text blocks.

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #67591 from bluikko/wip-doc-sphix-warnings-202603

doc: Fix Sphinx warnings

mgr/dashboard: Fix auth logic in subsystem and remove traffic encryption columns

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #65877 from edwinzrodriguez/ceph-wip-73421

mds: Avoid use of moved 'fs' in FSMap

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #65879 from edwinzrodriguez/ceph-wip-73425

mds: MDSCap Use default impl for copy and assign op

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #65883 from edwinzrodriguez/ceph-wip-73429

mds: Refactor Mantle::balance to use std::string

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #66288 from chrisphoffman/wip-73825

client: Only use known fields when comparing policies

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: Fix Sphinx warnings

Add missing blank lines in two files to fix Sphinx warnings.

Signed-off-by: Ville Ojamo <git2233+ceph@ojamo.eu>

Merge pull request #67517 from rhcs-dashboard/backup-storage-consumption

mgr/dashboard: add storage consumption card

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67276 from rhcs-dashboard/74636-service-modal-with-certmgr

mgr/dashboard : Update create/edit service modal to support certmg

Reviewed-by: Abhishek Desai <abhishek.desai1@ibm.com>

mgr/dashboard: add storage consumption card

Fixes: https://tracker.ceph.com/issues/75181
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

rgw/tests: add os-specific java 1.7 install commands to keycloak task

Add commands to keycloak task specific for rocky, rhel, centos, and
ubuntu. Also, clean-up installed package(s) after test is run.

This is necessary as rocky can't install the same package(s) that the
other os types currently can.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/dashboard: Add DHCHAP controller key to NVME host commands
Fixes: https://tracker.ceph.com/issues/74939
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

mgr/dashboard: Allow adding authentication to subsystem flow

- integrates third step with overall tearsheet

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67535 from rhcs-dashboard/final-merged-branch

mgr/dashboard: Final merged branch for subsystem complete flow

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #64718 from VallariAg/wip-nvmeof-top-tool

mgr/dashboard: Add nvmeof_top_cli service

Merge pull request #67345 from rhcs-dashboard/nfs-pyobj-serialze-err

mgr/dashboard: nfs export creation fails with obj deserialization

Reviewed-by: Abhishek Desai <abhishek.desai1@ibm.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>

Merge pull request #67095 from cbodley/wip-74591

qa/radosgw_admin: replace boto2 with boto3

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #66879 from Hezko/nvme-cli-responses

mgr/dashboard: introduce NvmeofCLICommand's success_message_template …

Merge pull request #67167 from Hezko/nvme-module-two

mgr/nvmeof: add nvmeof module which will create .nvmeof rbd pool

Merge pull request #67524 from idryomov/wip-75161

librbd: don't complete ImageUpdateWatchers::shut_down() prematurely

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #67482 from idryomov/wip-68279

librbd/mirror: detect trashed snapshots in UnlinkPeerRequest

Reviewed-by: Ramana Raja <rraja@redhat.com>

mgr/cephadm: spliting test_services UT into per-service UT files

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

mgr/cephadm: removed big test_services file

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #66262 from ceph/updates_for_lf_csc_charter

CONTRIBUTING: link to Governance doc

Merge pull request #67486 from cbodley/wip-qa-rgw-multisite-boto2-decoder

qa/rgw/multisite: remove boto2 BotoJSONEncoder

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #67169 from kchheda3/wip-fix-notification-queue-full

rgw/notification: Prevent reserved_size leak by decrementing overhead on commit/abort.

 reviewed-by: yuvalif

Merge pull request #67083 from smanjara/wip-fix-multisite-sigv4

rgw/multisite: fix forwarded requests using awsv4 with UNSIGNED-PAYLOAD

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Remove tabs under subsystem
Fixes: https://tracker.ceph.com/issues/74904
Signed-off-by: pujaoshahu <pshahu@redhat.com>

rgw: Add Keystone scope information to ops logging

Fixes: https://tracker.ceph.com/issues/73702
On-behalf-of: SAP <filipp.akinfiev@clyso.com>
Signed-off-by: Filipp Akinfiev <filipp.akinfiev@clyso.com>

src/pybind/mgr: Add nvmeof-top tool

Add src/pybind/mgr/dashboard/services/nvmeof_top_cli.py
which adds commands:
"ceph nvmeof top io <subsystem>"
"ceph nvmeof top cpu"

This file is moved from the original work in:
https://github.com/pcuzner/ceph-nvmeof-top

Fixes: https://tracker.ceph.com/issues/64578
Co-authored-by: Paul Cuzner <pcuzner@ibm.com>
Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

mgr/dashboard: introduce NvmeofCLICommand's success_message_template and success_message_map parameters to allow meaningful success messages

Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/dashboard: add-reviewstep-in-subsystem

Fixes: https://tracker.ceph.com/issues/75085
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>>

mgr/dashboard: NVme- Add namespace in subsystem resource page

Fixes: https://tracker.ceph.com/issues/74338
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>

mgr/dashboard: NVmeof Add listeners in subsytem and resource page.

Fixes: https://tracker.ceph.com/issues/74339
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>

mgr/dashboard: Nvmeof add hosts(Add Initiator) in subsystem details

Fixes: https://tracker.ceph.com/issues/75006
Signed-off-by: Sagar Gopale <sagar.gopale@ibm.com>

Merge pull request #67472 from rhcs-dashboard/removetab-subsystem

mgr/dashboard: Remove tabs under subsystem

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67503 from afreen23/data-resiliency

mgr/dashboard: Fixes for overview page

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Devika Babrekar <devika.babrekar@ibm.com>

mgr/nvmeof: add nvmeof module introduction to pending release notes

Fixes: https://tracker.ceph.com/issues/74702
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/nvmeof: add unittests

Fixes: https://tracker.ceph.com/issues/74702
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

Merge pull request #67123 from kshtsk/wip-qa-tasks-keystone-rocky

qa/tasks/keystone: restart mariadb for rocky and alma linux too

Merge pull request #67088 from rzarzynski/wip-crimson-ctwrn-unnused

crimson/os: fix multiple compile-time warnings on unused variables

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/nvmeof: use nvmeof module during orch nvmeof apply

Added a call to create_pool_if_not_exists during the execution of ceph orch apply nvmeof command.

Fixes: https://tracker.ceph.com/issues/74702
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/nvmeof: set nvmeof module to be enabled by default

Fixes: https://tracker.ceph.com/issues/74702
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/nvmeof: intergrate module into build and debian pkg

Fixes: https://tracker.ceph.com/issues/74702
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

mgr/nvmeof: introduce the new nvmeof module

Introduce a new NVMe-oF mgr module and which create the pool
used for storing NVMe-related metadata ceph orch nvmeof apply command.
This removes the need for users to manually create and configure the
metadata pool before using the NVMe-oF functionality, simplifying
setup and reducing the chance of misconfiguration.

Fixes: https://tracker.ceph.com/issues/74702
Signed-off-by: Tomer Haskalovitch <tomer.haska@ibm.com>

Merge pull request #66802 from cloudbehl/Subvolume-dashboard

monitoring: Add subvolume dashboard

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard : Add cephadm config params to dashboard
fixes : https://tracker.ceph.com/issues/75109
Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>

Merge pull request #67385 from kotreshhr/fix-mirror-qa-blocklist

qa: Fix test_cephfs_mirror_restart_sync_on_blocklist

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: nfs export creation fails with obj deserialization

with the rocky10 patches, the nfs export creation through dashboard
breaks when it tries to deserialize the obj returned by
`fetch_nfs_export_obj` interface. So removing that and use export apply
to create the nfs exports

Fixes: https://tracker.ceph.com/issues/74930
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Add filtering of alerts via route

- when user click on alerts of particular severity only those alerts should be shown

Fixes https://tracker.ceph.com/issues/75158

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Add skeleton states for alerts card

Fixes https://tracker.ceph.com/issues/75159

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Fix css in alerts card

- fixes alignment of icons

Fixes https://tracker.ceph.com/issues/75160

Signed-off-by: Afreen Misbah <afreen@ibm.com>

mgr/dashboard: Fix breaking layout in overview page

Issue:

One opening the notification panel overview page breaks.

Reason:

Due to the css grid used in notification panel and flex grid in overview page.

Fix:

Removed css grid from the notification header

Signed-off-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #67522 from rhcs-dashboard/notification-banner-simple-mode

mgr/dashboard : update telemetry notification for simple mode

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #66069 from ceph/r10-containerfile

Containerfile: Support Rocky 10 as base image

doc/CONTRIBUTING: link to Governance rules

The Ceph Project charter links to this file where we are to define
detailed project operations and processes. Link to our our existing
Governance rules to connect the dots here.

Signed-off-by: Dan van der Ster <dan.vanderster@clyso.com>
Signed-off-by: Josh Durgin <jdurgin@ibm.com>
Signed-off-by: Neha Ojha <nojha@ibm.com>

Merge pull request #67525 from idryomov/wip-rbd-make-check-filter

test: disable known flaky tests in run-rbd-unit-tests

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/tasks: add Kafka SSL and SASL security testing support

Add comprehensive SSL and SASL authentication support for Kafka bucket
notification tests, including SCRAM-SHA-256/512 mechanisms and multiple
listener protocols.

Signed-off-by: sujay-d07 <sujaydongre07@gmail.com>

rgw/bucket_notification: enhance Kafka security testing

Add support for SCRAM-SHA-512, dynamic hostname/IP configuration for SSL
certificates, and improved SASL mechanism handling in Kafka security tests.

Changes:
- kafka-security.sh: Support KAFKA_CERT_HOSTNAME and KAFKA_CERT_IP environment
  variables for dynamic certificate generation with proper SAN fields
- test_bn.py: Add SCRAM user setup, support SASL_SSL/SASL_PLAINTEXT protocols,
  add SCRAM-SHA-512 test cases
- README.rst: Update with step-by-step Kafka configuration instructions
- requirements.txt: Replace nose-py3 with pynose, add requests>=2.28.0

Signed-off-by: sujay-d07 <sujaydongre07@gmail.com>

Merge pull request #67500 from batrick/ceph-technical-charter

doc: add the Ceph Technical Charter

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #67234 from ljflores/wip-improve-qa-summary-script

script: improve qa-summary.sh script

qa/rgw/multisite: remove unused assert_raises() and tools.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw/multisite: remove boto2 BotoJSONEncoder

removes leftover boto2 stuff to resolve `run-tox-qa` error:

> tasks/rgw_multi/tools.py:52: error: Name "boto.s3.user.User" is not defined [name-defined]

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw/multisite: move append_query_arg() to zone_es.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #66057 from mheler/rgw-restore-wait-timeout

rgw/restore: add configurable wait timeout for cloud restore read-through

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

mgr/dashboard : Update create/edit service modal to support certmgr
fixes: https://tracker.ceph.com/issues/74636
Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>

mgr/dashboard: Fix unit tests

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

mgr/dashboard: Added capacity metrics

Signed-off-by: Ankush Behl <cloudbehl@gmail.com>

monitoring: Add subvolume dashboard

- Add IO, Throughput and Latency

Fixes: https://tracker.ceph.com/issues/74323
Signed-off-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #67419 from adk3798/cephadm-apply-mon-warning

qa/cephadm: ensure host has been fully saved before considering bootstrap complete

Reviewed-by: phlogistonjohn

Merge pull request #67236 from adk3798/cephadm-pickle-osd-class

mgr/cephadm: add __getstate__ so OSD class can be pickled

reviewed-by: batrick

mgr/dashboard: Remove tabs under subsystem
Fixes: https://tracker.ceph.com/issues/74904
Signed-off-by: pujaoshahu <pshahu@redhat.com>

Merge pull request #67477 from vshankar/wip-skip-mirroring-failure

qa: temporarily disable some mirroring test (see: https://tracker.ceph.com/issues/74984)

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

test: disable known flaky tests in run-rbd-unit-tests

The failures seem to be more frequent on newer hardware.  In the
absence of immediate fixes, disable a few tests that have been known to
be flaky for a long time to avoid disrupting "make check" runs.

Fixes: https://tracker.ceph.com/issues/75163
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard : update telemetry notification for simple mode
fixes : https://tracker.ceph.com/issues/75157
Signed-off-by: Abhishek Desai <abhishek.desai1@ibm.com>

Merge pull request #66572 from kotreshhr/mirror-multithreaded

tools/cephfs_mirror: Multi-threaded Mirroring

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #67509 from bill-scales/peeringstate2

test: Fix unittest_peeringstate as fix for 74218 has merged

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>

librbd: don't complete ImageUpdateWatchers::shut_down() prematurely

ImageUpdateWatchers::flush() requests aren't tracked with
m_in_flight-like mechanism the way ImageUpdateWatchers::send_notify()
requests are, but in both cases callbacks that represent delayed work
that is very likely to (indirectly) reference ImageCtx are involved.
When the image is getting closed, ImageUpdateWatchers::shut_down() is
called before anything that belongs to ImageCtx is destroyed.  However,
the shutdown can complete prematurely in the face of a pending flush if
one gets sent shortly before CloseRequest is invoked.  The callback for
that flush will then race with CloseRequest and may execute after parts
of or even the entire ImageCtx is destroyed, leading to use-after-free
and various segfaults.

Fixes: https://tracker.ceph.com/issues/75161
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>