mgr/dashboard: remove orch required decorator from host UI router (list)

https://tracker.ceph.com/issues/68101

Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #59727 from ivoalmeida/snap-schedule-start-time-fix

mgr/dashboard: fix start time format

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59805 from afreen23/wip-nvme-mtls

mgr/dashboard: Nvme mTLS support and  service name changes

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59781 from ivoalmeida/table-filter-fix

mgr/dashboard: fix table filter

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59668 from rhcs-dashboard/fix-modal-hidden-in-nav

mgr/dashboard: fix UI modal issues

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: Add mTLS support

- enables mTLS support from dashboard
- adds unit tests related to mTLS support
- can enable mTLS
- can disable mTLS
- inlcuded refactoring from prev commit

Fixes https://tracker.ceph.com/issues/66416

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: Adding group and pool name to service name

- Pre-populating the service name field with the format `nvmeof.<pool_name>.<group_name>`.
- This can be changed by user but by default this value will be there.
- This will help user to quickly fill form and proceed hence improving usability.
- cephadm also uses this format as of now this convention so it will make UI aligned with CLI experience

- updates unit tests to improve coverage

- hides`count` values as that is not needed for 'nvmeof' only hosts and labels required.

Fixes https://tracker.ceph.com/issues/68065

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #58618 from rhcs-dashboard/mgmt-gateway-ui

mgr/dashboard: add service management for mgmt-gateway

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #58456 from rhcs-dashboard/auth2-sso

mgr/dashboard: Add SSO through oauth2 protocol

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #59752 from guits/drive-group-spec-bool-args

orch: refactor boolean handling in drive group spec

Reviewed-by: Adam King <adking@redhat.com>

mgr/dashboard: fix UI modal issues

- Fix bs-modal getting hidden behind navbar
- Reuse the project name for tab title
- Readonly user shows an empty space in the administration dropdown
- Physical Disks Identify modal not opening up
   - add e2e in cephadm to make sure the modal always comes up
- Primary action doesn't need to be displayed for the read-only user

Fixes: https://tracker.ceph.com/issues/67547
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59766 from rkachach/fix_issue_68052

mgr/cephadm: renaming whitelist_domains field to allowlist_domains

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #59557 from afreen23/wip-ec-profile

mgr/dashboard: mgr/dashboard: Select no device by default in EC profile

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #45807 from mchangir/doc-document-random-selection-teuthology-operator

doc/dev/developer_guide/testing_integration_tests: Document the random selection operator

mgr/dashboard: add SSO through oauth2 protocol

Fixes: https://tracker.ceph.com/issues/66900
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

mgr/dashboard: add service management for mgmt-gateway

Fixes: https://tracker.ceph.com/issues/66963
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #57459 from mchangir/mds-snap_schedule-relocate-test-and-overrides-to-correct-location

qa: relocate subvol creation overrides and test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #59379 from rhcs-dashboard/oauth2-proxy-ui

mgr/dashboard: add service management for oauth2-proxy

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #59061 from rhcs-dashboard/rgw-multisite-sync-policy-edit-zones-fix

mgr/dashboard: RGW multisite sync remove zones fix

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: add service management for oauth2-proxy

Fixes: https://tracker.ceph.com/issues/67651
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #59797 from zdover23/wip-doc-2024-09-15-install-install-storage-cluster

doc/install: add prompts to install-storage-cluster

Merge pull request #59801 from zdover23/wip-doc-2024-09-15-rados-ops-health-checks

doc/rados: add osd_deep_scrub_interval setting operation

Merge pull request #59755 from baum/nvmeofspec_addr_zero

pybind/mgr/cephadm/services/nvmeof.py: allow setting '0.0.0.0' as add…

doc/rados: add osd_deep_scrub_interval setting operation

Add a second method of changing the value of osd_deep_scrub_interval to
remedy the condition indicated by the "PGs not deep-scrubbed in time"
warning.

This procedure was developed by Eugen Block, and is at the time of this
commit available on his blog at
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/install: add prompts to install-storage-cluster

Add prompts to doc/install/install-storage-cluster.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59636 from ronen-fr/wip-rf-smaller-deep-chunk

osd/scrub: decrease default deep scrub chunk size

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #59732 from zdover23/wip-doc-2024-09-11-README-md-formatting

doc/README.md: improve formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54777 from cbodley/wip-rgw-meta-handlers

rgw: remove unused backend abstraction for metadata handlers

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

mgr/dashboard: fix table filter

Fixes: https://tracker.ceph.com/issues/68063
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

orch: refactor boolean handling in drive group spec

The intent of 42721c03ee6f was to address an issue where boolean
parameters weren't handled correctly.

I noticed that a parameter (`tpm2`) was missed, which made me realize
that maintaining a list of these boolean parameters is necessary.

To simplify things, we should only accept `"true"` or `"false"` (in any case),
allowing us to avoid the need to maintain a list of boolean parameters.

This change introduces a `list_drive_group_spec_bool_arg` to store boolean
arguments related to drive group specifications, simplifying the validation
process for boolean values by directly checking if the values are 'true' or 'false'.

Fixes: https://tracker.ceph.com/issues/68045
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/dashboard: RGW multisite sync remove zones fix

Fixes: https://tracker.ceph.com/issues/66945
Signed-off-by: Naman Munet <nmunet@redhat.com>

mgr/dashboard: fix start time format

Fixes: https://tracker.ceph.com/issues/68024
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

rgw/service: remove unused metadata and metadata backend services

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove unused RGWSI_Role_RADOS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: add rgwrados::role interface for role metadata

move everything from class RadosRole into driver/rados/role.cc and use
RGWSI_SysObj instead of the metadata backend. narrows the RGWRole
interface by handling the name/path objects internally

rgwrados::role::write() adds/remove name and path linkages in case they
change. admin ops don't allow this, but 'metadata put' could upload
arbitrary json that does change them

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for role metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: encapsulate role metadata handler in rgw_role.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for user metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: remove unused 'meta backend ctx' arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: call sysobj directly instead of 'sysobj meta backend'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for bucket instances

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for bucket entrypoints

RGWBucketMetadataHandler inherits from RGWMetadataHandler directly
instead of RGWMetadataHandler_GenericMetaBE

replace RGWSI_Bucket_SObj_Module with the RGWMetadataLister abstraction

RGWSI_Bucket_SObj now depends on RGWSI_MDLog and writes mdlog entries
itself

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: remove unused 'meta backend ctx' arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: call sysobj directly instead of 'sysobj meta backend'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: free functions for bucket instance metadata keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove otp service and metadata backend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: remove backend stuff from otp metadata handler

inherit from RGWMetadataHandler instead of
RGWMetadataHandler_GenericMetaBE to avoid all of the metadata backend
abstractions, and just call into the cls and sysobj services directly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove unused RGWOTPCtl

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clear output keys in RGWMetadataLister::get_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/posix: add missing include for cpp_strerror()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd/scrub: decrease default deep scrub chunk size

The previous default of 25 objects per chunk proved to take too long
(many hundreds of milliseconds) on a busy cluster. As the scrubber
locks all objects in the chunk for the duration, a large chunk size
can cause a significant impact on the client ops' latencies.

Fixes: https://tracker.ceph.com/issues/68057
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #59696 from wanglinke521/wlk-perfect-comment

os/bluestore: perfect comments in hybrid_allocator

Reviewed-by: Adam Kupczyk <kupczyk@ibm.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #59438 from mkogan1/wip-d3n-ex-co-ar

doc/rgw/d3n: pass cache dir volume to extra_container_args

mgr/cephadm: renaming whitelist_domains field to allowlist_domains

Fixes: https://tracker.ceph.com/issues/68052
Signed-off-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #59634 from rkachach/fix_issue_67934

mgr/cephadm: adding spec fields for oauth2-proxy whitelist_domains

Reviewed-by: Adam King <adking@redhat.com>

python-common/ceph/deployment/service_spec.py: Allow the cephadm deployment to determine the default addresses

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

Merge pull request #59456 from xxhdx1985126/wip-67731

crimson/osd/pg: clear backfill_state when the PG goes clean

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59652 from xxhdx1985126/wip-67945

crimson/osd/osd_operations/snaptrim_event: only increase osd_osd_param_t::at_version after clone adjustments

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59564 from Matan-B/wip-matanb-crimson-osd-release

crimson/osd: write require_osd_release only when needed

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #59506 from Matan-B/wip-matanb-crimson-seastore-logs

qa/config/seastore: enable all subsystem debug

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59753 from wodeshijie33941/main

doc/install: Keep the name field of the created user consistent with …

Reviewed-by: Zac Dover <zac.dover@proton.me>

pybind/mgr/cephadm/services/nvmeof.py: allow setting '0.0.0.0' as address in the spec file

- Partial revert of https://github.com/ceph/ceph/pull/59738/commits/9eb3b99b99b0b8c6c4e5f75e59c2e4835a9d6796
- Part of https://github.com/ceph/ceph/pull/59738

Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>

Merge pull request #59741 from yuvalif/wip-yuval-68033

rgw/kafka: refactor topic creation to avoid rd_kafka_topic_name()

Reviewed-By: Casey Bodley <cbodley@ibm.com>

doc/install: Keep the name field of the created user consistent with the node name in the Start RADOSGW service command

If the user name does not match the name of the node that started the RADOSGW service, this will cause confusion for those who are new to ceph. Because they can't start the radosgw service normally as shown in the tutorial.

Signed-off-by: hejindong <wodeshijie3394@126.com>

Merge pull request #59482 from rhcs-dashboard/nvmeof-api-group-support

mgr/dashboard: add gw_groups support to nvmeof api

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: caroav <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #59576 from rhcs-dashboard/hosts-selection-show-only-10-entries-fix

mgr/dashboard: service form hosts selection only show up to 10 entries

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59738 from gbregman/main

Make the discovery and gateway IPs configurable in NVMEof configuration

Merge pull request #59681 from adk3798/cephadm-enable-module-bootstrap-retry

cephadm: bootstrap: verify orch module is running before setting cephadm backend

Reviewed-by: John Mulligan <jmulligan@redhat.com>

rgw/kafka: refactor topic creation to avoid rd_kafka_topic_name()

also, use unique_ptr for safer memory management.

Fixes: https://tracker.ceph.com/issues/68033
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

mgr/cephadm: Make the discovery and gateway IPs configurable in NVMEof configuration
Fixes https://tracker.ceph.com/issues/68030

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: Make the discovery and gateway IPs configurable in NVMEof configuration
Fixes https://tracker.ceph.com/issues/68030

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #59635 from zdover23/wip-doc-2024-09-06-rados-ops-health-checks

doc/rados: add "pgs not deep scrubbed in time" info

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/README.md: improve formatting

Improve the formatting in the section "Building Ceph" in the file
README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: mgr/dashboard: Select no device by default in EC profile

Fixes https://tracker.ceph.com/issues/67853

When EC pools are created with device class specified, the pools are created with just 1 PG and autoscaler does not work.
PG autoscaler not working on a cluster where pools have multiple overlapping roots is a known issue, and bug is raised for same :>

Issue documented already : https://docs.ceph.com/en/reef/rados/operations/placement-groups/#viewing-pg-scaling-recommendations

Also renames "let ceph decide" option to "All devices" in crush rule and ec profile component.
Updates unit tests for ec profile modal
Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: expose gw_groups list api

also if the cephadm is available, make the api smart enough to
understand the running daemons and get its gateway_addr instead of
taking the first one in the list so that the HA will be there for the UI
and API as well

Fixes: https://tracker.ceph.com/issues/67774
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: add gw_groups support to nvmeof api

Introduces a gw_group query-param to gateway and subsystem requests.

Modified the dashboard config to support adding more than one gateway to
the list.

Fixes: https://tracker.ceph.com/issues/67774
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59066 from xxhdx1985126/wip-67380

crimson/osd: cancel ongoing pglog-based recoveries on recovery defering

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59485 from mcv21/cephadm_image_warn

cephadm: emit warning if daemon's image is not to be used

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59599 from rhcs-dashboard/fix-67893-main

mgr/dashboard: Rephrase dedicated pool helper in rbd create form

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59447 from afreen23/wip-nvmeof-microcopy

mgr/dashboard: Update nvmeof microcopies

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59569 from afreen23/wip-nvme-ns

mgr/dashboard: Increase maximum namespace count to 1024

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #56858 from rhcs-dashboard/fix-grafana-server-depreciation

mgr/dashboard: use grafana server instead of grafana-server in grafana 10.4.0

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@ibm.com>

Merge pull request #59610 from adk3798/cephadm-ssh-configurable-keepalive

mgr/cephadm: make ssh keepalive settings configurable

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59643 from rkachach/fix_issue_67939

mgr/cephadm: changing the nginx upstream image used by mgmt-gateway

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59658 from phlogistonjohn/jjm-remove-object-deleting-func

mgr/smb: stop trying to clean external store during cluster sync

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #58994 from phlogistonjohn/jjm-smb-vfs-ceph-new

mgr/smb: add support for new ceph vfs module

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>

Merge pull request #58880 from aclamk/wip-aclamk-bs-faster-objectstore-test

test/store_test: Make ceph_test_objectstore run faster

Merge pull request #59107 from nbalacha/wip-nbalacha-async-sorted-snaps

librbd: make "group snap list" async and optionally sorted by snap creation time

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/osd/ops_executer: cleanup snap_map_*

Since https://github.com/ceph/ceph/pull/58868, snap map updates
are now maintained via PG::update_snap_map.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: only increase
osd_osd_param_t::at_version after clone adjustments

Fixes: https://tracker.ceph.com/issues/67945
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/osd_operations/snaptrim_event: we do SnapMapper updates in
ReplicatedBackend::submit_transaction(), drop all SnapMapper updates
before that

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59675 from zdover23/wip-doc-2024-09-10-README-md-vstart

doc/README.md: improve formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

os/bluestore: perfect comments in hybrid_allocator

co-author: Jrchyang Yu <yuzhiqiang_yewu@cmss.chinamobile.com>
Signed-off-by: Wang Linke <wanglinke_yewu@cmss.chinamobile.com>

Merge pull request #58986 from NitzanMordhai/wip-nitzan-crimson-op_gate-alianstore-multi-core

crimson: use gate per shard for AlienStore and OSD

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58868 from xxhdx1985126/wip-crimson-clean_region-based-clone-recovery

crimson/osd/pg: properly propagate snap mapper updates and do clean-region-based clone objects recovery

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: service form hosts selection only show up to 10 entries

exposed new UIRouter API to get all hosts

Fixes: https://tracker.ceph.com/issues/67413
Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #59677 from zdover23/wip-doc-2024-09-10-man-ceph-bluestore-tool

doc/man: edit ceph-bluestore-tool.rst

cephadm: bootstrap: verify orch module is running before setting cephadm backend

There have been occasional failures when bootstrapping with error
messages like

```
Module 'orchestrator' is not enabled/loaded (required by command 'orch set backend'): use `ceph mgr module enable orchestrator` to enable it
```

despite the orchestrator being meant to be an always on module.

Since trying to enable an already enabled module causes no issues,
trying to enable it ourselves seems like a sensible workaround.

Fixes: https://tracker.ceph.com/issues/67969
Signed-off-by: Adam King <adking@redhat.com>

doc/man: edit ceph-bluestore-tool.rst

Edit the section "bluefs-bdev-migrate" in
doc/man/8/ceph-bluestore-tool.rst to add the information that this
operation expands the target storage by updating its size label, making
"bluefs-bdev-expand" unnecessary.

Improve the subject-verb agreement in this section, and supply some
absent definite articles.

Co-authored-by: Peter Gervai <grin@drop.grin.hu>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58603 from cbodley/wip-rgw-sal-concrete-lc

rgw/sal: LCHead and LCEntry don't need abstraction

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>