librbd: introduce rbd_group_snap_namespace_type_t enum

Commit e5ccce14c4b0 ("rbd: add group snap info command") added
a commented out field in rbd_group_snap_info2_t struct but didn't
define the corresponding enum, expecting it to be brought in with the
patchset that introduces mirror group snapshots.  That work is still
in progress, but there is already interest in the new command and APIs
that use rbd_group_snap_info2_t struct and even a backport request.

Finalize the struct definition to avoid a breaking change for those
users in the future.  RBD_GROUP_SNAP_NAMESPACE_TYPE_USER can be treated
as dummy in terms of meaning until RBD_GROUP_SNAP_NAMESPACE_TYPE_MIRROR
is added.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #59653 from NitzanMordhai/wip-nitzan-crimson-add-crontab-weekly-tests

qa/crontab: Add weekly crimson-rados suites

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59859 from afreen23/wip-nvme-service-fixes

mgr/dashboard: Service form fixes for mTLS

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59856 from zdover23/wip-doc-2024-09-18-rados-ops-health-checks

doc/rados: add confval directives to health-checks

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #59809 into main

* refs/pull/59809/head:
COPYING,common/*s390x*: distribute under Apache 2 license for s390x
arch/s390x: add license notices, use LGPL

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge pull request #59839 from Naveenaidu/add-email-details

mailmap, githubmap, organizationmap: add Naveen Naidu

Reviewed-by: Shraddha Agrawal <shraddhaag@ibm.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #59826 from adk3798/main-latest-release-squid

cephadm: update LATEST_STABLE_RELEASE to squid

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59694 from rkachach/fix_issue_67975

mgr/cephadm: open ceph-exporter when firewalld is enabled

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59591 from rhcs-dashboard/grafana-bump-10.4.8

mgr/cephadm: bump grafana to 10.4.8

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: cloudbehl <cloudbehl@gmail.com>

Merge pull request #59174 from adk3798/cephadm-rgw-generate-cert

mgr/cephadm: add ability for cephadm to generate frontend cert for rgw

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55426 from adk3798/cephadm-systemd-ls

mgr/cephadm: add command to expose systemd units of all daemons

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59831 from guits/drive-group-spec-bool-args-followup

orch: Fix `daemon add osd` boolean param handling

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #59558 from jskumari/pic-change-ppc

Change code from non-PIC to PIC for ppc64

mgr/dashboard: Service form fixes for mTLS

Due to conflicts with SSO PR, mTLS fields are not updated properly in code.
This is not happening only in main as noted: https://github.com/ceph/ceph/pull/59819#issuecomment-2356523491

The effect of this issue is mtls based fields not showing in service form

Fixes https://tracker.ceph.com/issues/68133

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59667 from leonidc/gw-no-subsystems

Gw no subsystems

Merge pull request #59417 from nbalacha/wip-nbalacha-ns-mirroring

rbd-mirror: allow mirroring to a different namespace

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

COPYING,common/*s390x*: distribute under Apache 2 license for s390x

The Ceph project cannot distributed GPL code as it is incompatible with the
LGPL that all Ceph code is licensed under. Since s390x is dual-licensed, we
choose to use the Apache 2 license.

Fixes: https://tracker.ceph.com/issues/68084
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

arch/s390x: add license notices, use LGPL

These are original code files introduced by Aliaksei Makarau for s390x
enablement. With his permission, we've licensed these under LGPL.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/rados: add confval directives to health-checks

Add confval directives to doc/rados/operations/health-checks.rst, as
requested by Anthony D'Atri here: https://github.com/ceph/ceph/pull/59635#pullrequestreview-2286205705

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59543 from xxhdx1985126/wip-63844

crimson/osd/pg: do_osd_ops_execute fixes

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59823 from rhcs-dashboard/remove-orch-required-decorator

mgr/dashboard: remove orch required decorator from host UI router (list)

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: dnyanee1997 <NOT@FOUND>

Merge PR #59111 into main

* refs/pull/59111/head:
doc: document earmark option for subvolume and new commands
qa/cephfs: update tests for test_volumes & unit-test for earmarking
mgr/volumes: add earmarking for subvol

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59765 from rhcs-dashboard/carbon-ui-cleanups

mgr/dashboard: fix minor issues in carbon tables

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>

Merge pull request #59373 from rhcs-dashboard/test-ms-alertmanager-v2

qa/task: update alertmanager endpoints version

Reviewed-by: Adam King <adking@redhat.com>

mailmap, githubmap, organizationmap: add Naveen Naidu

Signed-off-by: Naveen Naidu <naveen.naidu@ibm.com>

Merge pull request #59788 from zdover23/wip-doc-2024-09-14-README-md-command-formatting

doc/README.md: create selectable commands

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54277 from rhcs-dashboard/nfs-export-apply-fix

mgr/nfs: generate user_id & access_key for apply_export(CephFS)

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58728 from ifed01/wip-ifed-ret-error-kv-stats

kv/rocksdb: return error for dump_objectstore_kv_stats asok command

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

orch: Fix `daemon add osd` boolean param handling

- Correct usage examples for 'ceph orch daemon add osd'
- Fix boolean argument handling for drive group specifications

Fixes: https://tracker.ceph.com/issues/68045
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/dashboard: fix minor issues in carbon tables

- fixes table in table structure's unusual padding
- fixes the ceph fs details page where table was getting hidden
- improving the subvolume list page by changing the spacings there
- hide the refresh button where it shouldn't be shown.

Fixes: https://tracker.ceph.com/issues/68050
Signed-off-by: Nizamudeen A <nia@redhat.com>

cephadm: update LATEST_STABLE_RELEASE to squid

Signed-off-by: Adam King <adking@redhat.com>

Merge PR #55421 into main

* refs/pull/55421/head:
qa/cephfs: add test to verify backtrace update failure on deleted data pool
mds: batch backtrace updates by pool-id when expiring a log segment
mds: dump log segment in segment expiry callback
mds: dump log segment end along with offset

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #59093 from joscollin/wip-fix-get-set-dirty-snap-id

qa: do the set/get attribute on the remote filesystem

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59777 from ivoalmeida/table-column-pipe-transform-fix

mgr/dashboard: fix table column pipe transform

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

rbd: display mirror uuid for mirror pool info output

Includes the mirror_uuid in the mirror pool info command
output.

Signed-off-by: N Balachandran <nibalach@redhat.com>

mgr/dashboard: remove orch required decorator from host UI router (list)

https://tracker.ceph.com/issues/68101

Signed-off-by: Naman Munet <nmunet@redhat.com>

rbd-mirror: allow mirroring to a different namespace

Allows a namespace in a pool to be mirrored to a differently named
namespace in the secondary cluster.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #59727 from ivoalmeida/snap-schedule-start-time-fix

mgr/dashboard: fix start time format

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

doc: document earmark option for subvolume and new commands

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

qa/cephfs: update tests for test_volumes & unit-test for earmarking

Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mgr/volumes: add earmarking for subvol

- Implemented earmarking functionality for subvolumes and subvolume groups,
ensuring that they do not share the same subvolume or subvolume group to avoid
data corruption due to unsupported mixed protocol use.
- Added a command to set set/rm earmark from a given subvolume / subvolume group
- Adding __init__.py for python-common/ceph/fs dir to have it as a package.
- Fixed subvolume info command when no earmark is set.

Fixes: https://tracker.ceph.com/issues/67460
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #59805 from afreen23/wip-nvme-mtls

mgr/dashboard: Nvme mTLS support and  service name changes

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #59781 from ivoalmeida/table-filter-fix

mgr/dashboard: fix table filter

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

doc: nit fixes for nfs doc

Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Fixes some doc lint and also fixed qa tests for having both 3 & 4 protocols
by default in expot config.

Merge pull request #59668 from rhcs-dashboard/fix-modal-hidden-in-nav

mgr/dashboard: fix UI modal issues

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: Add mTLS support

- enables mTLS support from dashboard
- adds unit tests related to mTLS support
- can enable mTLS
- can disable mTLS
- inlcuded refactoring from prev commit

Fixes https://tracker.ceph.com/issues/66416

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: Adding group and pool name to service name

- Pre-populating the service name field with the format `nvmeof.<pool_name>.<group_name>`.
- This can be changed by user but by default this value will be there.
- This will help user to quickly fill form and proceed hence improving usability.
- cephadm also uses this format as of now this convention so it will make UI aligned with CLI experience

- updates unit tests to improve coverage

- hides`count` values as that is not needed for 'nvmeof' only hosts and labels required.

Fixes https://tracker.ceph.com/issues/68065

Signed-off-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #58618 from rhcs-dashboard/mgmt-gateway-ui

mgr/dashboard: add service management for mgmt-gateway

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #58456 from rhcs-dashboard/auth2-sso

mgr/dashboard: Add SSO through oauth2 protocol

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #59752 from guits/drive-group-spec-bool-args

orch: refactor boolean handling in drive group spec

Reviewed-by: Adam King <adking@redhat.com>

mgr/dashboard: fix UI modal issues

- Fix bs-modal getting hidden behind navbar
- Reuse the project name for tab title
- Readonly user shows an empty space in the administration dropdown
- Physical Disks Identify modal not opening up
   - add e2e in cephadm to make sure the modal always comes up
- Primary action doesn't need to be displayed for the read-only user

Fixes: https://tracker.ceph.com/issues/67547
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59766 from rkachach/fix_issue_68052

mgr/cephadm: renaming whitelist_domains field to allowlist_domains

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

 mon/NVMeofGw*:
 fix issue that GW was down when last subsystem  was deleted

Signed-off-by: Leonid Chernin <leonidc@il.ibm.com>

Merge pull request #59557 from afreen23/wip-ec-profile

mgr/dashboard: mgr/dashboard: Select no device by default in EC profile

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

Merge pull request #45807 from mchangir/doc-document-random-selection-teuthology-operator

doc/dev/developer_guide/testing_integration_tests: Document the random selection operator

mgr/dashboard: add SSO through oauth2 protocol

Fixes: https://tracker.ceph.com/issues/66900
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

mgr/dashboard: add service management for mgmt-gateway

Fixes: https://tracker.ceph.com/issues/66963
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #57459 from mchangir/mds-snap_schedule-relocate-test-and-overrides-to-correct-location

qa: relocate subvol creation overrides and test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #59379 from rhcs-dashboard/oauth2-proxy-ui

mgr/dashboard: add service management for oauth2-proxy

Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #59061 from rhcs-dashboard/rgw-multisite-sync-policy-edit-zones-fix

mgr/dashboard: RGW multisite sync remove zones fix

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

mgr/dashboard: add service management for oauth2-proxy

Fixes: https://tracker.ceph.com/issues/67651
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #59797 from zdover23/wip-doc-2024-09-15-install-install-storage-cluster

doc/install: add prompts to install-storage-cluster

Merge pull request #59801 from zdover23/wip-doc-2024-09-15-rados-ops-health-checks

doc/rados: add osd_deep_scrub_interval setting operation

Merge pull request #59755 from baum/nvmeofspec_addr_zero

pybind/mgr/cephadm/services/nvmeof.py: allow setting '0.0.0.0' as add…

doc/rados: add osd_deep_scrub_interval setting operation

Add a second method of changing the value of osd_deep_scrub_interval to
remedy the condition indicated by the "PGs not deep-scrubbed in time"
warning.

This procedure was developed by Eugen Block, and is at the time of this
commit available on his blog at
https://heiterbiswolkig.blogs.nde.ag/2024/09/06/pgs-not-deep-scrubbed-in-time/

Co-authored-by: Eugen Block <eblock@nde.ag>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/install: add prompts to install-storage-cluster

Add prompts to doc/install/install-storage-cluster.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59636 from ronen-fr/wip-rf-smaller-deep-chunk

osd/scrub: decrease default deep scrub chunk size

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/README.md: create selectable commands

An indentation of five spaces relative to the previous line creates a
command that is copyable with a single mouse click. This commit adds
those copyabale commands to the procedure in the section "Building
Ceph".

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59732 from zdover23/wip-doc-2024-09-11-README-md-formatting

doc/README.md: improve formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

crimson/osd/pg: make "PG::submit_error_log()" and
"PG::complete_error_log" interruptible

Fixes: https://tracker.ceph.com/issues/67800
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/ops_executer: revoke OpsExecuter::get_obc()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/ops_executor: simplify OpsExecuter::rollback_obc_if_modified

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg: rollback ops by copying obc beforehand and recover after
errors

Fixes: https://tracker.ceph.com/issues/63844
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #54777 from cbodley/wip-rgw-meta-handlers

rgw: remove unused backend abstraction for metadata handlers

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

mgr/dashboard: fix table filter

Fixes: https://tracker.ceph.com/issues/68063
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

orch: refactor boolean handling in drive group spec

The intent of 42721c03ee6f was to address an issue where boolean
parameters weren't handled correctly.

I noticed that a parameter (`tpm2`) was missed, which made me realize
that maintaining a list of these boolean parameters is necessary.

To simplify things, we should only accept `"true"` or `"false"` (in any case),
allowing us to avoid the need to maintain a list of boolean parameters.

This change introduces a `list_drive_group_spec_bool_arg` to store boolean
arguments related to drive group specifications, simplifying the validation
process for boolean values by directly checking if the values are 'true' or 'false'.

Fixes: https://tracker.ceph.com/issues/68045
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mgr/dashboard: RGW multisite sync remove zones fix

Fixes: https://tracker.ceph.com/issues/66945
Signed-off-by: Naman Munet <nmunet@redhat.com>

mgr/dashboard: fix start time format

Fixes: https://tracker.ceph.com/issues/68024
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

mgr/dashboard: fix table column pipe transform

Fixes: https://tracker.ceph.com/issues/68062
Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

rgw/service: remove unused metadata and metadata backend services

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove unused RGWSI_Role_RADOS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/role: add rgwrados::role interface for role metadata

move everything from class RadosRole into driver/rados/role.cc and use
RGWSI_SysObj instead of the metadata backend. narrows the RGWRole
interface by handling the name/path objects internally

rgwrados::role::write() adds/remove name and path linkages in case they
change. admin ops don't allow this, but 'metadata put' could upload
arbitrary json that does change them

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for role metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: encapsulate role metadata handler in rgw_role.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for user metadata

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: remove unused 'meta backend ctx' arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/user: call sysobj directly instead of 'sysobj meta backend'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for bucket instances

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: use raw RGWMetadataHandler for bucket entrypoints

RGWBucketMetadataHandler inherits from RGWMetadataHandler directly
instead of RGWMetadataHandler_GenericMetaBE

replace RGWSI_Bucket_SObj_Module with the RGWMetadataLister abstraction

RGWSI_Bucket_SObj now depends on RGWSI_MDLog and writes mdlog entries
itself

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: remove unused 'meta backend ctx' arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: call sysobj directly instead of 'sysobj meta backend'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/bucket: free functions for bucket instance metadata keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove otp service and metadata backend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/metadata: remove backend stuff from otp metadata handler

inherit from RGWMetadataHandler instead of
RGWMetadataHandler_GenericMetaBE to avoid all of the metadata backend
abstractions, and just call into the cls and sysobj services directly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/services: remove unused RGWOTPCtl

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: clear output keys in RGWMetadataLister::get_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw/posix: add missing include for cpp_strerror()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd/scrub: decrease default deep scrub chunk size

The previous default of 25 objects per chunk proved to take too long
(many hundreds of milliseconds) on a busy cluster. As the scrubber
locks all objects in the chunk for the duration, a large chunk size
can cause a significant impact on the client ops' latencies.

Fixes: https://tracker.ceph.com/issues/68057
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #59696 from wanglinke521/wlk-perfect-comment

os/bluestore: perfect comments in hybrid_allocator

Reviewed-by: Adam Kupczyk <kupczyk@ibm.com>
Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

Merge pull request #59438 from mkogan1/wip-d3n-ex-co-ar

doc/rgw/d3n: pass cache dir volume to extra_container_args