ceph-volume: update unit tests

This commit consolidates all necessary changes to address unit tests
related to the introduction of Python type annotations.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: remove duplicate code

This check is already performed in `arg_validators.py`.
By the way, this check is only valid when the user passes a VG/LV with
the `<vg>/<lv>` format.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.batch

This commit adds the Python type annotations to `devices.lvm.batch`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.listing

This commit adds the Python type annotations to `devices.lvm.listing`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: improve clarity and consistency in List.single_report

- Rename the parameter `arg` to `osd` for better readability.
- Simplify the check for numeric OSD identifiers using `isdigit()`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.common

This commit adds the Python type annotations to `devices.lvm.common`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.prepare

This commit adds the Python type annotations to `devices.lvm.prepare`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.trigger

This commit adds the Python type annotations to `devices.lvm.trigger`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.main

This commit adds the Python type annotations to `devices.lvm.main`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.deactivate

This commit adds the Python type annotations to `devices.lvm.deactivate`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.create

This commit adds the Python type annotations to `devices.lvm.create`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to devices.lvm.activate

This commit adds the Python type annotations to `devices.lvm.activate`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: fix type annotation in `objectore`

This commit addresses some python type annotations errors
in `objectstore` code.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotation to api.lvm

This adds Python type annotations to `api.lvm`, along with all
necessary adjustments to ensure compatibility and maintain code
clarity.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add type annotations to util.device

This adds Python type annotations to `ceph_volume.util.device`,
along with all necessary adjustments to ensure compatibility
and maintain code clarity.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add typing hints to lvm.zap

This adds the python typing hints to lvm.zap

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: set default value for BlueStore.block_lv to None

This change updates the `BlueStore` class in
`ceph_volume.objectstore` by initializing the `block_lv` attribute
to `None` with the type `Optional[Volume]`. This ensures that the
attribute has a default value and avoids potential runtime errors
when the attribute is accessed before being explicitly assigned.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: improve wipefs retry logic in lvm.zap

- Simplify the initialization of `tries` and `interval` variables for clarity.
- Adjust the retry logic in the `wipefs` function to:
  - Include the attempt count in the warning message for better debugging.
  - Start the retry loop at 1 and increment up to `tries`.
- Remove unnecessary unpacking of `stdout` and `stderr` since they were unused.
- Update the loop to increment `tries` by 1 to reflect the intended number of attempts.

This change improves code readability and makes retry behavior more transparent.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #61240 from neesingh-rh/wip-68974

doc: add snapshots under cephfs concepts

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61243 from anthonyeleven/etags-fix

doc/radosgw/s3: correct eTag op match tables

doc/radosgw/s3: correct eTag op match tables

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

doc: add snapshots in docs under Cephfs concepts

Fixes: https://tracker.ceph.com/issues/68974
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #60720 from batrick/i68913

qa: write out ESubtreeMap more frequently to find large events

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #59593 from xxhdx1985126/wip-67888

crimson/osd/backfill_state: treat `Cancelled` as a pause of the ongoing backfilling

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60597 from xxhdx1985126/wip-68806

crimson/osd/replicated_recovery_backend: call on_global_recover() only when all replicas and the primary have been recovered

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60041 from xxhdx1985126/wip-68286

crimson/osd/pg_shard_manager: discard outdated operations when the corresponding pgs are already removed

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61216 from athanatos/sjust/wip-crimson-backfill-teuth

qa/suites/crimson-rados: enable short_pg_log

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61227 from zdover23/wip-doc-2024-01-05-README

doc: README.md - improve "Tshooting" and "Tips & Tricks"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61148 from ronen-fr/wip-rf-abortReserv

osd/scrub: abort running scrub in replica-reservation if an operator-initiated scrub is requested

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: README.md - improve "Tshooting" and "Tips & Tricks"

Improve the formatting and English language in the sections
"Troubleshooting" and "Tips and Tricks", and move those sections to a
place where they don't interrupt the flow of the vstart cluster
installation instructions. Some of the strings in "Tips and Tricks" are
not yet unambiguous sentences that will make sense to the uninitiated,
but this PR represents a step in that direction.

This PR is part of a series of PRs meant to preserve the integrity of
the README.md file after some recent additions that break the flow of
the document.

This PR follows https://github.com/ceph/ceph/pull/61226 and
https://github.com/ceph/ceph/pull/61221.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61226 from zdover23/wip-doc-2025-01-04-README

doc: README.md - format "Troubleshooting"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61086 from athanatos/sjust/wip-rep-pipeline

crimson: allow replica side write commits to pipeline

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

doc: README.md - format "Troubleshooting"

Format "Troubleshooting" into its own section so that it doesn't confuse
readers of the vstart installation procedure.

This PR is part of a series of PRs meant to preserve the integrity of
the README.md file after some recent additions that break the flow of
the document.

This PR follows https://github.com/ceph/ceph/pull/61221.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61221 from zdover23/wip-doc-2025-01-03-README

doc: README.md - format "Tips and Tricks"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60832 from clwluvw/bucket-delete-meta

rgw: consider multi zonegroup for is_syncing_bucket_meta

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60685 from clwluvw/data-sync-perm

rgw: respect policies in data sync in user mode

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #61217 from gbregman/main

mgr/cephadm/nvmeof: Add key verification field to NVMeOF configuration

doc: README.md - format "Tips and Tricks"

Format "Tips and Tricks" into its own section so that it doesn't confuse
readers of the vstart installation procedure.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/cephadm/nvmeof: Add key verification field to NVMeOF configuration
Fixes https://tracker.ceph.com/issues/69413

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #59980 from kchheda3/wip-fix-v1-v2-notification

rgw/notification: Forward Topic & Notification creation request to master when notification_v2 enabled

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #60430 from ivancich/wip-fix-multipart-empty-storage-class

rgw: fix empty storage class on display of multipart uploads

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59631 from thotz/create-user-without-creds-cli

radosgw-admin: create user without creds cli

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #60411 into main

* refs/pull/60411/head:
client: Fix a deadlock when osd is full

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #61103 from yuvalif/wip-yuval-fix-test-names

test/rgw/noitifications: fix test names

Reviewed-By: Ali Masarwe <ali.masarwa@ibm.com>

qa/standalone/scrub: osd-scrub-test.sh - test operator overrides

verify that an operator scrub aborts a reserving scrub of the
same PG.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: reset m_active_target when the scrub ends

... as it is now queried to determine whever we are scrubbing,
but not yet 'm_active', as the scrubber is in ReservingReplicas.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: convey 'reserving replicas' status in query results

... and not just in 'pg dump' output.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/standalone/scrub: add build_pg_dicts()

a helper function that builds bash dictionaries:
pg to acting set, pg to primary & pg to pool.

Also added are two helper functions that make use of the dictionaries:

count_common_active() to count the number of common OSDs
in the acting set of two PGs, and find_disjoint_but_primary()
to find a PG that is disjoint from the first PG, apart from
possibly having the same primary OSD.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61197 from bebehei/fix-haproxy-dashboard-docs

doc/mgr/dashboard: Fix HAProxy TLS example

doc/mgr/dashboard: Fix HAProxy TLS example

With `ssl` set on the `server` option, HAProxy strips the TLS protocol
for all clients. You would need to connect to it with `http://<ip>:443`.

To have an active health check, which uses SSL, but does not strip it
for clients, you'd need to add:

- `check` to enable active health checks.
- `check-ssl` to instruct the health check to use TLS
- `verify none` to skip verification on the health check requests from
  HAProxy
- _REMOVE_ `ssl` to stop stripping TLS

The active health checks are required to not route any requests to the
inactive managers. These would redirect to any unusable IP from the
active mgr.

---

Alternatively you could add another certificate in the frontend and then
re-encrypt the traffic. But this would require tracking the certs also
in HAProxy.

Signed-off-by: Benedikt Heine <bebe@bebehei.de>

test/rgw/noitifications: fix test names

for persistent topic stats tests

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #60794 from dparmar18/wip-68571

doc/cephfs: document purge queue and its perf counters

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa: write out ESubtreeMap more frequently to find large events

With the trimming changes by 9d2b3aa, ESubtreeMap wasn't written reliably often
enough to pass the test.

Fixes: https://tracker.ceph.com/issues/68913
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #61191 from zdover23/wip-doc-2024-12-29-README-cleanup

doc: README.md - format admonition

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: README.md - format admonition

Format an admonition correctly. This commit is a prelude to a cleanup of
a recent addition to README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

osd/scrub: abort reserving scrub if an operator-initiated scrub is
requested

Handling the case of receiving an operator command while the PG is
scrubbing, but
is waiting for replicas' reservations:

Now that the reservations are queued, the wait may be a very prolonged
one.
Usually - an operator direct scrub command has a priority high enough
to not require waiting for reservations. But in the current
implementation,
it would wait until the running scrub session terminates, and only then
will rerun at that high priority. This is not the intended behavior.

The solution is to abort the existing scrub session, and start the new
one.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: register for 'osd_max_scrubs' config changes

Since https://github.com/ceph/ceph/pull/55340, the
osd_max_scrubs (also) affects the parameters of the
async scrub reserver used by the replicas. Thus,
the code must notice and acknowledge changes to this config.

Fixes: https://tracker.ceph.com/issues/69362
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61142 from Dedsec0098/wip-doc-shrish

doc: Update vstart section in readme.md

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61156 from zdover23/wip-doc-2024-12-20-radosgw-uadk-accel

doc/radosgw: line edit uadk-accel.rst (1st half)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61184 from ronen-fr/wip-rf-watch-scrubmax

osd/scrub: register for 'osd_max_scrubs' config changes

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #55616 into main

* refs/pull/55616/head:
PendingReleaseNotes: add note for replay completion warning
qa: test to verify `MDS_ESTIMATED_REPLAY_TIME` warning
doc: add a note for `MDS_ESTIMATED_REPLAY_TIME` MDS warning
mds: emit warning for estinated replay time

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #60640 into main

* refs/pull/60640/head:
qa: use a larger timeout for kernel_untar_build workunit

Reviewed-by: Milind Changire <mchangir@redhat.com>

osd/scrub: register for 'osd_max_scrubs' config changes

Since https://github.com/ceph/ceph/pull/55340, the
osd_max_scrubs (also) affects the parameters of the
async scrub reserver used by the replicas. Thus,
the code must notice and acknowledge changes to this config.

Fixes: https://tracker.ceph.com/issues/69362
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #60988 from thuvh/feat/kafka_multiple_brokers

rgw: allow bucket notification send message to kafka with multiple brokers

Merge pull request #61144 from cyx1231st/wip-seastore-improve-backref

crimson/os/seastore/cache: improve committing allocations by backref entries

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Zhang Song <zhangsong02@qianxin.com>

doc/cephfs: document purge queue and its perf counters

Fixes: https://tracker.ceph.com/issues/68571
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

Merge PR #60653 into main

* refs/pull/60653/head:
mds: do not process client metrics message with fast dispatch

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #61143 from kotreshhr/qa-nfsganesha-fio-test

qa: Add async io test to nfs ganesha using fio

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55145 from mchangir/mgr-snap_schedule-handle-volume-delete

mgr/snap_schedule: handle volume delete

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw: allow bucket notification send message to kafka with multiple brokers

- add new parameter to define list of broker
- update document
- change nose to nose-py3
- add test case for multiple brokers (happy case)

Signed-off-by: Hoai-Thu Vuong <thuvh87@gmail.com>

Merge pull request #61177 from Skalador/wip-doc-typo-hardware-controllers

doc/start: Fixes a typo in controllers section of hardware recommendations

doc: Fixes a typo in controllers section of hardware recommendations

Signed-off-by: Kevin Niederwanger <k.niederwanger@gmail.com>

Merge pull request #59116 from rzarzynski/wip-constbl-compressor

compressor: switch out of get_current_ptr() for the sake of const-stricter bufferlist

Reviewed-by: Bill Scales <bill_scales@uk.ibm.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #60808 from ronen-fr/wip-rf-reservStat2

osd/scrub: show reservation status in 'pg dump' output

Reviewed-by: Nitzan Mordechai <nmordech@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd/scrub: show reservation status in 'pg dump' output

Whenever a PG is selected for scrubbing, and is waiting for
remote reservations, the 'pg dump' output will include the
following text (under the 'SCRUB_SCHEDULING' column):
Reserving. Waiting Ns for OSD.k (n/m)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os/seastore/cache: move committing existing blocks to prepare_record()

So the 2 steps of remapping are atomically committed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Zhang Song <zhangsong02@qianxin.com>

crimson/os/seastore/journal: hide RBM specific finish_commit()

By introducing a callback upon completing submission.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: minor cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc: Update vstart section in readme.md

Add guidance about vstart clusters to README.md.

Signed-off-by: Shrish Mishra <shrish409@gmail.com>

Merge pull request #60821 from Matan-B/wip-matanb-crimson-seastore-deafult

crimson: switch to SeaStore as default object store

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61155 from idryomov/wip-69324

rbd: handle --{group,image}-namespace in "rbd group image {add,rm}"

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #61150 from idryomov/wip-69319

rbd: open images in read-only mode for "rbd mirror pool status --verbose"

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #61129 from idryomov/wip-68998

librbd: avoid data corruption on flatten when object map is inconsistent

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #61125 from ljflores/wip-release-note

Merge PR #61149 into main

* refs/pull/61149/head:
script/ceph-backport: allow hyphen in project during ceph fork deduction

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #60777 from VallariAg/wip-nvmeof-prometheus-rbd-image-reused

monitoring: Add prometheus alert NVMeoFMultipleNamespacesOfRBDImage

Reviewed-by: Afreen Misbah <afreen@ibm.com>

rbd: drop --pool option from "rbd group image {add,rm}"

It stopped working with removal of get_special_pool_group_names() in
commit 3e8624f157a1 ("rbd: add support for namespaces") over six years
ago.  Given how much time has passed, stop accepting this option.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/radosgw: line edit uadk-accel.rst (1st half)

Line edit the first half of doc/radosgw/uadk-accel.rst.

This is one of a series of pull requests intended to improve the content
of uadk-accel.rst. This series includes the following PRs:

https://github.com/ceph/ceph/pull/59514
https://github.com/ceph/ceph/pull/60953
https://github.com/ceph/ceph/pull/61128

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61146 from zdover23/wip-doc-2024-12-19-radosgw-uadk-accel

doc/radosgw: edit uadk-accel.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rbd: handle --{group,image}-namespace in "rbd group image {add,rm}"

Currently only passing the namespace as part of the group or image spec
works.  If --group-namespace or --image-namespace options are used, the
namespace isn't picked up.

Fixes: https://tracker.ceph.com/issues/69324
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

PendingReleaseNotes: add note about tracker #69012

We merged a fix for v19.2.1 that helps alleviate
the worst of this problem (https://tracker.ceph.com/issues/68657),
but it still comes up on occasion. This release note addresses the
remaining issues tracked in https://tracker.ceph.com/issues/69012.

Signed-off-by: Laura Flores <lflores@ibm.com>

rbd: open images in read-only mode for "rbd mirror pool status --verbose"

This is cleaner and makes the command run a bit faster because watches
won't be established.

Fixes: https://tracker.ceph.com/issues/69319
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

script/ceph-backport: allow hyphen in project during ceph fork deduction

This will allow the fork deduction code to recognize project names
that include a hyphen, such as:

    git@github.com:USERNAME/ceph-fork.git

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #61113 from leonidc/wip-leonidc_no_listener_fix

nvmeofgw*: fix no-listeners

Merge pull request #61107 from cbodley/wip-69241

rgw/posix: std::ignore return value of write()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #59703 from rhcs-dashboard/usermanagement-form

mgr/dashboard: carbonize user management form

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #61130 from baum/wip-baum-20241218-00

src/nvmeof/NVMeofGwMonitorClient: remove MDS client, not needed

doc/radosgw: edit uadk-accel.rst

Edit the sections of doc/radosgw/uadk-accel.rst that concern the
automatic and manual building of UADK.

This is one in a series of uadk-accel.rst-related changes that includes
the following PRs:
https://github.com/ceph/ceph/pull/60953
https://github.com/ceph/ceph/pull/61128

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61128 from zdover23/wip-doc-2024-12-18-radosgw-uadk-accel

doc/radosgw: edit uadk-accel.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

vstart: use crimson_osd_objectstore

Signed-off-by: Matan Breizman <mbreizma@redhat.com>