common/io_exerciser: Reorder lines in RadosIo send_command functions

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common/io_exerciser: Clear the stringstream in RadosIO class

Ensures the stringstream is empty before sending a command in the RadosIO class

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

Merge pull request #61254 from kamoltat/wip-ksirivad-fix-stretch-mode-doc

doc/rados/operations/stretch-mode: Improve doc
Reviewed-by: zdover23
Reviewed-by: anthonyeleven

Merge pull request #61288 from adamemerson/wip-69303

rgw: Don't crash on exceptions from pool listing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Don't crash on exceptions from pool listing

Fixes: https://tracker.ceph.com/issues/69303
Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #60330 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_inject_error

common/io_exerciser: Add support to ceph_test_rados_io_sequence injecting errors for testing how erasure coding deals with error scenarios

Reviewed-by: Ronen Friedman <rfriedma@ibm.com>

Merge pull request #52791 from clwluvw/location-constraint

rgw: check for location constraint on master zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59960 from smanjara/wip-fix-missing-http-data

rgw/multisite:  include request body when CreateBucket op is forwarded to master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60881 from NitzanMordhai/wip-nitzan-ipv6-subnet-checks

common/pick_address: Add IPv6 support to is_addr_in_subnet

Merge pull request #60987 from drunkard/main

doc: fixes in doc building

Merge pull request #60883 from xxhdx1985126/wip-crimson-backfill-throttle

crimson/osd/pg_recovery: throttle backfills together with pglog based recoveries

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #61262 from VallariAg/fix-nvmeof-teuthology-basic-test

qa/workunits/nvmeof/basic_tests.sh: fix nvme list assert

Merge pull request #60623 from piyushagarwal1411/ceph-exporter

vstart: add ceph-exporter support

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #60891 from xxhdx1985126/wip-seastore-fadvise-backfill

crimson/os/seastore: add fadvise support to SeaStore and prevent recovery/backfill from polluting the cache of SeaStore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados/operations/stretch-mode: Improve doc

Added more content and rewrite some sections

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #60848 from cbodley/wip-rgw-deprecate-iam-tenant

docs/rgw: deprecate tenant-based IAM in favor of accounts

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61074 from chardan/wip-radowsgw-admin-jfw-restructure_file

rgw: migrate rgw_admin to new radosgw-admin directory

Merge pull request #60379 from cbodley/wip-librados-cancellation

librados/asio: forward asio cancellations to AioCompletion::cancel()

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rgw/multisite: the create_bucket forward request omits the
the request body, thus missing some data if specified inside
CreateBucketConfiguration xml on the non-master zone.
also, now that we perform cksum validation against empty payloads,
such a request would fail with -ERR_AMZ_CONTENT_SHA256_MISMATCH due
to a zero content-length but a non-empty payload hash.
this fix ensures that request body is forwarded during create_bucket

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #61127 from VallariAg/wip-nvmeof-delete-healthcheck

mon/NVMeofGwMap: add healthcheck warning NVMEOF_GATEWAY_DELETING

Merge pull request #61136 from cbodley/wip-69301

rgw: don't use merge_and_store_attrs() when recreating a bucket

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #61255 from Matan-B/wip-matanb-crimson-seastore-deafult-2

common/options/crimson.yaml.in: fallback to Bluestore by default

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61251 from guits/ceph-volume-69432

ceph-volume: fix loop devices support

vstart.sh: add support for launching the ceph-exporter daemon

Signed-off-by: Piyush Agarwal <piyushagarwal14.pa@gmail.com>

Merge pull request #61252 from guits/hints-create_id

ceph-volume: add python hints to util.prepare.create_id()

qa/workunits/nvmeof/basic_tests.sh: fix connect-all assert

There seems to be change in 'nvme list' json output
which caused failures in asserts after 'nvme connect-all'
command.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #61213 from Matan-B/wip-matanb-crimson-shared-lru-dtor

crimson/common/shared_lru: invalidate Deleter's cache

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60666 from ThomasLamprecht/debian-fix-librgw2-lua-dependencies

debian/control: fix overly broad lua dependency declaration for librgw2 package

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #60118 from kshtsk/wip-refactor-make-check

script/run-make: stop args duplication

Merge pull request #61241 from guits/ceph-volume-69430

ceph-volume: fix Zap.ensure_associated_raw()

ceph-volume: fix loop devices support

This commit updates the `is_device` function to correctly handle
loop devices.
The function now validates loop devices when they are explicitly
allowed, by checking their type (`loop`) in addition to `disk`
and `mpath`.

Changes include:
  - Extending the type check to include `loop` in the list of
    supported device types.
  - Enhancing the docstring for better documentation of the
    function's purpose and behavior.

These changes ensure that loop devices are properly recognized
and handled when configuring OSDs in ceph-volume.

Fixes: https://tracker.ceph.com/issues/69432
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

common/io_exerciser: Make chunksize so initial generated value is 4096 and random values are generated thereafter

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common/io_exerciser: Make sure Sequence 10 removes objects after finishing running

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/json: Modified JSON structures so they take advantage of ceph_json.h fully.

Also moved and renamed JSONStructures files so they structures are more easily identifiable and usable by others if desired.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/io_exerciser: Remove unneccisairy override in data_generation::SeededRandomGenerator

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/io_exerciser: add missing override statements to JsonStructures.h

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

src/common/io_exerciser: Formatting improvements using clang format

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common/io_exerciser: Add simple sequences for testing error injects

Add sequences to test IOs with simple error injects, along with some small fixes for previous error inject implementation.

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

common/io_exerciser: Add injecterror commands to ceph_test_rados_io_sequence interactive mode

Add injecterror commands that can be used in interactive mode to inject read and write errors as well as clear them

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

osd: EC error inject interfaces

Error inject interfaces for EC reads and writes using
ceph tell osd.<n> interface

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

src/common/io_exerciser: Code readability improvements

Implement inheritence for ceph::io_exerciser::IoOp to allow better differentiation between the different Op types and allow more complex Operations to be implemented

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

Revert "doc/dev/crimson: update SeaStore as default backend"

This reverts commit 41327dcf0f733777f0d022d1bab7bf72b9807b3c.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados-exp: add Seastore/thrash

As Seastore/trash was removed from the non-exp suite, it is
moved here until fully supported.
Follow-up to: 5150dae471c

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados-exp: remove basic

"basic" directory is included in the non-exp suite and is fully supported

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/suites/crimson-rados: disable thrash/seastore

Seastore supports thrash_simple only until https://tracker.ceph.com/issues/69405
is resolved.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

common/options/crimson.yaml.in: Bluestore by default

This partially reverts f4dee79c3309e5ac1e1a142b85f492851e6757e1
Until https://tracker.ceph.com/issues/69402 is resolved.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

ceph-volume: add python hints to util.prepare.create_id()

This commit introduces type annotations to the `create_id` function in `ceph_volume.util.prepare`.
The parameters and return value are now typed as follows:
  - `fsid` is a `str`.
  - `json_secrets` is a `str`.
  - `osd_id` is an optional `str` (`Optional[str]`).
  - The function returns a `str`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

mon/NVMeofGwMap: add delay to NVMEOF_GATEWAY_DELETING warning

Instead of immediately triggering, have this healthcheck trigger
after some time has elasped. This delay can be configured by
mon_nvmeofgw_delete_grace.

Track the time when gateways go into DELETING state in a new
member var (of NVMeofGwMon) 'gws_deleting_time'.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #61240 from neesingh-rh/wip-68974

doc: add snapshots under cephfs concepts

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw: migrate rgw_admin to new directory.

Some C++ fixes, macro fixes, etc.. Gathers other core
files, makes names more closely match executable.

Signed-off-by: Jesse F. Williamson <jfw@ibm.com>

rgw: don't use merge_and_store_attrs() when recreating a bucket

https://github.com/ceph/ceph/pull/56583 recently fixed
merge_and_store_attrs() to preserve existing attrs, but this broke the
swift api's ability to remove container metadata. RGWCreateBucket
handles this merging itself with prepare_add_del_attrs(), so we should
just assign createparams.attrs to the bucket and store it with
bucket->put_info()

make the same change for RGWPutMetadataBucket which swift uses to
add/remove existing metadata

Fixes: https://tracker.ceph.com/issues/69301
Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: fix Zap.ensure_associated_raw()

When an OSD creation fails, ceph-volume can zaps unrelated
existing raw based OSD as part of the 'rollback step'.

Fixes: https://tracker.ceph.com/issues/69430
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #61243 from anthonyeleven/etags-fix

doc/radosgw/s3: correct eTag op match tables

doc/radosgw/s3: correct eTag op match tables

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

src/common/options/mon.yaml.in: add mon_nvmeofgw_delete_grace

This config allows to configure the delay in triggering
NVMEOF_GATEWAY_DELETING healthcheck warning, which is
triggered when NVMeoF gateways are in DELETEING state
for too long (indicating a problem in namespace
load-balacing).
The default value for this config is 15 mins.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

doc: add snapshots in docs under Cephfs concepts

Fixes: https://tracker.ceph.com/issues/68974
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Merge pull request #60720 from batrick/i68913

qa: write out ESubtreeMap more frequently to find large events

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/osd/replicated_recovery_backend: prevent recovery/backfills from
polluting the cache of the underlying futurized store

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os: all "read/get" interfaces accept op_flags

To be used by exclusively by Seastore, See `seastore::cache_hint_t`

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore: introduce cache_hint_t

Layers above Cache can use cache_hint_t to notify Cache whether to put
the extents to Cache::lru.

CEPH_OSD_OP_FLAG_FADVISE_DONTNEED and CEPH_OSD_OP_FLAG_FADVISE_NOCACHE
will be treated as "no need to put into Cache::lru"

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg_recovery: throttle backfills together with pg-log based
recoveries

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #59593 from xxhdx1985126/wip-67888

crimson/osd/backfill_state: treat `Cancelled` as a pause of the ongoing backfilling

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60597 from xxhdx1985126/wip-68806

crimson/osd/replicated_recovery_backend: call on_global_recover() only when all replicas and the primary have been recovered

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60041 from xxhdx1985126/wip-68286

crimson/osd/pg_shard_manager: discard outdated operations when the corresponding pgs are already removed

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61216 from athanatos/sjust/wip-crimson-backfill-teuth

qa/suites/crimson-rados: enable short_pg_log

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61227 from zdover23/wip-doc-2024-01-05-README

doc: README.md - improve "Tshooting" and "Tips & Tricks"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61148 from ronen-fr/wip-rf-abortReserv

osd/scrub: abort running scrub in replica-reservation if an operator-initiated scrub is requested

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: README.md - improve "Tshooting" and "Tips & Tricks"

Improve the formatting and English language in the sections
"Troubleshooting" and "Tips and Tricks", and move those sections to a
place where they don't interrupt the flow of the vstart cluster
installation instructions. Some of the strings in "Tips and Tricks" are
not yet unambiguous sentences that will make sense to the uninitiated,
but this PR represents a step in that direction.

This PR is part of a series of PRs meant to preserve the integrity of
the README.md file after some recent additions that break the flow of
the document.

This PR follows https://github.com/ceph/ceph/pull/61226 and
https://github.com/ceph/ceph/pull/61221.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61226 from zdover23/wip-doc-2025-01-04-README

doc: README.md - format "Troubleshooting"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61086 from athanatos/sjust/wip-rep-pipeline

crimson: allow replica side write commits to pipeline

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

doc: README.md - format "Troubleshooting"

Format "Troubleshooting" into its own section so that it doesn't confuse
readers of the vstart installation procedure.

This PR is part of a series of PRs meant to preserve the integrity of
the README.md file after some recent additions that break the flow of
the document.

This PR follows https://github.com/ceph/ceph/pull/61221.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61221 from zdover23/wip-doc-2025-01-03-README

doc: README.md - format "Tips and Tricks"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60832 from clwluvw/bucket-delete-meta

rgw: consider multi zonegroup for is_syncing_bucket_meta

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60685 from clwluvw/data-sync-perm

rgw: respect policies in data sync in user mode

Reviewed-by: Adam Emerson <aemerson@redhat.com>

mon/NVMeofGwMap: add healthcheck warning NVMEOF_GATEWAY_DELETING

Add a warning when NVMeoF gateways are in DELETING state.
This happens when there are namespaces under the deleted gateway's
ANA group ID.

The gateways are removed completely after users manually move these
namespaces to another load balancing group. Or if a new gateway is
deployed on that host.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #61217 from gbregman/main

mgr/cephadm/nvmeof: Add key verification field to NVMeOF configuration

doc: README.md - format "Tips and Tricks"

Format "Tips and Tricks" into its own section so that it doesn't confuse
readers of the vstart installation procedure.

Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/cephadm/nvmeof: Add key verification field to NVMeOF configuration
Fixes https://tracker.ceph.com/issues/69413

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #59980 from kchheda3/wip-fix-v1-v2-notification

rgw/notification: Forward Topic & Notification creation request to master when notification_v2 enabled

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #60430 from ivancich/wip-fix-multipart-empty-storage-class

rgw: fix empty storage class on display of multipart uploads

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59631 from thotz/create-user-without-creds-cli

radosgw-admin: create user without creds cli

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/common/shared_lru: rename Deleter::cache

to not be confused with SharedLRU::cache

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/shared_lru: invalidate Deleter's cache

Once we destruct SharedLRU, SharedLRU::weak_refs map is destroyed.
As a weak refernce might outlive the SharedLRU itself, when destroying
the object via the custom Deleter, we try to access the already
destroyed SharedLRU instance's weak ref map.

Instead, invalidate the custom Deleter (Deleter::cache), when
destructing the SharedLRU.

Fixes: https://tracker.ceph.com/issues/66478
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge PR #60411 into main

* refs/pull/60411/head:
client: Fix a deadlock when osd is full

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #61103 from yuvalif/wip-yuval-fix-test-names

test/rgw/noitifications: fix test names

Reviewed-By: Ali Masarwe <ali.masarwa@ibm.com>

qa/standalone/scrub: osd-scrub-test.sh - test operator overrides

verify that an operator scrub aborts a reserving scrub of the
same PG.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: reset m_active_target when the scrub ends

... as it is now queried to determine whever we are scrubbing,
but not yet 'm_active', as the scrubber is in ReservingReplicas.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: convey 'reserving replicas' status in query results

... and not just in 'pg dump' output.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/standalone/scrub: add build_pg_dicts()

a helper function that builds bash dictionaries:
pg to acting set, pg to primary & pg to pool.

Also added are two helper functions that make use of the dictionaries:

count_common_active() to count the number of common OSDs
in the acting set of two PGs, and find_disjoint_but_primary()
to find a PG that is disjoint from the first PG, apart from
possibly having the same primary OSD.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61197 from bebehei/fix-haproxy-dashboard-docs

doc/mgr/dashboard: Fix HAProxy TLS example

doc/mgr/dashboard: Fix HAProxy TLS example

With `ssl` set on the `server` option, HAProxy strips the TLS protocol
for all clients. You would need to connect to it with `http://<ip>:443`.

To have an active health check, which uses SSL, but does not strip it
for clients, you'd need to add:

- `check` to enable active health checks.
- `check-ssl` to instruct the health check to use TLS
- `verify none` to skip verification on the health check requests from
  HAProxy
- _REMOVE_ `ssl` to stop stripping TLS

The active health checks are required to not route any requests to the
inactive managers. These would redirect to any unusable IP from the
active mgr.

---

Alternatively you could add another certificate in the frontend and then
re-encrypt the traffic. But this would require tracking the certs also
in HAProxy.

Signed-off-by: Benedikt Heine <bebe@bebehei.de>

test/rgw/noitifications: fix test names

for persistent topic stats tests

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #60794 from dparmar18/wip-68571

doc/cephfs: document purge queue and its perf counters

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa: write out ESubtreeMap more frequently to find large events

With the trimming changes by 9d2b3aa, ESubtreeMap wasn't written reliably often
enough to pass the test.

Fixes: https://tracker.ceph.com/issues/68913
Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #61191 from zdover23/wip-doc-2024-12-29-README-cleanup

doc: README.md - format admonition

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: README.md - format admonition

Format an admonition correctly. This commit is a prelude to a cleanup of
a recent addition to README.md.

Signed-off-by: Zac Dover <zac.dover@proton.me>

osd/scrub: abort reserving scrub if an operator-initiated scrub is
requested

Handling the case of receiving an operator command while the PG is
scrubbing, but
is waiting for replicas' reservations:

Now that the reservations are queued, the wait may be a very prolonged
one.
Usually - an operator direct scrub command has a priority high enough
to not require waiting for reservations. But in the current
implementation,
it would wait until the running scrub session terminates, and only then
will rerun at that high priority. This is not the intended behavior.

The solution is to abort the existing scrub session, and start the new
one.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: register for 'osd_max_scrubs' config changes

Since https://github.com/ceph/ceph/pull/55340, the
osd_max_scrubs (also) affects the parameters of the
async scrub reserver used by the replicas. Thus,
the code must notice and acknowledge changes to this config.

Fixes: https://tracker.ceph.com/issues/69362
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61142 from Dedsec0098/wip-doc-shrish

doc: Update vstart section in readme.md

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61156 from zdover23/wip-doc-2024-12-20-radosgw-uadk-accel

doc/radosgw: line edit uadk-accel.rst (1st half)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>