]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
3 months agorgw/multisite/datalog: Make add_entry a stackful coroutine
Adam Emerson [Thu, 16 May 2024 02:42:38 +0000 (22:42 -0400)]
rgw/multisite/datalog: Make add_entry a stackful coroutine

Since, outside of testing, it's only called from stackful coroutines,
for now.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: Test semaphores/recovery
Adam Emerson [Sat, 27 Apr 2024 17:20:12 +0000 (13:20 -0400)]
rgw/multisite/datalog: Test semaphores/recovery

Also tests renew_entry, backend, and list.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw/admin: Don't start background tasks in radosgw-admin
Adam C. Emerson [Tue, 1 Apr 2025 03:10:06 +0000 (23:10 -0400)]
rgw/admin: Don't start background tasks in radosgw-admin

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: Orderly shutdown
Adam Emerson [Wed, 1 May 2024 15:41:55 +0000 (11:41 -0400)]
rgw/multisite/datalog: Orderly shutdown

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: Parallelize recovery
Adam Emerson [Mon, 29 Apr 2024 23:49:16 +0000 (19:49 -0400)]
rgw/multisite/datalog: Parallelize recovery

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: Semaphores and Recovery
Adam Emerson [Tue, 2 Apr 2024 19:47:29 +0000 (15:47 -0400)]
rgw/multisite/datalog: Semaphores and Recovery

Increment in add_entry, decrement in renew_entry, and recover on
startup.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: de/serialize BucketShards from/to a string
Adam Emerson [Tue, 2 Apr 2024 17:52:48 +0000 (13:52 -0400)]
rgw/multisite/datalog: de/serialize BucketShards from/to a string

So we can reconstitute bucket shards from listings during recovery.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: add_entry() failures are fatal, actually
Adam Emerson [Sat, 30 Mar 2024 01:34:36 +0000 (21:34 -0400)]
rgw/multisite/datalog: add_entry() failures are fatal, actually

Except in one particular case.

This is necessary to close the data replication write hole, so that if
we fail to write to the semaphore set, the client will receive an
error.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: Remove cls_fifo_legacy
Adam Emerson [Mon, 29 Apr 2024 21:39:59 +0000 (17:39 -0400)]
rgw/multisite/datalog: Remove cls_fifo_legacy

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agotest/rgw_multi: Fix unbound variable in log message
Adam C. Emerson [Mon, 31 Mar 2025 19:32:41 +0000 (15:32 -0400)]
test/rgw_multi: Fix unbound variable in log message

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: C++20 Coroutine Refactor
Adam Emerson [Thu, 21 Nov 2024 22:58:21 +0000 (17:58 -0500)]
rgw/multisite/datalog: C++20 Coroutine Refactor

Rewrite all of `RGWDataChangesLog` and supporting classes to use
non-blocking, C++20 coroutines.

Make interfaces for `optional_yield` and `librados::AioCompletion`.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: Get rid of unneeded set_zero
Adam C. Emerson [Fri, 28 Apr 2023 20:13:29 +0000 (16:13 -0400)]
rgw/multisite/datalog: Get rid of unneeded set_zero

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agorgw/multisite/datalog: Take driver, not handle.
Adam C. Emerson [Wed, 26 Apr 2023 16:12:52 +0000 (12:12 -0400)]
rgw/multisite/datalog: Take driver, not handle.

Since we're plumbing the driver around and it's where our neorados
handle lives.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocommon/async: add spawn_group template for fork-join parallelism
Casey Bodley [Sun, 5 Feb 2023 15:10:34 +0000 (10:10 -0500)]
common/async: add spawn_group template for fork-join parallelism

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 months agorgw: Add neorados functions to RGWObjVersionTracker
Adam Emerson [Sat, 11 May 2024 02:27:25 +0000 (22:27 -0400)]
rgw: Add neorados functions to RGWObjVersionTracker

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agorgw: Add neorados utilities
Adam Emerson [Thu, 16 Nov 2023 00:47:22 +0000 (19:47 -0500)]
rgw: Add neorados utilities

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agocls: Add sem_set class
Adam Emerson [Tue, 5 Mar 2024 14:56:19 +0000 (09:56 -0500)]
cls: Add sem_set class

This stores a collection of key-based semaphores, intended to serve as
a backend to the project to close the datalog write-hole.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agogcc11: Rewrite some chunks of FIFO
Adam Emerson [Thu, 21 Nov 2024 20:46:10 +0000 (15:46 -0500)]
gcc11: Rewrite some chunks of FIFO

Revert this commit when we are at last free of gcc11.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoneorados/cls: Client for FIFO objclass
Adam C. Emerson [Thu, 26 Jan 2023 00:25:24 +0000 (19:25 -0500)]
neorados/cls: Client for FIFO objclass

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agogcc11: Temporary change to neorados CLS log
Adam Emerson [Wed, 20 Nov 2024 01:11:26 +0000 (20:11 -0500)]
gcc11: Temporary change to neorados CLS log

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoneorados/cls: Client for log objclass
Adam C. Emerson [Sat, 21 Jan 2023 06:03:48 +0000 (01:03 -0500)]
neorados/cls: Client for log objclass

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocls/log: C++ namespaces exist
Adam C. Emerson [Tue, 24 Jan 2023 21:35:23 +0000 (16:35 -0500)]
cls/log: C++ namespaces exist

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocls/log: Switch from utime_t to ceph::real_time
Adam C. Emerson [Tue, 24 Jan 2023 08:12:21 +0000 (03:12 -0500)]
cls/log: Switch from utime_t to ceph::real_time

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocls/log: Add non-default constructors to cls_log_entry
Adam C. Emerson [Tue, 24 Jan 2023 06:53:20 +0000 (01:53 -0500)]
cls/log: Add non-default constructors to cls_log_entry

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocls/log: Switch from std::list to std::vector
Adam C. Emerson [Sat, 21 Jan 2023 04:24:28 +0000 (23:24 -0500)]
cls/log: Switch from std::list to std::vector

We should not be using std::list everywhere, and this is an excellent
time to switch.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agogcc11: Pull out exec from cls version
Adam Emerson [Tue, 19 Nov 2024 18:45:26 +0000 (13:45 -0500)]
gcc11: Pull out exec from cls version

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoneorados/cls: Client for version objclass
Adam C. Emerson [Sat, 21 Jan 2023 02:50:50 +0000 (21:50 -0500)]
neorados/cls: Client for version objclass

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agoneorados/cls: Helper function for reading CLS values
Adam C. Emerson [Sat, 21 Jan 2023 01:58:00 +0000 (20:58 -0500)]
neorados/cls: Helper function for reading CLS values

Since reading CLS values in a friendly way requires calling out to RADOS
then decoding the returned structure, make a function to help with that.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocls/version: Add non-default constructor
Adam C. Emerson [Mon, 23 Jan 2023 22:51:02 +0000 (17:51 -0500)]
cls/version: Add non-default constructor

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocls/version: Move obj_version printer to cls_version_types.h
Adam C. Emerson [Mon, 23 Jan 2023 22:28:10 +0000 (17:28 -0500)]
cls/version: Move obj_version printer to cls_version_types.h

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocommon/async: Non-blocking condition variable
Adam C. Emerson [Wed, 31 May 2023 20:36:03 +0000 (16:36 -0400)]
common/async: Non-blocking condition variable

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocommon/async: Use AioCompletion as a completion for coroutines
Adam Emerson [Tue, 14 May 2024 19:09:14 +0000 (15:09 -0400)]
common/async: Use AioCompletion as a completion for coroutines

I forgot I had this already. Just extend it so we can handle things
like coroutines that return an `exception_ptr` rather than an
`error_code`.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agocommon/error_code: Add from_exception
Adam Emerson [Tue, 14 May 2024 19:08:08 +0000 (15:08 -0400)]
common/error_code: Add from_exception

Getting an integer error code out of an exception is something we
often want to do, so put it in one place and do it right.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agocommon/error_code: Default error code is -EIO
Adam Emerson [Tue, 14 May 2024 19:07:29 +0000 (15:07 -0400)]
common/error_code: Default error code is -EIO

-EDOM never really made that much sense and this is more coherent with
the rest of Ceph.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agocommon/async: Add async_{dispatch,post,defer}
Adam C. Emerson [Tue, 23 May 2023 00:15:05 +0000 (20:15 -0400)]
common/async: Add async_{dispatch,post,defer}

These functions let you run functions on arbitrary executors,
providing a convenient way to mix I/O less critical sections on a
strand in a coroutine.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agoneorados: `RAODS::notify` returns reply map and missed set
Adam Emerson [Fri, 15 Sep 2023 21:46:41 +0000 (17:46 -0400)]
neorados: `RAODS::notify` returns reply map and missed set

So the format doesn't spill out of the API boundary.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoneorados: Add Asio-idiomatic watch support
Adam Emerson [Fri, 15 Sep 2023 20:08:51 +0000 (16:08 -0400)]
neorados: Add Asio-idiomatic watch support

In addition to the librados-style callback function, add a variant of
`RADOS::watch` and a `next_notification` function that retrieves the
next notification.

Rationale:

The callback creates an inversion of control that was more noticeable
with coroutine heavy workflows. You could co_spawn and detach, but
even then there isn't an obvious way to report errors.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoneorados: Cancel ReadOp/WriteOp execute
Adam Emerson [Wed, 1 May 2024 05:18:02 +0000 (01:18 -0400)]
neorados: Cancel ReadOp/WriteOp execute

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoneorados: No move-only lambdas for async_initiate
Adam Emerson [Tue, 7 May 2024 22:10:52 +0000 (18:10 -0400)]
neorados: No move-only lambdas for async_initiate

Required to get things working under Clang.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoinclude/types: Add printing of unordered_map
Adam Emerson [Fri, 19 Apr 2024 12:23:17 +0000 (08:23 -0400)]
include/types: Add printing of unordered_map

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoinclude/types: Add `operator <<` for `std::span`
Adam C. Emerson [Fri, 16 Jun 2023 15:58:37 +0000 (11:58 -0400)]
include/types: Add `operator <<` for `std::span`

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
3 months agocommon/cohort_lru: Include needed standard headers
Adam Emerson [Wed, 8 May 2024 21:13:39 +0000 (17:13 -0400)]
common/cohort_lru: Include needed standard headers

So we don't get compilation errors as what other things include shifts around.

Signed-off-by: Adam Emerson <aemerson@redhat.com>
3 months agoMerge pull request #60477 from VVoidV/lbr-ETAG-consistent-with-AWS
Casey Bodley [Tue, 1 Apr 2025 14:59:27 +0000 (10:59 -0400)]
Merge pull request #60477 from VVoidV/lbr-ETAG-consistent-with-AWS

<rgw> Ensure the ETag format is consistent with AWS S3 API

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 months agoMerge pull request #60611 from Matan-B/wip-matanb-crimson-max-nr
Matan Breizman [Tue, 1 Apr 2025 13:26:48 +0000 (16:26 +0300)]
Merge pull request #60611 from Matan-B/wip-matanb-crimson-max-nr

ceph.spec.in,debian/rules: aio-max-nr for Crimson

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Kefu Chai <kefu.chai@scylladb.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 months agoMerge PR #62483 into main
Venky Shankar [Tue, 1 Apr 2025 13:07:44 +0000 (18:37 +0530)]
Merge PR #62483 into main

* refs/pull/62483/head:
libcephfs_proxy: workaround for delays and hangs in the proxy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
3 months agoMerge pull request #60786 from er0k/main
Kefu Chai [Tue, 1 Apr 2025 09:07:51 +0000 (17:07 +0800)]
Merge pull request #60786 from er0k/main

mgr/snap_schedule: fix typo in timestamp format exception

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
3 months agoMerge pull request #62557 from ronen-fr/wip-rf-preemptionpg
Ronen Friedman [Tue, 1 Apr 2025 08:46:10 +0000 (11:46 +0300)]
Merge pull request #62557 from ronen-fr/wip-rf-preemptionpg

osd/scrub: improve PgScrubber testability

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 months agoMerge pull request #61362 from rhcs-dashboard/link-rgw-user-with-account
Nizamudeen A [Tue, 1 Apr 2025 06:35:29 +0000 (12:05 +0530)]
Merge pull request #61362 from rhcs-dashboard/link-rgw-user-with-account

mgr/dashboard: Link rgw user with account

3 months agoMerge pull request #62566 from tchaikov/neorados-googletest
Kefu Chai [Tue, 1 Apr 2025 05:59:54 +0000 (13:59 +0800)]
Merge pull request #62566 from tchaikov/neorados-googletest

neorados/test, googletest: adapt to googletest and bump up googletest submodule

Reviewed-by: Adam Emerson <aemerson@redhat.com>
3 months agoMerge pull request #62498 from ronen-fr/wip-rf-with-array
Ronen Friedman [Tue, 1 Apr 2025 04:40:21 +0000 (07:40 +0300)]
Merge pull request #62498 from ronen-fr/wip-rf-with-array

common/Formatter: introduce helpers for formatting container types

Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
3 months agoosd/scrub: decoupling the FSM from the scrubber 62557/head
Ronen Friedman [Fri, 28 Mar 2025 16:16:52 +0000 (11:16 -0500)]
osd/scrub: decoupling the FSM from the scrubber

The scrubber FSM has always been accessing the PgScrubber
object via an abstract interface (ScrubMachineListener).
But the reverse was not true: the PgScrubber owned a unique_ptr
directly to an instance of Scrubber::ScrubMachine.

This PR introduces a new interface, ScrubFsmIf, that is
implemented by the ScrubMachine. This simplifies
unit-testing the PgScrubber, as we can now inject a mock
implementation of the FSM interface.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
3 months agosd/scrub: preemption_data_t needs not keep a PG ref
Ronen Friedman [Thu, 27 Mar 2025 16:36:44 +0000 (11:36 -0500)]
sd/scrub: preemption_data_t needs not keep a PG ref

Removing the PG* from PgScrubber::preemption_data_t.
Mainly - to simplify the creation of unit-tests for the
PgScrubber class.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
3 months agoMerge pull request #62374 from tchaikov/dmclock-variant
Kefu Chai [Tue, 1 Apr 2025 02:44:20 +0000 (10:44 +0800)]
Merge pull request #62374 from tchaikov/dmclock-variant

dmclock: migrate from boost::variant to std::variant

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
3 months agoMerge pull request #62564 from tchaikov/cmake-fmt
Kefu Chai [Tue, 1 Apr 2025 01:08:56 +0000 (09:08 +0800)]
Merge pull request #62564 from tchaikov/cmake-fmt

cmake: Allow building with fmt <= 11.1.4

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>
3 months agoMerge pull request #62565 from tchaikov/cmake-pre-build
Kefu Chai [Tue, 1 Apr 2025 01:08:16 +0000 (09:08 +0800)]
Merge pull request #62565 from tchaikov/cmake-pre-build

cmake: do not pass PRE_BUILD to add_custom_command(OUTPUT)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 months agoMerge pull request #62525 from phlogistonjohn/jjm-sidecar-redeploy
Adam King [Mon, 31 Mar 2025 23:35:42 +0000 (19:35 -0400)]
Merge pull request #62525 from phlogistonjohn/jjm-sidecar-redeploy

cephadm: stop sidecar systemd units when restarting main units

Reviewed-by: Adam King <adking@redhat.com>
3 months agoMerge pull request #62501 from Matan-B/wip-matanb-crimson-cephadm-aio-max-nr
Adam King [Mon, 31 Mar 2025 23:34:49 +0000 (19:34 -0400)]
Merge pull request #62501 from Matan-B/wip-matanb-crimson-cephadm-aio-max-nr

cephadm/cephadmlib/daemons/ceph.py: increase aio-max-nr

Reviewed-by: Adam King <adking@redhat.com>
3 months agoMerge pull request #62491 from Kushal-deb/rgw-port-summary
Adam King [Mon, 31 Mar 2025 23:33:32 +0000 (19:33 -0400)]
Merge pull request #62491 from Kushal-deb/rgw-port-summary

mgr/cephadm: 'ceph orch ls' to display all the ports used by RGW service

Reviewed-by: Adam King <adking@redhat.com>
3 months agorgw: add release note for newly-quoted ETag values 60477/head
Casey Bodley [Wed, 26 Mar 2025 16:22:07 +0000 (12:22 -0400)]
rgw: add release note for newly-quoted ETag values

Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 months ago<rgw> Ensure the ETag format is consistent with AWS S3 API
liubingrun [Thu, 24 Oct 2024 14:23:09 +0000 (10:23 -0400)]
<rgw> Ensure the ETag format is consistent with AWS S3 API

AWS S3 API quotes ETAG everywhere. We still missed a few places.

checked with following cmd.
```
$ rg -i "dump.*\"ETag\"" ceph
./src/rgw/rgw_rest_s3.cc
1863:        s->formatter->dump_format("ETag", "\"%s\"", iter->meta.etag.c_str());
1955:      s->formatter->dump_format("ETag", "\"%s\"", iter->meta.etag.c_str());
2030:        s->formatter->dump_format("ETag", "\"%s\"", iter->meta.etag.c_str());
2099:      s->formatter->dump_format("ETag", "\"%s\"", iter->meta.etag.c_str());
2785:      s->formatter->dump_format("ETag", "\"%s\"", etag.c_str());
3422:    s->formatter->dump_string("ETag", etag);
3725:      s->formatter->dump_format("ETag", "\"%s\"",etag.c_str());
4228:    s->formatter->dump_string("ETag", etag);
4298:      s->formatter->dump_format("ETag", "\"%s\"", part->get_etag().c_str());

./src/rgw/rgw_rest.cc
421:    return dump_header(s, "etag", etag);
423:    return dump_header_quoted(s, "ETag", etag);

./src/rgw/driver/rados/rgw_sync_module_es_rest.cc
347:      s->formatter->dump_format("ETag", "\"%s\"", e.meta.etag.c_str());

./src/rgw/rgw_admin.cc
8612:        handled = dump_string("etag", bl, formatter.get());
```

Signed-off-by: liubingrun <liubr1@chinatelecom.cn>
3 months agoMerge pull request #61684 from jzhu116-bloomberg/wip-69738
Casey Bodley [Mon, 31 Mar 2025 20:23:24 +0000 (16:23 -0400)]
Merge pull request #61684 from jzhu116-bloomberg/wip-69738

rgw: cleanup orphaned bucket entry in <user>.buckets OMAP

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 months agoMerge pull request #62235 from cbodley/wip-70399
Casey Bodley [Mon, 31 Mar 2025 20:22:55 +0000 (16:22 -0400)]
Merge pull request #62235 from cbodley/wip-70399

cls/rgw: non-versioned listings skip past version suffix

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
3 months agoMerge pull request #62286 from aodinokov/swift-acl-tests-fix
Casey Bodley [Mon, 31 Mar 2025 20:22:08 +0000 (16:22 -0400)]
Merge pull request #62286 from aodinokov/swift-acl-tests-fix

rgw: fixing tempest ObjectTestACLs and ObjectACLsNegativeTest cases

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 months agoMerge pull request #61878 from linuxbox2/wip-cksum-badalg
Casey Bodley [Mon, 31 Mar 2025 19:40:46 +0000 (15:40 -0400)]
Merge pull request #61878 from linuxbox2/wip-cksum-badalg

rgw: cksum:  implement support for new CRC64NVME checksum algorithm and related fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 months agoMerge pull request #62440 from phlogistonjohn/jjm-logging-uri-scheme
Adam King [Mon, 31 Mar 2025 19:18:28 +0000 (15:18 -0400)]
Merge pull request #62440 from phlogistonjohn/jjm-logging-uri-scheme

mgr/cephadm: do not log unexpected uri scheme at warning level

Reviewed-by: Adam King <adking@redhat.com>
3 months agoMerge pull request #62431 from rkachach/fix_making_mgmt_gw_oauth2_dep
Adam King [Mon, 31 Mar 2025 19:17:35 +0000 (15:17 -0400)]
Merge pull request #62431 from rkachach/fix_making_mgmt_gw_oauth2_dep

mgr/cephadm: making mgmt-gateway an oauth2-proxy dependency

Reviewed-by: Adam King <adking@redhat.com>
3 months agoMerge pull request #62558 from cbodley/wip-70712
Casey Bodley [Mon, 31 Mar 2025 18:02:24 +0000 (14:02 -0400)]
Merge pull request #62558 from cbodley/wip-70712

qa/rgw: fix ignorelist entry for 'HEALTH_WARN 1 pool(s) full'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
3 months agoosd: use with_array_sections() 62498/head
Ronen Friedman [Tue, 25 Mar 2025 14:41:23 +0000 (09:41 -0500)]
osd: use with_array_sections()

... in a couple of functions, as a demonstration of its usage.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
3 months agocommon/Formatter: introduce helpers for formatting container types.
Ronen Friedman [Tue, 18 Mar 2025 09:03:39 +0000 (04:03 -0500)]
common/Formatter: introduce helpers for formatting container types.

Specifically:

- with_array_sections() creates a Formmater::ArraySection, then
  formats every element in the container using the provided
  callable.
  Special handling for std::map: the callable is called with
  both the key and the value.

- with_obj_array_section() dumps containers of objects:
  First - an ArraySection is created; then - for each object
  in the container, the object is formatted within an ObjectSection,
  using the provided callable.
  If the container is an std::map, the callable is called with both
  the key and the object itself.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
3 months agoMerge pull request #62567 from tchaikov/qa-remove-unused-global
Kefu Chai [Mon, 31 Mar 2025 15:04:18 +0000 (23:04 +0800)]
Merge pull request #62567 from tchaikov/qa-remove-unused-global

cephfs-top, qa: Remove unnecessary global statements in tests

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 months agorgw:cksum: implement crc64nvme and combined 32- and 64-bit CRCs 61878/head
Matt Benjamin [Fri, 21 Feb 2025 19:43:53 +0000 (14:43 -0500)]
rgw:cksum: implement crc64nvme and combined 32- and 64-bit CRCs

* internally compensate for at-rest byteswapped crc64 representation (e.g., before combine step)
* generalize Cksum crc api for 32bit and 64bit crcs, other cleanup
* prototype abstract cksum::Combiner workflow
* add support for forward and backward handling of composite vs full object checksums
  by marking the composites and the update flag day
* clean up checksum formatting and checksum type reporting
* add unit tests for Combiner interface
* validate and track requested checksum type (i.e., composite or full), plus
  unit test fixture for combinations of full matrix of cksum types
* doh.  GET/HEAD checksums are in headers
* add crcany license to COPYING
* return ChecksumType as header in GET/HEAD

    Found by Casey in review

* avoid fmt of char* null when no checksum header supplied

    A cksum_hdr_t(nullptr, nullptr) results in this case, and is intended,
    but its components obviously can't be presented to std::format unless
    cast to a safe pointer type.

* fail checksum mismatch with BadDigest

    When uploading an S3 object with an invalid checksum, the return code
    should be BadDigest to mirror more closely the AWS S3 implementation.
    See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

Fixes: https://tracker.ceph.com/issues/70614
    Reported by Alex Wojno <awojno@bloomberg.net>

* fix comparison and display of composite checksums

A string comparision bounds error and a bitmask comparison
    error are fixed.

* fix build on centos9

On gcc(?13?) we can't declare rgw::cksum::FLAG_NONE
    and also rgw::cksum::Cksum::FlAG_NONE.

    SAD!!

* include <variant> invariantly

* fix checksum type return from complete-multipart

    This one is in the XML response

* aieee, don't leak Combiners

   Use unique_ptr to polymorphic type...correctly.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
3 months agoMerge pull request #62572 from anthonyeleven/expand-ec-table
Zac Dover [Sun, 30 Mar 2025 21:41:31 +0000 (07:41 +1000)]
Merge pull request #62572 from anthonyeleven/expand-ec-table

doc/rados/operations: Improve erasure-code.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>
3 months agoMerge pull request #62562 from tchaikov/librbd-atomic
Ilya Dryomov [Sun, 30 Mar 2025 21:27:36 +0000 (23:27 +0200)]
Merge pull request #62562 from tchaikov/librbd-atomic

librbd: replace deprecated atomic_store with std::atomic<shared_ptr>

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
3 months agodoc/rados/operations: Improve erasure-code.rst 62572/head
Anthony D'Atri [Sun, 30 Mar 2025 19:47:14 +0000 (15:47 -0400)]
doc/rados/operations: Improve erasure-code.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
3 months agocephfs-top: Removes unused `global` statements 62567/head
Kefu Chai [Sun, 30 Mar 2025 03:59:12 +0000 (11:59 +0800)]
cephfs-top: Removes unused `global` statements

Recent flake8 runs were failing with:
```
py3: flake8==7.2.0,mccabe==0.7.0,pip==25.0.1,pycodestyle==2.13.0,pyflakes==3.3.0,setuptools==75.8.0,wheel==0.45.1
py3: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/cephfs/top> flake8 --ignore=W503 --max-line-length=100 cephfs-top
cephfs-top:344:9: F824 `global fs_list` is unused: name is never assigned in scope
cephfs-top:466:13: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:872:9: F824 `global metrics_dict` is unused: name is never assigned in scope
cephfs-top:872:9: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:911:9: F824 `global fs_list` is unused: name is never assigned in scope
cephfs-top:981:9: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:1126:13: F824 `global current_states` is unused: name is never assigned in scope
py3: exit 1 (0.77 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/cephfs/top> flake8 --ignore=W503 --max-line-length=100 cephfs-top pid=2309605
  py3: FAIL code 1 (8.15=setup[7.38]+cmd[0.77] seconds)
  evaluation failed :( (8.24 seconds)
```

Since these variables are only being referenced and not assigned within
their scopes, the `global` declarations are unnecessary and can be
safely removed. This change:

- Removes all flagged `global` statements
- Fixes the failing flake8 checks in the CI pipeline
- Maintains the original code behavior as variable references still work without the `global` keyword

The `global` keyword is only needed when assigning to global variables
within a function scope, not when simply referencing them.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agoqa: Remove unnecessary global statements in tests
Kefu Chai [Sun, 30 Mar 2025 03:48:28 +0000 (11:48 +0800)]
qa: Remove unnecessary global statements in tests

Removes unused `global` statements from Python test files to fix flake8
F824 errors.

Recent flake8 runs were failing with:

```
./tasks/radosgw_admin.py:330:5: F824 `global log` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:99:5: F824 `global incompat_paths` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:164:5: F824 `global backward_compat` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:165:5: F824 `global fast_shouldnt_skip` is unused: name is never assigned in scope
```

Since these variables are only being referenced and not assigned within
their scopes, the `global` declarations are unnecessary and can be
safely removed. This change:

- Removes all flagged `global` statements
- Fixes the failing flake8 checks in the CI pipeline
- Maintains the original code behavior as variable references still work
  without the `global` keyword

The `global` keyword is only needed when assigning to global variables
within a function scope, not when simply referencing them.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agogoogletest: pick up change to silence CMake warnings 62566/head
Kefu Chai [Sun, 30 Mar 2025 02:31:21 +0000 (10:31 +0800)]
googletest: pick up change to silence CMake warnings

bump up googletest submodule to the latest stable release, in order to
silence the warning from CMake:

```
CMake Deprecation Warning at src/googletest/CMakeLists.txt:4 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

CMake Deprecation Warning at src/googletest/googlemock/CMakeLists.txt:45 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

CMake Deprecation Warning at src/googletest/googletest/CMakeLists.txt:56 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.
```

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agoneorados/test: use [[maybe_unused]] instead of GTEST_ATTRIBUTE_UNUSED_
Kefu Chai [Sun, 30 Mar 2025 03:14:01 +0000 (11:14 +0800)]
neorados/test: use [[maybe_unused]] instead of GTEST_ATTRIBUTE_UNUSED_

This change:

- Aligns with GoogleTest's own evolution, which now uses
  `[[maybe_unused]]` internally (via
  `GTEST_INTERNAL_ATTRIBUTE_MAYBE_UNUSED`) as of
  https://github.com/google/googletest/commit/5a37b517
- Leverages C++20 features already available in Ceph
- Removes dependency on GoogleTest internal macros
- Enables updating the GoogleTest submodule to silence CMake warnings

Using the standard C++ attribute provides better compatibility and
future-proofing than relying on GoogleTest's internal implementation
details.

This change also allows us to bump up googletest submodule to silence a
CMake warning.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agoneorados/test: Simplify copy prevention by using GtestNonCopyable
Kefu Chai [Sun, 30 Mar 2025 02:56:05 +0000 (10:56 +0800)]
neorados/test: Simplify copy prevention by using GtestNonCopyable

Updates the test framework to use GoogleTest's simplified copy
prevention mechanism by inheriting from
`::testing::internal::GtestNonCopyable` instead of custom macros.

This change:
- Replaces deprecated GTEST_DISALLOW_* macros that were removed in
  GoogleTest (https://github.com/google/googletest/commit/bf66935e07825318ae519675d73d0f3e313b3ec6)
- Aligns with GoogleTest's current approach for preventing test suite
  copying, See
  https://github.com/google/googletest/commit/93f08be653c36ddc6943e9513fc14c7292b4d007
- Enables updating the GoogleTest submodule to silence CMake warnings
- Improves compatibility with recent distro-packaged GoogleTest versions
- add the minimal required GTest version when finding this package,
  this change should have no impact to our packaging. as both deb
  and rpm packagings are using bundled googletest submodule. and
  googletest 1.14 has been included by recent stable releases of
  popular distros, like fedora 41 and ubuntu 24.04.
While `GtestNonCopyable` is technically an internal helper, we already
use other helpers from the `::testing::internal` namespace, so this
change maintains consistency with our existing approach.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agocmake: do not pass PRE_BUILD to add_custom_command(OUTPUT) 62565/head
Kefu Chai [Sun, 30 Mar 2025 02:10:28 +0000 (10:10 +0800)]
cmake: do not pass PRE_BUILD to add_custom_command(OUTPUT)

Removes the invalid PRE_BUILD argument from add_custom_command(OUTPUT)
calls that was triggering CMake warnings. The PRE_BUILD option is only
valid for target-based custom commands and specifically with Visual
Studio generators.

Additionally sets the CMP0175 policy to NEW to ensure CMake fails on
invalid arguments to add_custom_command(), helping catch similar issues
earlier in the development process.

This change:

- Removes unnecessary PRE_BUILD argument from OUTPUT-based custom commands
- Sets CMP0175 policy to enforce validation of add_custom_command() arguments
- Resolves CMake warning in src/common/options/CMakeLists.txt

Reference:
https://cmake.org/cmake/help/latest/command/add_custom_command.html#build-events

The CMake warning looks like:

```
CMake Warning (dev) at src/common/options/CMakeLists.txt:34 (add_custom_command):
  The following keywords are not supported when using
  add_custom_command(OUTPUT): PRE_BUILD.

  Policy CMP0175 is not set: add_custom_command() rejects invalid arguments.
  Run "cmake --help-policy CMP0175" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.
Call Stack (most recent call first):
  src/common/options/CMakeLists.txt:85 (add_options)
This warning is for project developers.  Use -Wno-dev to suppress it.
```

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agocmake: Allow building with fmt <= 11.1.4 62564/head
Kefu Chai [Sun, 30 Mar 2025 01:39:53 +0000 (09:39 +0800)]
cmake: Allow building with fmt <= 11.1.4

Testing confirms successful builds with fmt 11.1.4, which is currently
packaged in Fedora 42. This change relaxes the fmt version constraints
to improve compatibility with Fedora 42 and other distributions shipping
this version of fmt-devel.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agoMerge pull request #62550 from tchaikov/fmt-formattable
Kefu Chai [Sun, 30 Mar 2025 01:36:22 +0000 (09:36 +0800)]
Merge pull request #62550 from tchaikov/fmt-formattable

src: migrate to fmt::formattable for format support

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
3 months agoMerge pull request #62509 from rzarzynski/wip-bug-69724
Yuri Weinstein [Sat, 29 Mar 2025 14:08:25 +0000 (07:08 -0700)]
Merge pull request #62509 from rzarzynski/wip-bug-69724

osd: fix the tiering agent's shutdown wrt the OSD fast shutdown

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
3 months agosrc: migrate to fmt::formattable for format support 62550/head
Kefu Chai [Wed, 26 Mar 2025 03:43:43 +0000 (11:43 +0800)]
src: migrate to fmt::formattable for format support

Replace custom has_formatter concept with fmtlib's fmt::formattable

- Deprecate usage of fmt::has_formatter
- Resolve compiler warnings with recent fmtlib versions

The custom concept of has_formatter is no longer necessary with
recent updates to the fmtlib library, as fmt::formattable was introduced
in fmtlib v11.0.0. By switching to fmt::formattable, we simplify our
code and stop using the deprecated `fmt::has_formatter` template.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agolibrbd: replace deprecated atomic_store with std::atomic<shared_ptr> 62562/head
Kefu Chai [Sat, 29 Mar 2025 13:10:05 +0000 (21:10 +0800)]
librbd: replace deprecated atomic_store with std::atomic<shared_ptr>

Update shared pointer atomic operations to use C++20's std::atomic<std::shared_ptr<T>>
instead of the deprecated atomic_store functions. This change addresses deprecation
warnings from GCC-15's libstdc++ where atomic shared pointer operations outside the
std::atomic class are being phased out:
```
/home/kefu/dev/ceph/src/librbd/ImageCtx.cc:1010:5: warning: 'atomic_store<neorados::IOContext>' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Wdeprecated-declarations]
 1010 |     atomic_store(&data_io_context, ctx);
      |     ^
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_atomic.h:181:5: note: 'atomic_store<neorados::IOContext>' has been explicitly marked deprecated here
  181 |     _GLIBCXX20_DEPRECATED_SUGGEST("std::atomic<std::shared_ptr<T>>")
      |     ^
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/x86_64-redhat-linux/bits/c++config.h:2055:45: note: expanded from macro '_GLIBCXX20_DEPRECATED_SUGGEST'
 2055 | # define _GLIBCXX20_DEPRECATED_SUGGEST(ALT) _GLIBCXX_DEPRECATED_SUGGEST(ALT)
      |                                             ^
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/x86_64-redhat-linux/bits/c++config.h:2023:19: note: expanded from macro '_GLIBCXX_DEPRECATED_SUGGEST'
 2023 |   __attribute__ ((__deprecated__ ("use '" ALT "' instead")))
      |                   ^
```

The implementation now uses the standard-compliant approach that's recommended in
the compiler warnings, while maintaining backward compatibility with older compilers
by conditionally selecting the appropriate implementation.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agoMerge pull request #62474 from tchaikov/openssl-engine
Kefu Chai [Sat, 29 Mar 2025 06:32:03 +0000 (14:32 +0800)]
Merge pull request #62474 from tchaikov/openssl-engine

common: disable OpenSSL engine support if it is disabled

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
3 months agoMerge pull request #62141 from baum/nvmeof_cluster_alloc
baum [Fri, 28 Mar 2025 22:52:34 +0000 (01:52 +0300)]
Merge pull request #62141 from baum/nvmeof_cluster_alloc

NvmeofServiceSpec: ceph connection allocation strategies

3 months agoMerge pull request #62480 from tchaikov/erasure-code-free
Yuri Weinstein [Fri, 28 Mar 2025 22:27:55 +0000 (15:27 -0700)]
Merge pull request #62480 from tchaikov/erasure-code-free

erasure-code: use new/delete to alloc/free coefficients array

Reviewed-by: Alex Ainscow <aainscow@redhat.com>
3 months agoqa/rgw: fix ignorelist entry for 'HEALTH_WARN 1 pool(s) full' 62558/head
Casey Bodley [Fri, 28 Mar 2025 17:38:23 +0000 (13:38 -0400)]
qa/rgw: fix ignorelist entry for 'HEALTH_WARN 1 pool(s) full'

Fixes: https://tracker.ceph.com/issues/70712
Signed-off-by: Casey Bodley <cbodley@redhat.com>
3 months agoMerge pull request #62174 from ifed01/wip-ifed-fix-bluefs-reserved2
Igor Fedotov [Fri, 28 Mar 2025 15:45:45 +0000 (18:45 +0300)]
Merge pull request #62174 from ifed01/wip-ifed-fix-bluefs-reserved2

os/bluestore: use block size (4K) as minimal allocation unit for dedicated DB/WAL volumes

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>
3 months agoMerge pull request #62251 from ifed01/wip-ifed-print-discard-threads
Igor Fedotov [Fri, 28 Mar 2025 13:54:50 +0000 (16:54 +0300)]
Merge pull request #62251 from ifed01/wip-ifed-print-discard-threads

blk/kernel: New perf counter to view amount of discard threads.

Reviewed-by: Yite Gu <yitegu0@gmail.com>
3 months agoMerge pull request #62546 from tchaikov/crimson-rados-qa
Kefu Chai [Fri, 28 Mar 2025 12:27:25 +0000 (20:27 +0800)]
Merge pull request #62546 from tchaikov/crimson-rados-qa

qa: add missing .qa links for crimson-rados tests

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 months agoMerge pull request #62537 from zdover23/wip-doc-2025-03-27-mon_warn_pg_not_scrubbed_r...
Zac Dover [Fri, 28 Mar 2025 12:02:59 +0000 (22:02 +1000)]
Merge pull request #62537 from zdover23/wip-doc-2025-03-27-mon_warn_pg_not_scrubbed_ratio-guidance

src/common: add guidance for mon_warn_pg_not_scrubbed

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
3 months agoMerge pull request #62471 from YiteGu/fix-invalid-iterator-usge-after-erase
Adam Kupczyk [Fri, 28 Mar 2025 11:54:08 +0000 (12:54 +0100)]
Merge pull request #62471 from YiteGu/fix-invalid-iterator-usge-after-erase

blk/kerneldevice: fix invalid iterator usage after erase in discard_q…

3 months agoMerge pull request #62273 from aclamk/wip-aclamk-osd-superblock-corruption-resilient
Adam Kupczyk [Fri, 28 Mar 2025 11:53:34 +0000 (12:53 +0100)]
Merge pull request #62273 from aclamk/wip-aclamk-osd-superblock-corruption-resilient

osd/OSD.cc: make osd_superblock recovery more robust

3 months agomgr/dashboard: link user to rgw account & add root account user functionality 61362/head
Naman Munet [Tue, 14 Jan 2025 08:21:57 +0000 (13:51 +0530)]
mgr/dashboard: link user to rgw account & add root account user functionality

Fixes: https://tracker.ceph.com/issues/69529
Signed-off-by: Naman Munet <naman.munet@ibm.com>
3 months agoMerge pull request #62476 from rhcs-dashboard/fix-70646-main
Aashish Sharma [Fri, 28 Mar 2025 06:17:52 +0000 (11:47 +0530)]
Merge pull request #62476 from rhcs-dashboard/fix-70646-main

mgr/dashboard: fix bucket form encryption checkbox

Reviewed-by: Nizamudeen A <nia@redhat.com>
3 months agoNvmeofServiceSpec: ceph connection allocation strategies 62141/head
Alexander Indenbaum [Wed, 5 Mar 2025 17:32:56 +0000 (19:32 +0200)]
NvmeofServiceSpec: ceph connection allocation strategies

Co-authored-by: Adam King <47704447+adk3798@users.noreply.github.com>
Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>
3 months agoqa: add missing .qa links for crimson-rados tests 62546/head
Kefu Chai [Fri, 28 Mar 2025 03:17:44 +0000 (11:17 +0800)]
qa: add missing .qa links for crimson-rados tests

follow the suggestion by src/script/verify-qa to address the
failure of qa-symlink.yml workflow. without this change, the
workflow consistently fails.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
3 months agocommon: disable OpenSSL engine support if it is disabled 62474/head
Kefu Chai [Tue, 25 Mar 2025 04:03:30 +0000 (12:03 +0800)]
common: disable OpenSSL engine support if it is disabled

OpenSSL 3.0 documentation recommends moving from the ENGINE API to the
Providers API. Recent distributions may compile OpenSSL without engine
support by default, necessitating more flexible configuration handling.

So, in this change:

- Add a CMake option `WITH_OPENSSL_ENGINE` to explicitly control engine support
- Respect `openssl_engine_opts` when engine support is enabled
- Provide clear error messaging when engine options are set but support is disabled

See also:
- OpenSSL 3.0 documentation:
https://wiki.openssl.org/index.php/OpenSSL_3.0#Engines_and_.22METHOD.22_APIs

Fixes: https://tracker.ceph.com/issues/68059
Signed-off-by: Kefu Chai <tchaikov@gmail.com>