Casey Bodley [Wed, 8 Jan 2025 19:52:03 +0000 (14:52 -0500)]
rgw/s3: CreateBucket extension for layout type and shard count
extend s3's CreateBucketConfiguration [1] with a custom BucketIndex element
that can override rgw's default bucket index type and shard count
to create an indexless bucket:
<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<BucketIndex>
<Type>Indexless</Type>
</BucketIndex>
</CreateBucketConfiguration>
to create a normal pre-sharded bucket:
<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<BucketIndex>
<Type>Normal</Type>
<NumShards>1023</NumShards>
</BucketIndex>
</CreateBucketConfiguration>
Zac Dover [Tue, 25 Feb 2025 04:57:11 +0000 (14:57 +1000)]
doc/releases: correct squid release order
Put the releases of Squid in descending order. This change alters the
order of the Squid releases so that it is the same as the order of the
other Ceph releases.
1.Added option kmip in KMS dropdown in configuration page RGW
2.Made API dynamic for setting form data for Vault, kmip both for (S3 and
KMS)
3.Both Getdata and setdata API for RGW config page is modified
Casey Bodley [Mon, 24 Feb 2025 21:50:39 +0000 (16:50 -0500)]
doc/rgw: clarify path-style vs virtual-hosted-style access
instead of referring to "vhost-style", copy the "path-style" and
"virtual-hosted-style" language from https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html
expand the FQDN acronym to avoid potential confusion
"The second method is deprecated by AWS" had incorrectly referred to the
vhost-style method - clarify that it refers to path-style access
Patrick Donnelly [Mon, 24 Feb 2025 21:41:00 +0000 (16:41 -0500)]
Merge PR #61697 into main
* refs/pull/61697/head:
osdc/ObjectCacher: include cleanup
osdc/Objecter: include cleanup
osdc/Journaler: include cleanup
osdc/Striper: add missing includes
osdc/Filer: include cleanup
crush/CrushWrapper: un-inline methods to fix linker errors
client/Client: add missing include and forward declaration
John Mulligan [Thu, 13 Feb 2025 21:00:27 +0000 (16:00 -0500)]
install-deps.sh: enable crb repository for centos 10 stream
The centos 10 stream distribution is like centos 9 and needs the crb
repository enabled to even think about building ceph. Note that I made
it a separate block in case more stuff needs to be added for it later.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Ronen Friedman [Thu, 20 Feb 2025 16:15:44 +0000 (10:15 -0600)]
qa/standalone/scrub: retry in TEST_abort_periodic_for_operator()
The scenario created in the test requires two PGs that share the same
Primary, and at least one more member of their active sets.
The PR handles the (unlikely) case where the first PG tried does not
have a counterpart that matches the requirements.
A minor related fix in scrub-helpers.sh: using 'printf' to
return a value, instead of an out (ref) parameter.
Max Kellermann [Thu, 9 Jan 2025 13:49:22 +0000 (14:49 +0100)]
crush/CrushWrapper: un-inline methods to fix linker errors
The `crush_objs` library is not part of the `libceph-common` ABI, so
applications linking with `libceph-common` may fail to build because
the linker cannot find `crush_destroy()`.
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
```
Building C object src/libcephfs_proxy/CMakeFiles/cephfs_proxy.dir/proxy_link.c.o
/home/./ceph/src/libcephfs_proxy/proxy_link.c:198:10: warning: arithmetic on a pointer to void is a GNU extension [-Wgnu-pointer-arith]
198 | buffer += len;
| ~~~~~~ ^
/home/./ceph/src/libcephfs_proxy/proxy_link.c:232:18: warning: arithmetic on a pointer to void is a GNU extension [-Wgnu-pointer-arith]
232 | iov->iov_base += len;
| ~~~~~~~~~~~~~ ^
/home/./ceph/src/libcephfs_proxy/proxy_link.c:268:18: warning: arithmetic on a pointer to void is a GNU extension [-Wgnu-pointer-arith]
268 | iov->iov_base += len;
| ~~~~~~~~~~~~~ ^
/home/./ceph/src/libcephfs_proxy/proxy_link.c:328:17: warning: arithmetic on a pointer to void is a GNU extension [-Wgnu-pointer-arith]
328 | iov->iov_base += sizeof(proxy_link_req_t);
| ~~~~~~~~~~~~~ ^
/home/./ceph/src/libcephfs_proxy/proxy_link.c:391:17: warning: arithmetic on a pointer to void is a GNU extension [-Wgnu-pointer-arith]
391 | iov->iov_base += sizeof(proxy_link_ans_t);
| ~~~~~~~~~~~~~ ^
5 warnings generated.
```
```
[460/1945] Building C object src/erasure-code/jerasure/CMakeFiles/jerasure_objs.dir/jerasure/src/jerasure.c.o
ceph/src/erasure-code/jerasure/jerasure/src/jerasure.c:722:7: warning: variable 'ddf' set but not used [-Wunused-but-set-variable]
722 | int ddf, cdf;
| ^
ceph/src/erasure-code/jerasure/jerasure/src/jerasure.c:722:12: warning: variable 'cdf' set but not used [-Wunused-but-set-variable]
722 | int ddf, cdf;
| ^
ceph/src/erasure-code/jerasure/jerasure/src/jerasure.c:777:7: warning: variable 'ddf' set but not used [-Wunused-but-set-variable]
777 | int ddf, cdf;
| ^
ceph/src/erasure-code/jerasure/jerasure/src/jerasure.c:777:12: warning: variable 'cdf' set but not used [-Wunused-but-set-variable]
777 | int ddf, cdf;
| ^
4 warnings generated.
```
Matan Breizman [Mon, 17 Feb 2025 07:39:19 +0000 (07:39 +0000)]
src/CMakeLists.txt: add -Wno-vla-cxx-extension to clang
The following warning of:
```
error: variable length arrays in C++ are a Clang extension
[-Werror,-Wvla-cxx-extension]
```
is enabled by default in clang 18 and above.
Disable it in our builds due to large anmount of warnings.
We can revert this once we handle some of the warnings in future PRs.
Matan Breizman [Sun, 16 Feb 2025 16:43:50 +0000 (16:43 +0000)]
script/run-make: bump up to clang 19
Supporting new stable compilers releases early on
should make it easier in the future by preventing introduction of
deperacted code.
As this PR already upgrades to 16 with the relevant fixes, we can
use this opportunity to upgrade to the last stable release available.
Note: Most distinct change here is that -Wvla-extension enabled
by default and will result in lots of warnings.
Matan Breizman [Sun, 16 Feb 2025 14:30:13 +0000 (14:30 +0000)]
test/objectstore/..ops_replayer: cleanup include std::format
clang16 hides std::format behind `-fexperimental-library` [1].
This is no longer the case with clang 17 [2].
As std::fmt is not used in this file we can clean it up for now.
Note: Future commits will aim to switch to clang17.
Matan Breizman [Sun, 16 Feb 2025 13:18:18 +0000 (13:18 +0000)]
cmake/modules/BuildQATzip: Disable errors
qatzip/configure.ac enables -Werror. However, newer compilers
(e.g clang 16) will not compile with the existing warnings
identified. The fixes for the warnings are not merged yet [1]
in the submodule. Until then, in order to allow for compiler
upgrade - we should disable the problematic errors.
Matan Breizman [Mon, 10 Feb 2025 11:12:19 +0000 (11:12 +0000)]
src/script/run-make.sh: clang 14->16
Currently our jammy ci (jenkins) builders are only able to
(apt) install clang 14 as the latest version available.
Clang stable is already at 19 today (i.e we're 5 major releases behind).
Meaning, full support of C++20 and some bugs [1] requiring us to update
the compiler used for make check.
As updating to Ubuntu 24 is not feasable in the near future and
containerized builds is not yet merged - with this patch, we
would get clang-16 directly form llvm installation script.
Note: discover_compiler() would prefer clang-16 even if
clang-14 will be installed by apt in INSTALL_EXTRA_PACKAGES
defined below.
ceph-volume: migrate unit tests from 'mock' to 'unittest.mock'
unit tests in ceph-volume was still using the external 'mock' library
for unit tests, which is unnecessary since 'unittest.mock' is part
of the Python standard library (available since Python 3.3).
This commit updates all imports to use 'unittest.mock' instead,
ensuring better maintainability and removing the need for an extra
dependency.
This refactors `get_physical_osds()`.
The calculation of `data_slots` is now more concise. The handling of
`dev_size`, `rel_data_size`, and `abs_size` is standardized.
The initialization of `free_size` is moved outside the loop
for clarity. Redundant checks and assignments are removed to simplify
the code.
ceph-volume: support splitting db even on collocated scenario
This change enables ceph-volume to create OSDs where the DB is
explicitly placed on a separate LVM partition, even in collocated
scenarios (i.e., block and DB on the same device).
This helps mitigate BlueStore fragmentation issues.
Given that ceph-volume can't automatically predict a proper default size for the db device,
the idea is to use the `--block-db-size` parameter:
Passing `--block-db-size` and `--db-devices` makes ceph-volume create db devices
on dedicated devices (current implementation):
```
Total OSDs: 2
Type Path LV Size % of device
----------------------------------------------------------------------------------------------------
data /dev/vdb 200.00 GB 100.00%
block_db /dev/vdd 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 200.00 GB 100.00%
block_db /dev/vdd 4.00 GB 2.00%
```
Passing `--block-db-size` without `--db-devices` makes ceph-volume create a separate
LV for db device on the same device (new behavior):
```
Total OSDs: 2
Type Path LV Size % of device
----------------------------------------------------------------------------------------------------
data /dev/vdb 196.00 GB 98.00%
block_db /dev/vdb 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 196.00 GB 98.00%
block_db /dev/vdc 4.00 GB 2.00%
```
This new behavior is supported with the `--osds-per-device` parameter:
```
Total OSDs: 4
Type Path LV Size % of device
----------------------------------------------------------------------------------------------------
data /dev/vdb 96.00 GB 48.00%
block_db /dev/vdb 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdb 96.00 GB 48.00%
block_db /dev/vdb 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 96.00 GB 48.00%
block_db /dev/vdc 4.00 GB 2.00%
----------------------------------------------------------------------------------------------------
data /dev/vdc 96.00 GB 48.00%
block_db /dev/vdc 4.00 GB 2.00%
```
Nizamudeen A [Thu, 20 Feb 2025 09:10:28 +0000 (14:40 +0530)]
Merge pull request #61225 from rhcs-dashboard/rgw-ratelimit-integration1
mgr/dashboard: Rgw ratelimit feature for user and bucket
Reviewed-by: Aashish Sharma <aasharma@redhat.com> Reviewed-by: Ankush Behl <cloudbehl@gmail.com> Reviewed-by: Mark Nelson <mnelson@redhat.com> Reviewed-by: Nizamudeen A <nia@redhat.com> Reviewed-by: Naman Munet <nmunet@redhat.com>
Ronen Friedman [Mon, 17 Feb 2025 14:58:41 +0000 (08:58 -0600)]
qa/standalone/scrub: fix osd-scrub-repair()
following changes to 'scrub while recovering' logic, the
'what type of scrubs are allowed to run when recovering'
tests were broken, and are now fixed.
Expand scrub-helpers.sh::standard_scrub_cluster() to cater for
non-standard (i.e. - not "3") pool default size. Also - to
allow clients to specify a "nopool" pool name, preventing the
automatic creation of a pool.
The option was used to allow repair to be explicitly ordered by the
operator, even if the Primary OSD was in the middle of a recovery.
This is no longer necessary, as the code now accepts both repair and
scrub operator requests, even during recovery.
mgr/dashboard: add actions to create, edit and delete smb join-auth and usersgroups resources
Add join-auth and usersgroups resources management and improve the way to select those from the smb cluster form using a dropdown.
Add option to navigate to join-auth/usersgroups resource create form from smb form
Add some additional fixes left over from previous work, such as adding helper texts or adding missing smb cluster form fields
Fixes: https://tracker.ceph.com/issues/69483 Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Ronen Friedman [Thu, 20 Feb 2025 08:01:44 +0000 (10:01 +0200)]
Merge pull request #61812 from ronen-fr/wip-rf-fstab
qa/standalone: do not use /etc/fstab as an always-there bytes source Reviewed-by: Patrick Donnelly <pdonnell@redhat.com> Reviewed-by: Ilya Dryomov <idryomov@gmail.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Max Kellermann [Mon, 17 Feb 2025 07:30:18 +0000 (08:30 +0100)]
include/unordered_{map,set}: remove, use `std` directly instead
This is only an alias for the `std` classes anyway. This effectively
reverts commit 8e8672047ccc which added these aliases, probably
because C++11 was new and not all compilers supported it properly, but
that was 12 years ago and we're onl C++20 meanwhile.
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
projects / ceph.git / log