Matt Benjamin [Fri, 21 Feb 2025 19:43:53 +0000 (14:43 -0500)]
rgw:cksum: implement crc64nvme and combined 32- and 64-bit CRCs
* internally compensate for at-rest byteswapped crc64 representation (e.g., before combine step)
* generalize Cksum crc api for 32bit and 64bit crcs, other cleanup
* prototype abstract cksum::Combiner workflow
* add support for forward and backward handling of composite vs full object checksums
by marking the composites and the update flag day
* clean up checksum formatting and checksum type reporting
* add unit tests for Combiner interface
* validate and track requested checksum type (i.e., composite or full), plus
unit test fixture for combinations of full matrix of cksum types
* doh. GET/HEAD checksums are in headers
* add crcany license to COPYING
* return ChecksumType as header in GET/HEAD
Found by Casey in review
* avoid fmt of char* null when no checksum header supplied
A cksum_hdr_t(nullptr, nullptr) results in this case, and is intended,
but its components obviously can't be presented to std::format unless
cast to a safe pointer type.
* fail checksum mismatch with BadDigest
When uploading an S3 object with an invalid checksum, the return code
should be BadDigest to mirror more closely the AWS S3 implementation.
See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
Fixes: https://tracker.ceph.com/issues/70614
Reported by Alex Wojno <awojno@bloomberg.net>
* fix comparison and display of composite checksums
A string comparision bounds error and a bitmask comparison
error are fixed.
* fix build on centos9
On gcc(?13?) we can't declare rgw::cksum::FLAG_NONE
and also rgw::cksum::Cksum::FlAG_NONE.
SAD!!
* include <variant> invariantly
* fix checksum type return from complete-multipart
This one is in the XML response
* aieee, don't leak Combiners
Use unique_ptr to polymorphic type...correctly.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Kefu Chai [Sun, 30 Mar 2025 03:59:12 +0000 (11:59 +0800)]
cephfs-top: Removes unused `global` statements
Recent flake8 runs were failing with:
```
py3: flake8==7.2.0,mccabe==0.7.0,pip==25.0.1,pycodestyle==2.13.0,pyflakes==3.3.0,setuptools==75.8.0,wheel==0.45.1
py3: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/cephfs/top> flake8 --ignore=W503 --max-line-length=100 cephfs-top
cephfs-top:344:9: F824 `global fs_list` is unused: name is never assigned in scope
cephfs-top:466:13: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:872:9: F824 `global metrics_dict` is unused: name is never assigned in scope
cephfs-top:872:9: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:911:9: F824 `global fs_list` is unused: name is never assigned in scope
cephfs-top:981:9: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:1126:13: F824 `global current_states` is unused: name is never assigned in scope
py3: exit 1 (0.77 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/cephfs/top> flake8 --ignore=W503 --max-line-length=100 cephfs-top pid=2309605
py3: FAIL code 1 (8.15=setup[7.38]+cmd[0.77] seconds)
evaluation failed :( (8.24 seconds)
```
Since these variables are only being referenced and not assigned within
their scopes, the `global` declarations are unnecessary and can be
safely removed. This change:
- Removes all flagged `global` statements
- Fixes the failing flake8 checks in the CI pipeline
- Maintains the original code behavior as variable references still work without the `global` keyword
The `global` keyword is only needed when assigning to global variables
within a function scope, not when simply referencing them.
Kefu Chai [Sun, 30 Mar 2025 03:48:28 +0000 (11:48 +0800)]
qa: Remove unnecessary global statements in tests
Removes unused `global` statements from Python test files to fix flake8
F824 errors.
Recent flake8 runs were failing with:
```
./tasks/radosgw_admin.py:330:5: F824 `global log` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:99:5: F824 `global incompat_paths` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:164:5: F824 `global backward_compat` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:165:5: F824 `global fast_shouldnt_skip` is unused: name is never assigned in scope
```
Since these variables are only being referenced and not assigned within
their scopes, the `global` declarations are unnecessary and can be
safely removed. This change:
- Removes all flagged `global` statements
- Fixes the failing flake8 checks in the CI pipeline
- Maintains the original code behavior as variable references still work
without the `global` keyword
The `global` keyword is only needed when assigning to global variables
within a function scope, not when simply referencing them.
Kefu Chai [Wed, 26 Mar 2025 03:43:43 +0000 (11:43 +0800)]
src: migrate to fmt::formattable for format support
Replace custom has_formatter concept with fmtlib's fmt::formattable
- Deprecate usage of fmt::has_formatter
- Resolve compiler warnings with recent fmtlib versions
The custom concept of has_formatter is no longer necessary with
recent updates to the fmtlib library, as fmt::formattable was introduced
in fmtlib v11.0.0. By switching to fmt::formattable, we simplify our
code and stop using the deprecated `fmt::has_formatter` template.
Kefu Chai [Sat, 29 Mar 2025 13:10:05 +0000 (21:10 +0800)]
librbd: replace deprecated atomic_store with std::atomic<shared_ptr>
Update shared pointer atomic operations to use C++20's std::atomic<std::shared_ptr<T>>
instead of the deprecated atomic_store functions. This change addresses deprecation
warnings from GCC-15's libstdc++ where atomic shared pointer operations outside the
std::atomic class are being phased out:
```
/home/kefu/dev/ceph/src/librbd/ImageCtx.cc:1010:5: warning: 'atomic_store<neorados::IOContext>' is deprecated: use 'std::atomic<std::shared_ptr<T>>' instead [-Wdeprecated-declarations]
1010 | atomic_store(&data_io_context, ctx);
| ^
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_atomic.h:181:5: note: 'atomic_store<neorados::IOContext>' has been explicitly marked deprecated here
181 | _GLIBCXX20_DEPRECATED_SUGGEST("std::atomic<std::shared_ptr<T>>")
| ^
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/x86_64-redhat-linux/bits/c++config.h:2055:45: note: expanded from macro '_GLIBCXX20_DEPRECATED_SUGGEST'
2055 | # define _GLIBCXX20_DEPRECATED_SUGGEST(ALT) _GLIBCXX_DEPRECATED_SUGGEST(ALT)
| ^
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/x86_64-redhat-linux/bits/c++config.h:2023:19: note: expanded from macro '_GLIBCXX_DEPRECATED_SUGGEST'
2023 | __attribute__ ((__deprecated__ ("use '" ALT "' instead")))
| ^
```
The implementation now uses the standard-compliant approach that's recommended in
the compiler warnings, while maintaining backward compatibility with older compilers
by conditionally selecting the appropriate implementation.
Kefu Chai [Tue, 25 Mar 2025 04:03:30 +0000 (12:03 +0800)]
common: disable OpenSSL engine support if it is disabled
OpenSSL 3.0 documentation recommends moving from the ENGINE API to the
Providers API. Recent distributions may compile OpenSSL without engine
support by default, necessitating more flexible configuration handling.
So, in this change:
- Add a CMake option `WITH_OPENSSL_ENGINE` to explicitly control engine support
- Respect `openssl_engine_opts` when engine support is enabled
- Provide clear error messaging when engine options are set but support is disabled
See also:
- OpenSSL 3.0 documentation:
https://wiki.openssl.org/index.php/OpenSSL_3.0#Engines_and_.22METHOD.22_APIs
Zac Dover [Thu, 27 Mar 2025 12:24:13 +0000 (22:24 +1000)]
src/common: add guidance for mon_warn_pg_not_scrubbed
Add an explanation of how to set the value of
"mon_warn_pg_not_scrubbed_ratio" to the confval definition of that
variable. Although this variable contains the string "mon", it is set on
the Manager. I have added a note to direct users to set this value on
the Manager.
This issue was pointed out by Petr Tlapa on Slack in late March of 2025.
This issue is part of a small project that also encompasses
https://github.com/ceph/ceph/pull/62459, which is associated with commit aeef59a50ee31072648ba0c7436b6522137614cd
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
Previously, we relied on the __GNUC__ macro to check for std::atomic<std::shared_ptr>
support, which was inaccurate. This approach failed with Clang builds using libstdc++,
even when the feature was implemented. The warning looks like:
```
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_atomic.h:131:5: note: 'atomic_load_explicit<std::vector<entity_addrvec_t>>' has been explicitly marked deprecated here
131 | _GLIBCXX20_DEPRECATED_SUGGEST("std::atomic<std::shared_ptr<T>>")
| ^
```
This change uses a standard-compliant feature test macro (__cpp_lib_atomic_shared_ptr)
to correctly detect support for std::atomic<std::shared_ptr>. This resolves compilation
issues and improves portability across different compilers and standard library
implementations.
Matt Benjamin [Tue, 18 Feb 2025 19:11:08 +0000 (14:11 -0500)]
rgw_cksum: add 64bit and 32bit crc constructions from crcany
These constructions provide conforming implemenations of Amazon
S3 CRC32 (ISO hdlc), CRC32C (iscsi), and CRC64/NVME checksums,
in particular, linear combining of adjacent checksums, by
Mark Adler.
Source: https://github.com/madler/crcany
License: approximate BSD with "optional" advertising clause
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Ilya Dryomov [Thu, 20 Mar 2025 17:28:44 +0000 (18:28 +0100)]
rbd: mirror_uuids -> mirror_uuid in remote mirror peer listing
... in "rbd mirror image status" and "rbd mirror pool status --verbose"
formatted outputs. In unformatted outputs this value is shown in place
of a name in case a site name isn't available.
Ilya Dryomov [Thu, 20 Mar 2025 16:10:47 +0000 (17:10 +0100)]
rbd: extend use of "none" placeholder to IMAGES section
Currently if there are no mirror-enabled images, IMAGES section
in "rbd mirror pool status --verbose" output isn't terminated:
$ rbd mirror pool status data --verbose
health: OK
daemon health: OK
image health: OK
images: 0 total
DAEMONS
service 4388:
...
health: OK
IMAGES$
DAEMONS section has a "none" placeholder for when there are no
rbd-mirror daemons running. Fix some issues with the separator logic
and employ the placeholder in IMAGES section:
$ rbd mirror pool status data --verbose
health: OK
daemon health: OK
image health: OK
images: 0 total
DAEMONS
service 4388:
...
health: OK
IMAGES
none
$
$ rbd mirror pool status data --verbose
health: UNKNOWN
daemon health: UNKNOWN
image health: OK
images: 0 total
rgw: fix the version in DOCODE_START() of RGWZoneGroupPlacementTier
This commit fixes an undetected merge conflict between PRs #61745
and #60159. The dencoding problem has been introduced very recently,
it is straightforward and causes failures of the make check bot
everywhere, therefore -- if no objections -- I want to merge this
patch without the Teuthology testing.
Redouane Kachach [Fri, 21 Mar 2025 12:13:56 +0000 (13:13 +0100)]
mgr/cephadm: making mgmt-gateway an oauth2-proxy dependency
This change enables better automation, especially for complex setups
like high-availability configurations. Previously, users had to
manually deploy the mgmt-gateway before the oauth-proxy; if this
sequence wasn't followed, cephadm would raise an error.
Kefu Chai [Mon, 24 Mar 2025 02:05:25 +0000 (10:05 +0800)]
librbd: migrate from boost::variant to std::variant
migrate from boost::variant to std::variant
Complete migration started in commit 017f3339c, replacing boost::variant with
std::variant throughout the librbd codebase. This change is part of our ongoing
effort to reduce third-party dependencies by leveraging C++ standard library
alternatives where possible.
Benefits include:
- Improved code readability and maintainability
- Reduced external dependency surface
- More consistent API usage with other components
Aashish Sharma [Tue, 25 Mar 2025 11:35:05 +0000 (17:05 +0530)]
mgr/dashboard: fix image filter's query on rbd-details grafana panel
The image filter on the RBD Details grafana panel is using a query with a type "label_values(ceph_rbd_read_ops{cluster=~\"$cluster\", , pool=\"$pool\"}, image)". The extra comma needs to be removed.
Ilya Dryomov [Tue, 25 Mar 2025 08:13:27 +0000 (09:13 +0100)]
mgr/rbd_support: always parse interval and start_time in Schedules::remove()
Commit 1b62447071a9 ("mgr/rbd_support: fix schedule remove") addressed
the issue that it was concerned with in a rather suboptimal way: instead
of moving the parsing of interval and start_time upfront to be able to
bail early, it wrapped from_string() constructors with try/finally and
left the conditional behavior in place.
projects / ceph.git / log