John Mulligan [Wed, 2 Apr 2025 20:36:29 +0000 (16:36 -0400)]
doc/mgr: add a warning about the smb clustering option & placement
Add a warning to the docs highlighting that `clustering` is an advanced
option and setting it without out also setting a compatible placement
value may lead to unexpected behavior.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
common/mempool.cc: Improve performance of sharding
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com> Reviewed-by: Jose Juan Palacios-Perez <perezjos@uk.ibm.com> Reviewed-by: John Agombar <agombar@uk.ibm.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
On EC pool:
- Use host count instead of device count for host crush-failure-domain
- Host warning for k+m+1
On replicated:
- Set 'All devices' as default
Fixes: https://tracker.ceph.com/issues/70764 Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
cmake: disable deprecated warning when building googletest
In commit 986f6918, we updated the googletest submodule to silence a
CMake warning. However, this change broke the build due to a Clang issue
where it warns about deprecated declarations even from system headers
(see https://github.com/llvm/llvm-project/issues/76515). Since we use
`-Werror` in our compile options, these warnings become errors and fail
the build.
This change detects if we're affected by the specific combination of
compiler, standard library, and compile options that triggers this
issue. It then conditionally disables the `-Wdeprecated-declarations`
warning when building googletest to resolve build failures caused by
deprecated functions like `get_temporary_buffer<>` in the standard
library.
The build failure looks like:
```
FAILED: src/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
...
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-all.cc:38:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:55:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/memory:66:
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_tempbuf.h:263:8: error: 'get_temporary_buffer<testing::TestInfo *>' is deprecated [-Werror,-Wdeprecated-declarations]
263 | std::get_temporary_buffer<value_type>(_M_original_len));
| ^
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_algo.h:4996:15: note: in instantiation of member function 'std::_Temporary_buffer<__gnu_cxx::__normal_iterator<testing::TestInfo **, std::vector<testing::TestInfo *>>, testing::TestInfo *>::_Temporary_buffer' requested here
4996 | _TmpBuf __buf(__first, (__last - __first + 1) / 2);
| ^
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_algo.h:5070:23: note: in instantiation of function template specialization 'std::__stable_sort<__gnu_cxx::__normal_iterator<testing::TestInfo **, std::vector<testing::TestInfo *>>, __gnu_cxx::__ops::_Iter_comp_iter<(lambda at /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3024:20)>>' requested here
5070 | _GLIBCXX_STD_A::__stable_sort(__first, __last,
| ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3023:8: note: in instantiation of function template specializatio
```
Dan Mick [Tue, 1 Apr 2025 19:08:45 +0000 (12:08 -0700)]
doc/dev/release-process.rst: document new Jenkins job for containers
The manual instructions for building release containers were complex
and required access to an arm64 host; that was just too much,
so I've created a Jenkins job to build all of the prerelease
artifacts (see https://github.com/ceph/ceph-build/pull/2342).
Change the documentation to reflect that.
Adam King [Mon, 4 Nov 2024 19:13:02 +0000 (14:13 -0500)]
mgr/cephadm: trigger host metadata refrsh when taking host out of maintenance
If a host was in maintenance mode, we should make sure to check
all of the host information again once it's taken out even if
it hasn't been long enough for the daemon/device/facts etc. cache
to timeout. Some users may make changes to things like the node's
hardware or OS during maintenance mode that we should make sure to
pick up as soon as possible
Adam King [Mon, 4 Nov 2024 19:07:01 +0000 (14:07 -0500)]
mgr/cephadm: add func to invalidate all host metadata and kick serve loop
We already had one of these specifically for daemons. This commit
just adds the same but for all types of host metadata. Currently
the intended use is to call this when taking a host out of maintenace
mode as it's not just the daemons that could have been modified
while the host was in maintenance mode.
would create the my_zone.rgw.buckets.data pool as an
ec pool with a new ec porfile with k=8 and m=2. Currently
the ec profile fields this spec will make use of
are only "k", "m", "pg_num" and "crush-device-class".
If other attributes are set, or if the pool type is
"replicated" the key value pairs are assumed to
be ones that can be passed to the "ceph osd pool create"
command. The other pools for the rgw zone (e.g. buckets
index pool) are all made as replicated pool with
defaults settings
1. Add miscompare message containing object ID.
2. Fix compiler warning due to strangely placed assert.
3. Add barriers following error injects to insure they
are in place before IO.
4. Do not, by default, test EC profiles that are not
known to be good for EC optimisations.
5. Add "allow_unstable_pool_configs" to override above.
Ronen Friedman [Fri, 28 Mar 2025 16:16:52 +0000 (11:16 -0500)]
osd/scrub: decoupling the FSM from the scrubber
The scrubber FSM has always been accessing the PgScrubber
object via an abstract interface (ScrubMachineListener).
But the reverse was not true: the PgScrubber owned a unique_ptr
directly to an instance of Scrubber::ScrubMachine.
This PR introduces a new interface, ScrubFsmIf, that is
implemented by the ScrubMachine. This simplifies
unit-testing the PgScrubber, as we can now inject a mock
implementation of the FSM interface.
Ronen Friedman [Tue, 18 Mar 2025 09:03:39 +0000 (04:03 -0500)]
common/Formatter: introduce helpers for formatting container types.
Specifically:
- with_array_sections() creates a Formmater::ArraySection, then
formats every element in the container using the provided
callable.
Special handling for std::map: the callable is called with
both the key and the value.
- with_obj_array_section() dumps containers of objects:
First - an ArraySection is created; then - for each object
in the container, the object is formatted within an ObjectSection,
using the provided callable.
If the container is an std::map, the callable is called with both
the key and the object itself.
Matt Benjamin [Fri, 21 Feb 2025 19:43:53 +0000 (14:43 -0500)]
rgw:cksum: implement crc64nvme and combined 32- and 64-bit CRCs
* internally compensate for at-rest byteswapped crc64 representation (e.g., before combine step)
* generalize Cksum crc api for 32bit and 64bit crcs, other cleanup
* prototype abstract cksum::Combiner workflow
* add support for forward and backward handling of composite vs full object checksums
by marking the composites and the update flag day
* clean up checksum formatting and checksum type reporting
* add unit tests for Combiner interface
* validate and track requested checksum type (i.e., composite or full), plus
unit test fixture for combinations of full matrix of cksum types
* doh. GET/HEAD checksums are in headers
* add crcany license to COPYING
* return ChecksumType as header in GET/HEAD
Found by Casey in review
* avoid fmt of char* null when no checksum header supplied
A cksum_hdr_t(nullptr, nullptr) results in this case, and is intended,
but its components obviously can't be presented to std::format unless
cast to a safe pointer type.
* fail checksum mismatch with BadDigest
When uploading an S3 object with an invalid checksum, the return code
should be BadDigest to mirror more closely the AWS S3 implementation.
See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
Fixes: https://tracker.ceph.com/issues/70614
Reported by Alex Wojno <awojno@bloomberg.net>
* fix comparison and display of composite checksums
A string comparision bounds error and a bitmask comparison
error are fixed.
* fix build on centos9
On gcc(?13?) we can't declare rgw::cksum::FLAG_NONE
and also rgw::cksum::Cksum::FlAG_NONE.
SAD!!
* include <variant> invariantly
* fix checksum type return from complete-multipart
This one is in the XML response
* aieee, don't leak Combiners
Use unique_ptr to polymorphic type...correctly.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adam Kupczyk [Wed, 12 Mar 2025 09:55:00 +0000 (09:55 +0000)]
os/bluestore/write-v2: Tune down usage of 'unused'
We should not use 'unused' when allocation unit = disk block.
Blobs with 'unused' cannot be split which makes shard resharding
more problematic and spanning blobs more pronounced.
Kefu Chai [Sun, 30 Mar 2025 03:59:12 +0000 (11:59 +0800)]
cephfs-top: Removes unused `global` statements
Recent flake8 runs were failing with:
```
py3: flake8==7.2.0,mccabe==0.7.0,pip==25.0.1,pycodestyle==2.13.0,pyflakes==3.3.0,setuptools==75.8.0,wheel==0.45.1
py3: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/cephfs/top> flake8 --ignore=W503 --max-line-length=100 cephfs-top
cephfs-top:344:9: F824 `global fs_list` is unused: name is never assigned in scope
cephfs-top:466:13: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:872:9: F824 `global metrics_dict` is unused: name is never assigned in scope
cephfs-top:872:9: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:911:9: F824 `global fs_list` is unused: name is never assigned in scope
cephfs-top:981:9: F824 `global current_states` is unused: name is never assigned in scope
cephfs-top:1126:13: F824 `global current_states` is unused: name is never assigned in scope
py3: exit 1 (0.77 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/cephfs/top> flake8 --ignore=W503 --max-line-length=100 cephfs-top pid=2309605
py3: FAIL code 1 (8.15=setup[7.38]+cmd[0.77] seconds)
evaluation failed :( (8.24 seconds)
```
Since these variables are only being referenced and not assigned within
their scopes, the `global` declarations are unnecessary and can be
safely removed. This change:
- Removes all flagged `global` statements
- Fixes the failing flake8 checks in the CI pipeline
- Maintains the original code behavior as variable references still work without the `global` keyword
The `global` keyword is only needed when assigning to global variables
within a function scope, not when simply referencing them.
Kefu Chai [Sun, 30 Mar 2025 03:48:28 +0000 (11:48 +0800)]
qa: Remove unnecessary global statements in tests
Removes unused `global` statements from Python test files to fix flake8
F824 errors.
Recent flake8 runs were failing with:
```
./tasks/radosgw_admin.py:330:5: F824 `global log` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:99:5: F824 `global incompat_paths` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:164:5: F824 `global backward_compat` is unused: name is never assigned in scope
./workunits/dencoder/test_readable.py:165:5: F824 `global fast_shouldnt_skip` is unused: name is never assigned in scope
```
Since these variables are only being referenced and not assigned within
their scopes, the `global` declarations are unnecessary and can be
safely removed. This change:
- Removes all flagged `global` statements
- Fixes the failing flake8 checks in the CI pipeline
- Maintains the original code behavior as variable references still work
without the `global` keyword
The `global` keyword is only needed when assigning to global variables
within a function scope, not when simply referencing them.
Kefu Chai [Sun, 30 Mar 2025 02:31:21 +0000 (10:31 +0800)]
googletest: pick up change to silence CMake warnings
bump up googletest submodule to the latest stable release, in order to
silence the warning from CMake:
```
CMake Deprecation Warning at src/googletest/CMakeLists.txt:4 (cmake_minimum_required):
Compatibility with CMake < 3.10 will be removed from a future version of
CMake.
Update the VERSION argument <min> value. Or, use the <min>...<max> syntax
to tell CMake that the project requires at least <min> but has been updated
to work with policies introduced by <max> or earlier.
CMake Deprecation Warning at src/googletest/googlemock/CMakeLists.txt:45 (cmake_minimum_required):
Compatibility with CMake < 3.10 will be removed from a future version of
CMake.
Update the VERSION argument <min> value. Or, use the <min>...<max> syntax
to tell CMake that the project requires at least <min> but has been updated
to work with policies introduced by <max> or earlier.
CMake Deprecation Warning at src/googletest/googletest/CMakeLists.txt:56 (cmake_minimum_required):
Compatibility with CMake < 3.10 will be removed from a future version of
CMake.
Update the VERSION argument <min> value. Or, use the <min>...<max> syntax
to tell CMake that the project requires at least <min> but has been updated
to work with policies introduced by <max> or earlier.
```
Kefu Chai [Sun, 30 Mar 2025 03:14:01 +0000 (11:14 +0800)]
neorados/test: use [[maybe_unused]] instead of GTEST_ATTRIBUTE_UNUSED_
This change:
- Aligns with GoogleTest's own evolution, which now uses
`[[maybe_unused]]` internally (via
`GTEST_INTERNAL_ATTRIBUTE_MAYBE_UNUSED`) as of
https://github.com/google/googletest/commit/5a37b517
- Leverages C++20 features already available in Ceph
- Removes dependency on GoogleTest internal macros
- Enables updating the GoogleTest submodule to silence CMake warnings
Using the standard C++ attribute provides better compatibility and
future-proofing than relying on GoogleTest's internal implementation
details.
This change also allows us to bump up googletest submodule to silence a
CMake warning.
Kefu Chai [Sun, 30 Mar 2025 02:56:05 +0000 (10:56 +0800)]
neorados/test: Simplify copy prevention by using GtestNonCopyable
Updates the test framework to use GoogleTest's simplified copy
prevention mechanism by inheriting from
`::testing::internal::GtestNonCopyable` instead of custom macros.
This change:
- Replaces deprecated GTEST_DISALLOW_* macros that were removed in
GoogleTest (https://github.com/google/googletest/commit/bf66935e07825318ae519675d73d0f3e313b3ec6)
- Aligns with GoogleTest's current approach for preventing test suite
copying, See
https://github.com/google/googletest/commit/93f08be653c36ddc6943e9513fc14c7292b4d007
- Enables updating the GoogleTest submodule to silence CMake warnings
- Improves compatibility with recent distro-packaged GoogleTest versions
- add the minimal required GTest version when finding this package,
this change should have no impact to our packaging. as both deb
and rpm packagings are using bundled googletest submodule. and
googletest 1.14 has been included by recent stable releases of
popular distros, like fedora 41 and ubuntu 24.04.
While `GtestNonCopyable` is technically an internal helper, we already
use other helpers from the `::testing::internal` namespace, so this
change maintains consistency with our existing approach.
Kefu Chai [Sun, 30 Mar 2025 02:10:28 +0000 (10:10 +0800)]
cmake: do not pass PRE_BUILD to add_custom_command(OUTPUT)
Removes the invalid PRE_BUILD argument from add_custom_command(OUTPUT)
calls that was triggering CMake warnings. The PRE_BUILD option is only
valid for target-based custom commands and specifically with Visual
Studio generators.
Additionally sets the CMP0175 policy to NEW to ensure CMake fails on
invalid arguments to add_custom_command(), helping catch similar issues
earlier in the development process.
This change:
- Removes unnecessary PRE_BUILD argument from OUTPUT-based custom commands
- Sets CMP0175 policy to enforce validation of add_custom_command() arguments
- Resolves CMake warning in src/common/options/CMakeLists.txt
```
CMake Warning (dev) at src/common/options/CMakeLists.txt:34 (add_custom_command):
The following keywords are not supported when using
add_custom_command(OUTPUT): PRE_BUILD.
Policy CMP0175 is not set: add_custom_command() rejects invalid arguments.
Run "cmake --help-policy CMP0175" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
Call Stack (most recent call first):
src/common/options/CMakeLists.txt:85 (add_options)
This warning is for project developers. Use -Wno-dev to suppress it.
```
Kefu Chai [Sun, 30 Mar 2025 01:39:53 +0000 (09:39 +0800)]
cmake: Allow building with fmt <= 11.1.4
Testing confirms successful builds with fmt 11.1.4, which is currently
packaged in Fedora 42. This change relaxes the fmt version constraints
to improve compatibility with Fedora 42 and other distributions shipping
this version of fmt-devel.
projects / ceph.git / log