common/mempool.cc: Improve performance of sharding
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com> Reviewed-by: Jose Juan Palacios-Perez <perezjos@uk.ibm.com> Reviewed-by: John Agombar <agombar@uk.ibm.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
On EC pool:
- Use host count instead of device count for host crush-failure-domain
- Host warning for k+m+1
On replicated:
- Set 'All devices' as default
Fixes: https://tracker.ceph.com/issues/70764 Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
cmake: disable deprecated warning when building googletest
In commit 986f6918, we updated the googletest submodule to silence a
CMake warning. However, this change broke the build due to a Clang issue
where it warns about deprecated declarations even from system headers
(see https://github.com/llvm/llvm-project/issues/76515). Since we use
`-Werror` in our compile options, these warnings become errors and fail
the build.
This change detects if we're affected by the specific combination of
compiler, standard library, and compile options that triggers this
issue. It then conditionally disables the `-Wdeprecated-declarations`
warning when building googletest to resolve build failures caused by
deprecated functions like `get_temporary_buffer<>` in the standard
library.
The build failure looks like:
```
FAILED: src/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
...
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest-all.cc:38:
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:55:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/memory:66:
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_tempbuf.h:263:8: error: 'get_temporary_buffer<testing::TestInfo *>' is deprecated [-Werror,-Wdeprecated-declarations]
263 | std::get_temporary_buffer<value_type>(_M_original_len));
| ^
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_algo.h:4996:15: note: in instantiation of member function 'std::_Temporary_buffer<__gnu_cxx::__normal_iterator<testing::TestInfo **, std::vector<testing::TestInfo *>>, testing::TestInfo *>::_Temporary_buffer' requested here
4996 | _TmpBuf __buf(__first, (__last - __first + 1) / 2);
| ^
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/stl_algo.h:5070:23: note: in instantiation of function template specialization 'std::__stable_sort<__gnu_cxx::__normal_iterator<testing::TestInfo **, std::vector<testing::TestInfo *>>, __gnu_cxx::__ops::_Iter_comp_iter<(lambda at /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3024:20)>>' requested here
5070 | _GLIBCXX_STD_A::__stable_sort(__first, __last,
| ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3023:8: note: in instantiation of function template specializatio
```
Dan Mick [Tue, 1 Apr 2025 19:08:45 +0000 (12:08 -0700)]
doc/dev/release-process.rst: document new Jenkins job for containers
The manual instructions for building release containers were complex
and required access to an arm64 host; that was just too much,
so I've created a Jenkins job to build all of the prerelease
artifacts (see https://github.com/ceph/ceph-build/pull/2342).
Change the documentation to reflect that.
Adam King [Mon, 4 Nov 2024 19:13:02 +0000 (14:13 -0500)]
mgr/cephadm: trigger host metadata refrsh when taking host out of maintenance
If a host was in maintenance mode, we should make sure to check
all of the host information again once it's taken out even if
it hasn't been long enough for the daemon/device/facts etc. cache
to timeout. Some users may make changes to things like the node's
hardware or OS during maintenance mode that we should make sure to
pick up as soon as possible
Adam King [Mon, 4 Nov 2024 19:07:01 +0000 (14:07 -0500)]
mgr/cephadm: add func to invalidate all host metadata and kick serve loop
We already had one of these specifically for daemons. This commit
just adds the same but for all types of host metadata. Currently
the intended use is to call this when taking a host out of maintenace
mode as it's not just the daemons that could have been modified
while the host was in maintenance mode.
would create the my_zone.rgw.buckets.data pool as an
ec pool with a new ec porfile with k=8 and m=2. Currently
the ec profile fields this spec will make use of
are only "k", "m", "pg_num" and "crush-device-class".
If other attributes are set, or if the pool type is
"replicated" the key value pairs are assumed to
be ones that can be passed to the "ceph osd pool create"
command. The other pools for the rgw zone (e.g. buckets
index pool) are all made as replicated pool with
defaults settings
1. Add miscompare message containing object ID.
2. Fix compiler warning due to strangely placed assert.
3. Add barriers following error injects to insure they
are in place before IO.
4. Do not, by default, test EC profiles that are not
known to be good for EC optimisations.
5. Add "allow_unstable_pool_configs" to override above.
John Mulligan [Fri, 28 Mar 2025 15:56:12 +0000 (11:56 -0400)]
doc/mgr/smb: document the new set of smb mgr module options
Document the new --results and --password-filter{,-out} options.
These options are added for show and apply but also for
cluster {create,rm} as these take and emit these values too.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 17 Mar 2025 20:10:15 +0000 (16:10 -0400)]
mgr/smb: extend password filtering to cluster commands
The imperative commands like `ceph smb cluster create` may involve
setting and reporting outputs that have passwords. Use the previously
established password filter mechanism there (and use the new _apply_res
function everywhere for completeness).
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 12 Mar 2025 21:34:27 +0000 (17:34 -0400)]
mgr/smb: add password filter support to ceph smb apply command
The apply command both takes passwords in the input and can display them
in the results. Add two options --password-filter and
--password-filter-out that control how these passwords are filtered. If
--password-filter is given but --password-filter-out is not, the output
value is assumed to be the same as the input. So for example you could
run `ceph smb apply -i /file --password-filter=base64` which would
assume the password values in the input are base64 encoded and then
the results would include the same values.
`ceph smb apply -i /file --password-filter-out=hidden` would take
a file with unobscured passwords but emit results with passwords
hidden.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 12 Mar 2025 21:34:12 +0000 (17:34 -0400)]
mgr/smb: add a password filer enum for input passwords
Add a 2nd password filter enum but this one is only for input passwords.
Specifically, the 'hidden' filter makes no sense because we don't want
junk passwords in the data store so we only take unfiltered or base64
obscured password values.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 12 Mar 2025 21:33:04 +0000 (17:33 -0400)]
mgr/smb: add support for password conversion/filtering to results types
Because these results types embed resources and these are used to
display output of commands we want to ensure that they support password
conversion/filtering too.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 12 Mar 2025 18:28:15 +0000 (14:28 -0400)]
mgr/smb: initial support for content conversions - password filtering
Add the core of a basic support for converting the content of various
resources. The current goal being filtering passwords to either obscure
them (base64) or hide them (hidden). The default continues to be plain
text as before.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 12 Mar 2025 17:07:21 +0000 (13:07 -0400)]
mgr/smb: allow show command to collapse or keep full objects list
Previously, the `ceph smb show` command would "collapse" the output to
one single resource if exactly one resource matched, as opposed to
emitting a resource list with one single item. Add new switch,
`--results=[collapsed|full]`, that will either continue the default
behavior (collapsed) or retain the output as a resource list object
(full).
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Ronen Friedman [Fri, 28 Mar 2025 16:16:52 +0000 (11:16 -0500)]
osd/scrub: decoupling the FSM from the scrubber
The scrubber FSM has always been accessing the PgScrubber
object via an abstract interface (ScrubMachineListener).
But the reverse was not true: the PgScrubber owned a unique_ptr
directly to an instance of Scrubber::ScrubMachine.
This PR introduces a new interface, ScrubFsmIf, that is
implemented by the ScrubMachine. This simplifies
unit-testing the PgScrubber, as we can now inject a mock
implementation of the FSM interface.
cephadm will search for an IP address within 192.168.122.0/24
and have the RGW daemon only bind on that IP rather than on
all networks. What RGW binds to is controlled by the
"rgw_frontends" setting for the rgw daemon, so what cephadm
write there is all that is modified by this commit.
Ronen Friedman [Tue, 18 Mar 2025 09:03:39 +0000 (04:03 -0500)]
common/Formatter: introduce helpers for formatting container types.
Specifically:
- with_array_sections() creates a Formmater::ArraySection, then
formats every element in the container using the provided
callable.
Special handling for std::map: the callable is called with
both the key and the value.
- with_obj_array_section() dumps containers of objects:
First - an ArraySection is created; then - for each object
in the container, the object is formatted within an ObjectSection,
using the provided callable.
If the container is an std::map, the callable is called with both
the key and the object itself.
Matt Benjamin [Fri, 21 Feb 2025 19:43:53 +0000 (14:43 -0500)]
rgw:cksum: implement crc64nvme and combined 32- and 64-bit CRCs
* internally compensate for at-rest byteswapped crc64 representation (e.g., before combine step)
* generalize Cksum crc api for 32bit and 64bit crcs, other cleanup
* prototype abstract cksum::Combiner workflow
* add support for forward and backward handling of composite vs full object checksums
by marking the composites and the update flag day
* clean up checksum formatting and checksum type reporting
* add unit tests for Combiner interface
* validate and track requested checksum type (i.e., composite or full), plus
unit test fixture for combinations of full matrix of cksum types
* doh. GET/HEAD checksums are in headers
* add crcany license to COPYING
* return ChecksumType as header in GET/HEAD
Found by Casey in review
* avoid fmt of char* null when no checksum header supplied
A cksum_hdr_t(nullptr, nullptr) results in this case, and is intended,
but its components obviously can't be presented to std::format unless
cast to a safe pointer type.
* fail checksum mismatch with BadDigest
When uploading an S3 object with an invalid checksum, the return code
should be BadDigest to mirror more closely the AWS S3 implementation.
See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
Fixes: https://tracker.ceph.com/issues/70614
Reported by Alex Wojno <awojno@bloomberg.net>
* fix comparison and display of composite checksums
A string comparision bounds error and a bitmask comparison
error are fixed.
* fix build on centos9
On gcc(?13?) we can't declare rgw::cksum::FLAG_NONE
and also rgw::cksum::Cksum::FlAG_NONE.
SAD!!
* include <variant> invariantly
* fix checksum type return from complete-multipart
This one is in the XML response
* aieee, don't leak Combiners
Use unique_ptr to polymorphic type...correctly.
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>