osd/SnapMapper:
Maintain the prefix_itr between calls to SnapMapper::get_next_objects_to_trim() to prevent searching depleted prefixes.
We got 8 distinct hash prefixes used for searching objects owned by a given PG.
On each call to SnapMapper::get_next_objects_to_trim() we start from the first prefix even after all objects mapped to it were depleted.
This means that we will be searching for 1 non-existing prefix after the first prefix was depleted, 2 after the first two prefixes were depleted... and so on until we will search 7 non-existing prefixes after the first 7 prefixes were depleted.
This is a performance improvement PR only!
It maintains the existing behavior and does not try to fix/change any of the TRIM logic.
I added an extra step after the last object is trimmed doing a full scan of the DB and only if no object was found it will return ENOENT.
This should make the new code no-worse than existing code which returns ENOENT after a full scan found no object.
It should not impact performance in real life snaps as it should only happen once per-snap.
added snap-mapper tests to rados-test-suite
disabled osd_debug_trim_objects when running (SnapMapperTest, prefix_itr) to prevent asserts(as this code does illegal inserts into DELETED snaps)
Code beautifing
Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>
Adam King [Fri, 18 Aug 2023 16:32:38 +0000 (12:32 -0400)]
cephadm: have attempt_bind raise up OSErrors
Before it was always converting the OSError to
our self-defined "Error" class. This causes an issue
with the port_in_use function that has special handling
for OSError when the errno is EADDRNOTAVAIL or
EAFNOSUPPORT. Since the error being raised was no
longer an OSError it wasn't being caught and checked
properly in port_in_use.
This has the additional property of being necessary
to check port availability for haproxy on its VIP. If
we fail deployment when EADDRNOTAVAIL is raised, it becomes
difficult to deploy the ingress service. If we deploy
haproxy first it fails because the VIP isn't available
yet (since keepalive isn't up) and it fails saying the port
it wants to bind to is unavailable (specifically EADDRNOTAVAIL).
If we try to deploy keepalive first it fails because it
needs to know the location of the haproxy daemons in
order to build its config file. This has worked in the past
by just having the haproxy fail to bind at first and then
fix itself once the keepalive daemon is deployed. That
no longer works if the haproxy daemon fails to deploy
because cephadm is reporting the port it needs is
unavailable. Since EADDRNOTAVAIL when deploying
haproxy likely means the VIP is not up rather than
something else is taking up the port it needs, fixing
the handling of this allows ingress deployment to
work while also allowing multiple haproxy daemons
on the same host to use the same frontend port
bound to different VIPs.
Adam King [Fri, 11 Aug 2023 18:00:31 +0000 (14:00 -0400)]
mgr/cephadm: only check haproxy frontend_port conflicts on VIP
If we know what IP the frontend_port will be binding
to, we can pass that down through the port_ips mapping
so cephadm will only check if that port on that specific
VIP if in use. This allows multiple haproxy daemons
to be bound to the same port on different VIPs on the
same host.
Note that you still must use a different monitor port
for the two different ingress services as that port
is bound to on the actual IP of the host. Only the
frontend port can be the same for haproxies on the
same host as long as the VIP is different.
Fixes: https://tracker.ceph.com/issues/57614 Signed-off-by: Adam King <adking@redhat.com>
Adam King [Fri, 11 Aug 2023 17:35:10 +0000 (13:35 -0400)]
cephadm: support for specifying IP a port will be bound to on
This is mostly for checking for port conflicts.
Currently, we just check if the port is bound to
on any IP on the host. This mechanism should allow
certain daemon types to specify a port -> IP mapping
that will be passed to the cephadm binary. That
mapping will then be used by cephadm to only
check for the port being bound to on that specific
IP rather than any IP on the host. The end result
is we could have daemons bound to the same
port on different IPs on the same node.
It's expected that daemon types will set this
up as part of their prepare_create or generate_config
functions where they may have more info about the
specific IPs and ports they need.
Zac Dover [Tue, 22 Aug 2023 05:30:21 +0000 (15:30 +1000)]
doc/start: refactor ABC test chart
Refactor the ABC test chart so that the information about which tests
have been run is presented in the center of the chart instead of, as it
was before, in a superscript.
Venky Shankar [Thu, 17 Aug 2023 13:01:42 +0000 (18:31 +0530)]
Revert "qa/cephfs: fix ior build failure"
PR #52416 is incomplete and got merged on the basis of a linked
test run which passed, however, subsequent update to the PR
caused the test to start failing again.
Casey Bodley [Tue, 8 Aug 2023 14:00:22 +0000 (10:00 -0400)]
script: fix credits.sh showing 0 lines changed for mapped emails
script was showing 0 for several contributors because the commit Author:
did not match their mapped address:
```
Number of lines added and removed, by authors
...
306 0 Radoslaw Zarzynski <rzarzynski@redhat.com>
307 0 Mike Perez <miperez@redhat.com>
308 0 Michael J. Kidd <linuxkidd@redhat.com>
309 0 Lukas Mayer <lmayer@wind.gmbh>
310 0 Luis Henriques <lhenriques@suse.com>
311 0 Kyle McGough <kmcgough@digitalocean.com>
312 0 João Eduardo Luís <joao@suse.de>
313 0 JinyongHa <jy200.ha@samsung.com>
314 0 Ilya Dryomov <idryomov@redhat.com>
```
in the case of Ilya and Radoslaw:
```
$ git log --no-merges --pretty='%ae' v17.2.6..v18.2.0 | sed -e "s/'/ /g" | sort -u | grep -e idryomov -e rzarzyns
idryomov@gmail.com
rzarzyns@redhat.com
```
this output gets fed into `git log --numstat --author="$mail"` to fetch
the stats, but that command maps idryomov@gmail.com -> idryomov@redhat.com
and rzarzyns@redhat.com -> rzarzynski@redhat.com so no commits matched.
disabling the mapping for this command with --no-mailmap allows it to
fetch their stats correctly:
```
14 11809 Radoslaw Zarzynski <rzarzynski@redhat.com>
16 10051 Ilya Dryomov <idryomov@redhat.com>
```
Tobias Urdin [Mon, 7 Aug 2023 20:34:43 +0000 (20:34 +0000)]
rgw/auth: handle HTTP OPTIONS with v4 auth
This adds code to properly verify the signature
for HTTP OPTIONS calls that is preflight CORS
requests passing the expected method in the
access-control-request-method header.
tengjie5 [Tue, 8 Aug 2023 02:13:58 +0000 (10:13 +0800)]
doc: modify and add descriptions for cephfs-shell commands. Fixes: https://tracker.ceph.com/issues/62074 Signed-off-by: Teng Jie <tengjie5@asiainfo.com>
Kefu Chai [Thu, 17 Aug 2023 02:23:45 +0000 (10:23 +0800)]
cmake: update FindPython3 module
* use the one shiped by the latest CMake (ab379e5054aa792df9572078dcf95bddd75f7661)
* use the new policy to use the new find strategy.
* accomodate the vanilla FindPython3 module to Ceph by:
- dropping the `cmake_policy()` calls which set the policy not supported
by 3.16.
- `include (FindPackageHandleStandardArgs)` without specifying the
relative path.
- dropping the `HANDLE_VERSION_RANGE` from `FindPackageHandleStandardArgs()` call.
this option was introduced by CMake v3.19, see
https://cmake.org/cmake/help/latest/module/FindPackageHandleStandardArgs.html
but Ubuntu focal comes with CMake 3.16, which is our minimal required CMake version.
the new FindPython3 module from CMake:
* enables us to find the recent Python intepreter and development files up to
CPython 3.13.
* finds Python intepreter with the new `Python_FIND_STRATEGY`. the old and
default strategy always finds the most recent version with all specified name
and in all locations. so, if /usr/bin/python exists, it would accept, even
if it is a symlink to python3.9 and what we want is python3.6. the
new policy stops at the one which satisfies the constraints. this
helps us to address the problem of issue we have when recreating
the building system after the CMakeLists.txt is updated.
Kefu Chai [Mon, 14 Aug 2023 15:27:42 +0000 (23:27 +0800)]
cmake: update FindPython3 module
* use the one shiped by the latest CMake (ab379e5054aa792df9572078dcf95bddd75f7661)
* use the new policy to use the new find strategy.
* accomodate the vanilla FindPython3 module to Ceph by:
- dropping the `cmake_policy()` calls which set the policy not supported
by 3.16.
- `include (FindPackageHandleStandardArgs)` without specifying the
relative path.
- dropping the `HANDLE_VERSION_RANGE` from `FindPackageHandleStandardArgs()` call.
this option was introduced by CMake v3.19, see
https://cmake.org/cmake/help/latest/module/FindPackageHandleStandardArgs.html
but Ubuntu focal comes with CMake 3.16, which is our minimal required CMake version.
the new FindPython3 module from CMake:
* enables us to find the recent Python intepreter and development files up to
CPython 3.13.
* finds intepreter with the new `Python_FIND_STRATEGY`. the old and default
strategy always finds the most recent version with all specified name
and in all locations. so, if /usr/bin/python exists, it would accept, even
if it is a symlink to python3.9 and what we want is python3.6. while
the new policy stops at the one which satisfies the constraints.
simpler this way and less error prone.
crimson/os/seastore/journal/cbj: introduce magic value to identify that written records are correct
The cbj would replay invalid records if the same write pattern (mkfs -> write -> mkfs -> write) repeats.
On the second mkfs(), current cbj wrongly recognizes the old records are valid beceause they have
correct seq. number at appropriate offset even though the records are expired.
To solve this, this commit introduces random value to identify the journal during mkfs()
Lucian Petrut [Fri, 4 Nov 2022 10:30:43 +0000 (12:30 +0200)]
test/librbd/fsx: add rbd-wnbd tests
rbd-wnbd can be used to map rbd images as Windows block devices.
This change will allow exercising rbd-wnbd and the wnbd driver
as opposed to using librbd directly.
One thing to note here is that read and write operations must be
512B aligned. Also, wnbd does not currently support "compare
and write".
For this reason, we'll stick to the basic operations for the time
being.
Note that we're renaming krbd_read/krbd_write to generic_pread
and generic_pwrite, reusing those functions for wnbd operations.
We're also removing the duplicated ggate read/write operations,
using the generic functions instead.
Adam King [Fri, 2 Jun 2023 00:06:35 +0000 (20:06 -0400)]
cephadm: add tcmu-runner to logrotate config
This process could be used to set up the tcmu-runner
to log to a file much like other ceph daemons
- create /etc/tcmu directory
- create /etc/tcmu/tcmu.conf directory with default options
- change dir to /var/log
- change log level to 4
- add -v /etc/tcmu:/etc/tcmu to tcmu-runner container podman line in unit.run
In order to support this (mostly for debugging) we should
add tcmu-runner to the logrotate config
Fixes: https://tracker.ceph.com/issues/61571 Signed-off-by: Adam King <adking@redhat.com>
projects / ceph.git / log