Casey Bodley [Mon, 8 Jan 2024 16:24:18 +0000 (08:24 -0800)]
make-dist: don't use --continue option for wget
the boost jfrog mirror is broken and returns an HTML error page instead
of the archive. the file size of this page is 11534 bytes
when download_from() retries the download from download.ceph.com, the -c
option tells it to resume the download of the existing file. the
resulting boost_1_82_0.tar.bz2 ends up with the correct total file size
of 121325129 bytes, but the first 11534 bytes still correspond to the
HTML from jfrog. that causes the sha256sum mismatch
remove the -c option so that wget fetches the archive in its entirety
Ronen Friedman [Thu, 28 Dec 2023 19:41:19 +0000 (13:41 -0600)]
osd/scrub: avoid "over clearing" queued_or_active flag
If two StartScrub messages are received in quick succession, the earlier
one might clear the queued_or_active flag as it fails for being from an old
interval.
When that happens - a 3'rd scrub request will actually be allowed to go
through, while the scrubber is still handling the second one.
Myoungwon Oh [Sun, 17 Dec 2023 08:51:22 +0000 (17:51 +0900)]
crimson/os/seastore: introduce modified_region in DATA_BLOCK to keep track of modified region
It has a limitation to keep track of the modified region using the existing
deltas because we can not get the correct region in two cases: 1) a case where replay
is done and 2) duplicate_for_write. This commit introduces modified region
to solve the problem.
Matan Breizman [Sun, 1 Jan 2023 11:41:34 +0000 (11:41 +0000)]
crimson/osd: Keep track of modified_ranges
* `modifies_ranges` interval_set is added to osd_op_params_t
* keep track of modified_ranges while executing relevant ops
* Add `osd_op_params` parameter to `PGBackend::remove()`.
Zac Dover [Wed, 3 Jan 2024 08:41:51 +0000 (18:41 +1000)]
doc/radosgw: edit "Add/Remove a Key"
Edit the section "Add/Remove a Key" in doc/radosgw/admin.rst. Each
operation (e.g. "Adding an S3 key pair for a user", "Removing an S3 key
pair for a user") now has its own subsection. This increased granularity
should make it easier in the future to link to each of these specific
operations, if needed.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
John Mulligan [Tue, 19 Dec 2023 19:10:39 +0000 (14:10 -0500)]
cehphadm: disable restarting init containers service
Disable the Restart= line in the init containers systemd template.
It is commented out as there are probably more appropriate workarounds
to use in the future but this should fix testing on centos8 based
systems.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 11 Dec 2023 19:45:46 +0000 (14:45 -0500)]
cephadm: convert init containers script to use a template
Convert the init containers run script to be based on a template, like
the sidecar run scripts are. The new script is loosely based on the
sidecars run script but only does actions in batches - logically
iterating over each init container configured.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Tue, 5 Dec 2023 21:20:57 +0000 (16:20 -0500)]
cephadm: move init container script generation to a function
Move the generation of the init container run script to a small function
fixing a missing `set -e` along the way. This isolates the logic of
generating this run script.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 21:35:06 +0000 (16:35 -0500)]
cephadm: call functions instead of executing rm in rm_cluster
Convert a bunch of invocations of rm via a subprocess to function calls.
This should make it easier (or possible?) to test the function
in the unit test framework as well as possibly saving a few resources.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 21:06:38 +0000 (16:06 -0500)]
cephadm: remove a non-functional line in _rm_cluster
Remove a line from _rm_cluster that has no effect. The line uses
call_throws to execute an `rm -rf` command. The argument contains
asterisk chars that indicate that the file(s)/dir(s) to be created are
expected to match the given pattern. However, globs do not work in this
context in contemporary versions of cephadm.
To double check I added the following temporary unit test:
```
def test_does_it_glob(tmp_path):
from cephadmlib.call_wrappers import call_throws
d1 = (tmp_path / 'dir1')
d1.mkdir()
fns = ['f1.txt', 'f2.txt', 'f3.txt', 'f4']
for fn in fns:
with open(d1/fn, "w") as fh:
fh.write("xxx")
assert d1.exists()
for fn in fns:
assert (d1 / fn).exists()
ctx = FakeContext()
call_throws(ctx, ['rm', '-rf', f'{d1}/*.txt'])
print('files:', os.listdir(d1))
assert d1.exists()
for fn in fns:
if fn.endswith('.txt'):
assert not (d1 / fn).exists()
else:
assert (d1 / fn).exists()
```
If globs worked in this context this test would have passed. It does
not. I confirmed that the current implementation of call/call_wrapper
does not execute the command in a shell context.
I wondered if it was possible that an earlier version of cephadm did
execute this command in a shell context and some changes along the way
changed the behavior. I tracked the origin of the line back to 16ebc620349f6e7c9afa6b992c85900f56fcfca3 the first change to implement
rm-cluster. In this commit the code uses subprocess.check_output
directly. I am familiar with check_output and unless `shell=True` is
passed this function doesn't execute the args in a shell context. The
`shell=True` argument is not passed to check_output. This means that the
very first implementation of this line suffered from the same issue
- it would have no effect on any files except one named with actual
asterisk (`*`) characters.
While I'm sure there was a good intention behind this line, the fact
that it persisted in the code so long in a non-functional state and no
one noticed in both production and qa testing makes me feel that it can
be safely removed with no negative effect. Removing the line simplifies
the code and avoids needing to spend effort unit-testing or manually
checking a fix.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 12:56:46 +0000 (07:56 -0500)]
cephadm: update rm cluster to remove sidecar/init-ctr services
A previous patch updated the rm-daemon functionality to remove sidecar
and init-container systemd services. This patch does the same for the
rm-cluster codepath.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 00:09:29 +0000 (19:09 -0500)]
cephadm: update _rm_cluster to call terminate_service
The terminate_service function was added to encapsulate the act of
terminating a systemd service. The rm-deamon handler was updated to use
this function previously; this commit updates rm-cluster function(s)
to do the same. This needed a bunch of test updates do to how the tests
were mocking commands.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 19:03:34 +0000 (14:03 -0500)]
cephadm: add support to remove sidecar/init-ctr systemd units
Add support to the command_rm_daemon function to remove systemd services
and corresponding unit files for any sidecars or init-containers
associated with a primary daemon.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 18:07:57 +0000 (13:07 -0500)]
cephadm: replace some calls to rm with fs function calls
This should be a little cleaner, perhaps a tiny performance improvement,
and most importantly work better with the virtual-fs provided for the
unit tests and make this function more testable in the future.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 16:50:29 +0000 (11:50 -0500)]
cephadm: minor reorgs to command_rm_daemon
Make a few small chanes to command_rm_daemon to clean it up in
preparation for future changes. Move the construction of a
DaemonIdentity earlier and use the service name derived directly from it
if possible. I didn't remove the old approach of looking up a service
name in the listing because I don't know if doing so would break
old/adopted services and there's no unit test coverage for that.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 00:00:38 +0000 (19:00 -0500)]
cephadm: add a terminate_service systemd function
Add a function that encapsulates the common 3-step process found in
cephadm of stopping, resetting errors, and disabling a service.
This will be used to handle new services to stop as well as cleaning
up some places where the three steps are performed inline.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 18:59:19 +0000 (13:59 -0500)]
cephadm: add sidecars_from_dropin function
Add a sidecars_from_dropin function that takes a minimal PathInfo object
and returns a PathInfo populated with sidecars information based on
reading the content of the dropin file. This will be useful when we need
information on configured sidecars but are not on the deploy path.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 18:57:48 +0000 (13:57 -0500)]
cephadm: add unlink_file function
Add the unlink_file function to file_utils. This fills in a gap between
python 3.6 and features provided in pathlib.Path in later versions of
python. Adds an option to ignore all errors for good measure.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Rename get_unit_name_by_daemon_name to lookup__unit_name_by_daemon_name
to emphasize that the function does a full lookup (via the daemon list)
to determine the systemd service name for the given values and that
there's a potential performance implication to doing that.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
projects / ceph.git / log