Adam King [Wed, 29 Nov 2023 16:49:38 +0000 (11:49 -0500)]
qa/upgrade/reef-x: pull compiled cephadm to start upgrades from reef
The compiled zipapp cephadm that began in reef needs
to be pulled differently than the old single python script
cephadm from earlier releases. This commit updates the reef-x
upgrade suite to pull cephadm in this new way.
Adam King [Wed, 16 Aug 2023 23:56:38 +0000 (19:56 -0400)]
qa/cephadm: support to pull stable branch compiled cephadm
This is to allow us to pull the latest build of
cephadm off of a stable branch (currently the only
valid option for that is reef, although this hopefully
will work with squid, T release, etc. in the future).
This should allow us to bootstrap cliusters based on
those stable branches for use in upgrade testing
Ronen Friedman [Thu, 28 Dec 2023 19:41:19 +0000 (13:41 -0600)]
osd/scrub: avoid "over clearing" queued_or_active flag
If two StartScrub messages are received in quick succession, the earlier
one might clear the queued_or_active flag as it fails for being from an old
interval.
When that happens - a 3'rd scrub request will actually be allowed to go
through, while the scrubber is still handling the second one.
Myoungwon Oh [Sun, 17 Dec 2023 08:51:22 +0000 (17:51 +0900)]
crimson/os/seastore: introduce modified_region in DATA_BLOCK to keep track of modified region
It has a limitation to keep track of the modified region using the existing
deltas because we can not get the correct region in two cases: 1) a case where replay
is done and 2) duplicate_for_write. This commit introduces modified region
to solve the problem.
Matan Breizman [Sun, 1 Jan 2023 11:41:34 +0000 (11:41 +0000)]
crimson/osd: Keep track of modified_ranges
* `modifies_ranges` interval_set is added to osd_op_params_t
* keep track of modified_ranges while executing relevant ops
* Add `osd_op_params` parameter to `PGBackend::remove()`.
Zac Dover [Wed, 3 Jan 2024 08:41:51 +0000 (18:41 +1000)]
doc/radosgw: edit "Add/Remove a Key"
Edit the section "Add/Remove a Key" in doc/radosgw/admin.rst. Each
operation (e.g. "Adding an S3 key pair for a user", "Removing an S3 key
pair for a user") now has its own subsection. This increased granularity
should make it easier in the future to link to each of these specific
operations, if needed.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
John Mulligan [Tue, 19 Dec 2023 19:10:39 +0000 (14:10 -0500)]
cehphadm: disable restarting init containers service
Disable the Restart= line in the init containers systemd template.
It is commented out as there are probably more appropriate workarounds
to use in the future but this should fix testing on centos8 based
systems.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 11 Dec 2023 19:45:46 +0000 (14:45 -0500)]
cephadm: convert init containers script to use a template
Convert the init containers run script to be based on a template, like
the sidecar run scripts are. The new script is loosely based on the
sidecars run script but only does actions in batches - logically
iterating over each init container configured.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Tue, 5 Dec 2023 21:20:57 +0000 (16:20 -0500)]
cephadm: move init container script generation to a function
Move the generation of the init container run script to a small function
fixing a missing `set -e` along the way. This isolates the logic of
generating this run script.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 21:35:06 +0000 (16:35 -0500)]
cephadm: call functions instead of executing rm in rm_cluster
Convert a bunch of invocations of rm via a subprocess to function calls.
This should make it easier (or possible?) to test the function
in the unit test framework as well as possibly saving a few resources.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 21:06:38 +0000 (16:06 -0500)]
cephadm: remove a non-functional line in _rm_cluster
Remove a line from _rm_cluster that has no effect. The line uses
call_throws to execute an `rm -rf` command. The argument contains
asterisk chars that indicate that the file(s)/dir(s) to be created are
expected to match the given pattern. However, globs do not work in this
context in contemporary versions of cephadm.
To double check I added the following temporary unit test:
```
def test_does_it_glob(tmp_path):
from cephadmlib.call_wrappers import call_throws
d1 = (tmp_path / 'dir1')
d1.mkdir()
fns = ['f1.txt', 'f2.txt', 'f3.txt', 'f4']
for fn in fns:
with open(d1/fn, "w") as fh:
fh.write("xxx")
assert d1.exists()
for fn in fns:
assert (d1 / fn).exists()
ctx = FakeContext()
call_throws(ctx, ['rm', '-rf', f'{d1}/*.txt'])
print('files:', os.listdir(d1))
assert d1.exists()
for fn in fns:
if fn.endswith('.txt'):
assert not (d1 / fn).exists()
else:
assert (d1 / fn).exists()
```
If globs worked in this context this test would have passed. It does
not. I confirmed that the current implementation of call/call_wrapper
does not execute the command in a shell context.
I wondered if it was possible that an earlier version of cephadm did
execute this command in a shell context and some changes along the way
changed the behavior. I tracked the origin of the line back to 16ebc620349f6e7c9afa6b992c85900f56fcfca3 the first change to implement
rm-cluster. In this commit the code uses subprocess.check_output
directly. I am familiar with check_output and unless `shell=True` is
passed this function doesn't execute the args in a shell context. The
`shell=True` argument is not passed to check_output. This means that the
very first implementation of this line suffered from the same issue
- it would have no effect on any files except one named with actual
asterisk (`*`) characters.
While I'm sure there was a good intention behind this line, the fact
that it persisted in the code so long in a non-functional state and no
one noticed in both production and qa testing makes me feel that it can
be safely removed with no negative effect. Removing the line simplifies
the code and avoids needing to spend effort unit-testing or manually
checking a fix.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 12:56:46 +0000 (07:56 -0500)]
cephadm: update rm cluster to remove sidecar/init-ctr services
A previous patch updated the rm-daemon functionality to remove sidecar
and init-container systemd services. This patch does the same for the
rm-cluster codepath.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 00:09:29 +0000 (19:09 -0500)]
cephadm: update _rm_cluster to call terminate_service
The terminate_service function was added to encapsulate the act of
terminating a systemd service. The rm-deamon handler was updated to use
this function previously; this commit updates rm-cluster function(s)
to do the same. This needed a bunch of test updates do to how the tests
were mocking commands.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 19:03:34 +0000 (14:03 -0500)]
cephadm: add support to remove sidecar/init-ctr systemd units
Add support to the command_rm_daemon function to remove systemd services
and corresponding unit files for any sidecars or init-containers
associated with a primary daemon.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 18:07:57 +0000 (13:07 -0500)]
cephadm: replace some calls to rm with fs function calls
This should be a little cleaner, perhaps a tiny performance improvement,
and most importantly work better with the virtual-fs provided for the
unit tests and make this function more testable in the future.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 16:50:29 +0000 (11:50 -0500)]
cephadm: minor reorgs to command_rm_daemon
Make a few small chanes to command_rm_daemon to clean it up in
preparation for future changes. Move the construction of a
DaemonIdentity earlier and use the service name derived directly from it
if possible. I didn't remove the old approach of looking up a service
name in the listing because I don't know if doing so would break
old/adopted services and there's no unit test coverage for that.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 19 Nov 2023 00:00:38 +0000 (19:00 -0500)]
cephadm: add a terminate_service systemd function
Add a function that encapsulates the common 3-step process found in
cephadm of stopping, resetting errors, and disabling a service.
This will be used to handle new services to stop as well as cleaning
up some places where the three steps are performed inline.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 18:59:19 +0000 (13:59 -0500)]
cephadm: add sidecars_from_dropin function
Add a sidecars_from_dropin function that takes a minimal PathInfo object
and returns a PathInfo populated with sidecars information based on
reading the content of the dropin file. This will be useful when we need
information on configured sidecars but are not on the deploy path.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 18 Nov 2023 18:57:48 +0000 (13:57 -0500)]
cephadm: add unlink_file function
Add the unlink_file function to file_utils. This fills in a gap between
python 3.6 and features provided in pathlib.Path in later versions of
python. Adds an option to ignore all errors for good measure.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Rename get_unit_name_by_daemon_name to lookup__unit_name_by_daemon_name
to emphasize that the function does a full lookup (via the daemon list)
to determine the systemd service name for the given values and that
there's a potential performance implication to doing that.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 17 Nov 2023 20:45:55 +0000 (15:45 -0500)]
cepahdm: avoid subclass related issues for data_dir
Do not have data dir depend on the daemon_name property. This property
is overriden by the subclass(es) but we want DaemonIdentities and
DaemonSubIdentities to share the same data_dir.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 17 Nov 2023 16:23:47 +0000 (11:23 -0500)]
cephadm: use service_name method of daemon identity
Use the new service_name method of the DaemonIdentity type in some basic
cases where string concatenation was being used.
I also converted the name of the agent's helper method to match and made
it private to avoid future confusion.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 29 Sep 2023 18:47:10 +0000 (14:47 -0400)]
cephadm: add systemd name generation to daemon identity
There are a bunch of places in the code that wants the full systemd
service name but the code generally is just doing string concatenation.
Add a method to avoid stringing it all the time. This will help as there
will be more systemd work in the future.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
projects / ceph.git / log