John Mulligan [Mon, 26 Feb 2024 21:17:22 +0000 (16:17 -0500)]
qa/tasks: add a template filter to map a role name to a remote
Add a `role_to_remote` template filter function that has the ability to
map a role name to a remote. Attributes of the remote can then be
used to get the actual node ip or name.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 26 Feb 2024 21:16:57 +0000 (16:16 -0500)]
qa/tasks: a new cephadm exec task similar to vip.exec but generalized
Add a new cephadm.exec task that works similarly to the existing
vip.exec but instead of only considering VIP related string replacements
it uses that templating feature that was recently added to the
cephadm module for generalized string templating.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 26 Feb 2024 18:47:04 +0000 (13:47 -0500)]
qa/tasks: add a cephadm.exclude role
Add a cephadm.exclude role that excludes a test node from cluster setup
and related commands. I need this as I have test node that will be set
up as an AD Domain Controller for testing Samba and do not want that
node to be have *any* other services running on it.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 24 Feb 2024 19:26:36 +0000 (14:26 -0500)]
qa/tasks: allow passing stdin string to cephadm shell commands
There are cases where I want to pass some large-ish strings to ceph
commands executed via cephadm shell. Allow items within the commands
list to be dicts containing a command (as before) and an optional
stdin variable. This change also supports possible future extensions as
well.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Tue, 20 Feb 2024 23:28:58 +0000 (18:28 -0500)]
qa/tasks: add a new cephadm task for setting up samba ad dc
Add a new task function to cephadm.py that sets up a container running
the Samba based domain controller on a node using podman or docker.
Much of the function actually deals with disabling systemd-resolved
because that service conflicts with the DNS server component of the DC.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 5 Jan 2024 15:45:08 +0000 (10:45 -0500)]
mgr/cephadm: simplify _get_container_image a bit
Because the "if-ladder" was only ever assigning a single variable with
a value it can be directly replaced by a dict & dict-lookup which is
much more succinct.
Also take the opportunity to sort the (non-comment) lines as there's
no meaning to the previous order and this makes it easier for a reader
to scan through.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 4 Jan 2024 21:38:08 +0000 (16:38 -0500)]
mgr/cepahdm: add various touch points to enable smb service
Add the smb service by name or by type to one of the many, many touch
points in the orchestrator and cephadm packages needed to get the
orchestrator aware of smb.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 14 Dec 2023 00:20:45 +0000 (19:20 -0500)]
python-common: reformat ServiceSpec class level service type lists
Reformat the ServiceSpec classes properties KNOWN_SERVICE_TYPES and
REQUIRES_SERVICE_ID. These were previously strings that were converted
to lists via a call to split. With a string there's very little a human
or a tool can do to validate the content. Changing these into proper
lists in the source code brings clarity of intent and the ability to
analyze the code. Because there's no semantic difference what services
are listed where (this means the type could probably be a set - a quest
for another day) I also took the opportunity to sort the contents of the
lists and add some basic comments for what these lists are for.
It also removes the use of (ugly, IMO) line continuations. The downside
is that it makes more total lines, but if that bugs you - use code
folding :-).
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 5 Jan 2024 15:24:10 +0000 (10:24 -0500)]
mgr/cephadm: refactor keyring simplification out of get_keyring_with_caps
Refactor get_keyring_with_caps such that the keyring simplification code
is moved into a new function that can be used in other locations.
get_keyring_with_caps will now call the new function to return the
simplified & consistent keyring output.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 13 Dec 2023 20:49:12 +0000 (15:49 -0500)]
mgr/cephadm: reformat the _service_classes variable
Reformat the _service_classes variable so that it uses a multi-line list
with a single item on each line in a more black-ish style that is more
readable (especially if you use code-folding wisely).
Sort the list while we're at it.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 13 Dec 2023 21:05:27 +0000 (16:05 -0500)]
mgr/orchestrator: fix the sorting of the imports
While ceph doesn't enforce sorted imports I prefer them when possible. I
had once sorted these imports but then nvmeof came along an ruined
things. Put nvmeof back in it's place.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 13 Dec 2023 19:33:20 +0000 (14:33 -0500)]
mgr/cephadm: fix test failure on newer python
Tests that touch this enum fail for me locally but pass in the CI. This
seems to be due to new enum related behavior in Python 3.11.
See: https://blog.pecar.me/python-enum
Instead of fixing it as suggested in the above blog, adding a __str__
method works on all python versions I care to know about.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Tue, 16 Jan 2024 20:37:27 +0000 (15:37 -0500)]
cephadm: fix issue joining to ad by using a virtual hostname
The not-a-real-fqdn hostname that the containers got were causing
performance issues joining AD (and running testjoin and winbind).
Define a virtual hostname that can be passed in from the service or
automatically derived from the system's hostname.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 6 Dec 2023 20:14:32 +0000 (15:14 -0500)]
cephadm: import and enable deployment of SMB daemon class
Enable the use of the SMB container daemon form class by importing, and
thus registering, it. Note that the only way to invoke this feature is
by hand rolling some JSON to feed to the `ceph _orch deploy` command.
Connecting this with the cephadm mgr module is left as a future task.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 6 Dec 2023 20:14:31 +0000 (15:14 -0500)]
cephadm: add an SMB daemon module and classes
Add an incomplete but largely viable SMB/Samba container daemon form
implementation to cephadm. Currently unused but it lays out some of the
basics needed to create smb sharing using samba containers under cephadm
orchestration.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 3 Dec 2023 16:01:05 +0000 (11:01 -0500)]
cephadm: add generic methods for sharing namespaces across containers
In the future, some sidecar containers will need to share namespaces
with the primary container (or each other). Make it easy to set this up
by creating a enable_shared_namespaces function and Namespace enum.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Vallari Agrawal [Tue, 20 Feb 2024 07:44:32 +0000 (13:14 +0530)]
qa/suite/rbd/nvmeof: Deploy multiple gateways and namespaces
1. Deploy 2 gateways on different nodes, then check for multi-path.
To add another gateway, only "roles" need to be changed in job yaml.
2. Create "n" nvmeof namespaces, configured by 'namespaces_count'
3. Rename qa/suites/rbd/nvmeof/cluster/fixed-3.yaml to fixed-4.yaml
which contains 2 gateways and 2 initiators.
Sachin Punadikar [Tue, 19 Mar 2024 09:41:53 +0000 (05:41 -0400)]
vstart: Ganesha should not be started in DEBUG mode
Currently vstart script, deploy NFS Ganesha in debug mode. Enabling
DEBUG mode for Ganesha leads to logging lot of debug messages, which may
not be required all the time. One can enable DEBUG mode on need basis.
Hence removing the default DEBUG mode.
myoungwon oh [Mon, 18 Mar 2024 06:48:07 +0000 (06:48 +0000)]
crimson/os/seastore: cache metadata during trimming to prevent from disk read
I encountered continous disk reads during trimming even though there are sufficient
cache available, in 4K random write test with RBM (RBD).
This is because metadata is note cached if its source is background transaction
within touch_extent(). So, seastore, including the trimming process, needs to
constantly retrieve metadata (e.g., BACKREF_LEAF).
Based on the previous commits making the remote executables auditable
and explicit, document the admin's ability to restrict password-less
sudo access to only the set of commands cephadm actually uses.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 14 Mar 2024 18:02:17 +0000 (14:02 -0400)]
mgr/cephadm: add a simple unit test for RemoteCommand class
Converting a remote command to something that other libs uses requires
converting the enum to a string. Python behavior in the area varies
across versions so add a unit test that verifies the conversion
behaves as intended.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 14 Feb 2024 16:35:57 +0000 (11:35 -0500)]
mgr/cephadm: make remote command execution auditable
Update ssh.py and other code using it to only allow commands wrapped
in particular python types as executables on the remote hosts.
By using a specific type for remote executables we make the code more
auditable, avoiding the possibility of executing arbitrary strings
as commands with sudo. This is all enforced by mypy's type checking.
The result is a list of commands that the cephadm mgr module may
execute on a remote host using sudo:
```
$ git ls-files -z | xargs -0 grep 'RemoteExecutable(' -d skip -h | grep
-v '(str)' | sed -e 's/.*RemoteExecutable(//' -e 's/)//' -e 's/,$//'
'which'
'/usr/bin/cephadm'
python
'chmod'
'ls'
'sysctl'
'chown'
'mkdir'
'mv'
'touch'
'rm'
'true'
```
Note that *python* is special as it is based on the output of which and
may vary from OS to OS. The quoted items are used exactly as named.
Only the binary at `/usr/bin/cephadm` _or_ the dynamically discovered
python3 binary will be used. This depends on a configuration option for
the cephadm module.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Adam King [Wed, 13 Mar 2024 19:30:25 +0000 (15:30 -0400)]
mgr/cephadm: refresh public_network for config checks before checking
The place it was being run before meant it would only grab the
public_network setting once at startup of the module. This meant
if a user changed the setting, which they are likely to do if they
get the warning, cephadm would ignore the change and continue
reporting that the hosts don't match up with the old setting
for the public_network. This moves the call to refresh the
setting to right before we actually run the checks. It does
mean we'll do the `ceph config dump --format json` call
each serve loop iteration, but I've found that only tends
to take a few milliseconds, which is nothing compared to
the time to refresh other things we check during the serve
loop.
I additionally modified the use of this option to use
the attribute on the mgr, rather than calling
`get_module_option`. This was just to get it more in
line with how we tend to handle other config options
Fixes: https://tracker.ceph.com/issues/64902 Signed-off-by: Adam King <adking@redhat.com>
Adam King [Tue, 12 Mar 2024 14:26:18 +0000 (10:26 -0400)]
cephadm: fix `cephadm shell --name <daemon-name>` for stopped/failed daemon
This previously would always try to use 'podman
inspect' on the running container of the daemon,
but this doesn't work if the daemon is stopped
or failed. Doing this for stopped/failed daemons
is a valid use case as we recommend cephadm shell
with --name for running debugging tools (often
for OSDs)
Fixes: https://tracker.ceph.com/issues/64879 Signed-off-by: Adam King <adking@redhat.com>
Adam King [Mon, 11 Mar 2024 18:44:17 +0000 (14:44 -0400)]
cephadm: allow list_daemons for only a specific daemon
At the moment, my thoughts are to use this internally
in the binary for when we need infor from list_daemons
but only for a specific daemon. I could also see wanting
this just on the command line to get info on a certain
daemon, so I've added it as a flag for `cephadm ls` as well
After some tests, it turns out that depending on the hardware,
the header 'Location' which is returned by the server after logged can be different.
I could notice the following:
projects / ceph.git / log