John Mulligan [Tue, 28 Mar 2023 21:07:20 +0000 (17:07 -0400)]
mgr/dashboard: ignore type checking for exception handling module
Add a `# type: ignore` comment to the exception handling dashboard
module just like the instance to lines below. This module does not
already import typing, so I'm not going to add it.
This change is needed in order to run mypy 0.981.
Adam King [Mon, 30 Jan 2023 16:27:09 +0000 (11:27 -0500)]
qa/cephadm: add check that iscsi daemon /etc/hosts matches host /etc/hosts
To make sure we aren't being affected by any podman introduced
changes to the /etc/hosts file and test that we're properly
mounting /etc/hosts in our daemon containers
Adam King [Sat, 21 Jan 2023 23:44:22 +0000 (18:44 -0500)]
cephadm: mount host /etc/hosts for containers in podman deployments
Podman messes with the /etc/hosts file in certain version. There
was already a past issue with it placing the container name
there fixed by https://github.com/ceph/ceph/pull/42242. This time
it is adding an entry for "host.containers.internal" (seems to be
podman 4.1 onward currently). Iscsi figures out the FQDN for a
host by running
which is resolving to "host.containers.internal" when run in
the container with the podman modified /etc/hosts.
There is also an issue with grafana dashboard with
this entry present
Passing --no-hosts resolves this, but I think in the past
we avoided that due to not wanting to break deployments
where host name resolution was handled using /etc/hosts.
That's why we had that workaround previously linked. This
time I'm not sure such a workaround exists. The try here
is to mount a copy of the host's version of /etc/hosts
into the iscsi container. That copy won't have the extra
entry podman adds in but will have any user created entries in
case they were actually using it for host name resolution.
If /etc/hosts file isn't present for whatever reason, we're
assuming that this user isn't using /etc/hosts for hostname
resolution, and just going back to passing --no-hosts.
Fixes: https://tracker.ceph.com/issues/58532 Fixes: https://tracker.ceph.com/issues/57018 Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit dd8627bbe3ebc6d924912a37785859d8124f95e5)
Adam King [Wed, 1 Mar 2023 15:58:43 +0000 (10:58 -0500)]
mgr/cephadm: add more aggressive force flag for host maintenance enter
Right now, we have safety checks that will either say things are okay to stop,
return warnings, or returns "alerts". Warnings can be bypassed already with
the --force flag that exists for the command. However, the alerts cannot be
bypassed and cephadm will refuse to attempt to put the host in maintenance mode.
The idea here is for an additional flag that also bypasses that alerts, in cases
where a user really needs to put the host into maintenance mode even if that
means causing issues within the cluster.
Adam King [Wed, 15 Mar 2023 17:18:02 +0000 (13:18 -0400)]
mgr/cephadm: handle HostConnectionError when checking for valid addr
Otherwise, the error is not properly passed back up the chain
and the user can get an error message like
TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr'
when trying to add a host, despite the actual problem being
cephadm.ssh.HostConnectionError: Failed to connect to vm-01 (192.168.122.248). Permission denied
The tracker shows a bit more, but generally trying to add a host
that doesn't have the proper pub-key set as an authorized key
will get a misleasing error message. With this patch, the error message looks like
[ceph: root@vm-00 /]# ceph orch host add vm-01 192.168.122.29
Error EINVAL: Failed to connect to vm-01 (192.168.122.29). Permission denied
Log: Opening SSH connection to 192.168.122.29, port 22
[conn=1] Connected to SSH server at 192.168.122.29, port 22
[conn=1] Local address: 192.168.122.156, port 49552
[conn=1] Peer address: 192.168.122.29, port 22
[conn=1] Beginning auth for user root
[conn=1] Auth failed for user root
[conn=1] Connection failure: Permission denied
[conn=1] Aborting connection
Redouane Kachach [Wed, 29 Mar 2023 08:48:30 +0000 (10:48 +0200)]
mgr/cephadm: use a dedicated cephadm tmp dir to copy remote files Fixes: https://tracker.ceph.com/issues/59189 Signed-off-by: Redouane Kachach <rkachach@redhat.com>
(cherry picked from commit ef958d47b44f8e4579ae380bd9d6890c0c62e13a)
Adam King [Wed, 15 Mar 2023 17:55:26 +0000 (13:55 -0400)]
cephadm: handle exceptions applying secondary services during bootstrap
Otherwise we risk hitting a mismatch between the cephadm binary version
and the container image version we're bootstrapping on, resulting in
bootstrap failing. Example in the tracker.
Filestore is no longer supported in cephadm and both the doc [1] and the
DriveGroupValidation [2] raise an exception if this method is used. This
patch removes the legacy code that is supposed to produce filestore
ceph-volume related commands.
Adam King [Fri, 3 Mar 2023 20:31:03 +0000 (15:31 -0500)]
mgr/prometheus: remove dependency on cephadm module
https://github.com/ceph/ceph/commit/f967ac061ebee362cdc82c458e955da75a9045e9
introduced an import of something in the cephadm module
in the prometheus module. This seems to break the prometheus
module in some non-cephadm setups. For example, the ceph-ansible
ci hit
failed: [mgr0 -> mon0] (item=prometheus) => changed=true
ansible_loop_var: item
cmd:
- ceph
- -n
- client.admin
- -k
- /etc/ceph/ceph.client.admin.keyring
- --cluster
- ceph
- mgr
- module
- enable
- prometheus
delta: '0:00:00.389965'
end: '2023-03-03 15:30:07.631308'
item: prometheus
rc: 2
start: '2023-03-03 15:30:07.241343'
stderr: 'Error ENOENT: module ''prometheus'' reports that it cannot run on the active manager daemon: No module named ''cephadm'' (pass --force to force enablement)'
stderr_lines: <omitted>
stdout: ''
stdout_lines: <omitted>
so we need to be a bit more careful with this import and
make sure the prometheus module works fine without cephadm
Adam King [Wed, 15 Feb 2023 22:07:09 +0000 (17:07 -0500)]
mgr/cephadm: be aware of host's shortname and FQDN
The idea is to gether the shortname and FQDN as part
of gather-facts, and then if we ever try to check if a certain
host is in our internal inventory by hostname, we can check
these other known names. This should avoid issues where
we think a hostname specified by FQDN is not in our
inventory because we know the host by the shortname
or vice versa.
John Mulligan [Mon, 27 Feb 2023 19:38:50 +0000 (14:38 -0500)]
cephadm: fix timeout argument to call function
The timeout argument to call function, for executing sub-processes, did
not function - this patch makes timeout work as (probably) intended.
Use the `process.communicate()` method rather than `tee` functions to
handle IO collection. Since no logging is done until after the exit code
is known the tee calls are not necessary. Add calls to kill the child
process when the time out occurs. This helps prevent event loop "leaks"
that generate python warnings.
John Mulligan [Thu, 23 Feb 2023 19:51:13 +0000 (14:51 -0500)]
cephadm/tests: add initial test coverage for call function
The call function provides the ability to run subprocesses, log output,
and provides an optional timeout parameter. This timeout parameter does
not appear to function correctly today, so we make use of
pytest.param/pytest.mark.xfail to mark these cases as already known to
fail.
John Mulligan [Wed, 22 Feb 2023 18:57:21 +0000 (13:57 -0500)]
cephadm: disable coverage for some compatibility blocks
This change disables reporting missing coverage for blocks that
contain copy and pasted code from other python versions and exist
to make those functions available to older python versions.
drive_group: fix limit filter in drive_selection.selector
When multiple osd service specs with 'limit' filter are applied,
the current logic makes the second service speec
try to pick devices that are already used by the first service spec.
doc/start: edit first 50 lines of documenting-ceph
Edit the first 150 lines of doc/start/documenting-ceph.rst. This is part
of an initiative to harvest the fruits of Cephalocon 2023, at which
documentation proved to be in demand to a surprising degree.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit dd37f94aa4f1de947b1eaf5d82cc529925f5823e)
Conrad Hoffmann [Wed, 22 Mar 2023 22:03:57 +0000 (23:03 +0100)]
doc: account for PG autoscaling being the default
The current documentation tries really hard to convince people to set
both `osd_pool_default_pg_num` and `osd_pool_default_pgp_num` in their
configs, but at least the latter has undesirable side effects on any
Ceph version that has PG autoscaling enabled by default (at least quincy
and beyond).
Assume a cluster with defaults of `64` for `pg_num` and `pgp_num`.
Starting `radosgw` will fail as it tries to create various pools without
providing values for `pg_num` or `pgp_num`. This triggers the following
in `OSDMonitor::prepare_new_pool()`:
- `pg_num` is set to `1`, because autoscaling is enabled
- `pgp_num` is set to `osd pool default pgp_num`, which we set to `64`
- This is an invalid setup, so the pool creation fails
Likewise, `ceph osd pool create mypool` (without providing values for
`pg_num` or `pgp_num`) does not work.
Following this rationale:
- Not providing a default value for `pgp_num` will always do the right
thing, unless you use advanced features, in which case you can be
expected to set both values on pool creation
- Setting `osd_pool_default_pgp_num` in your config breaks pool creation
for various cases
This commit:
- Removes `osd_pool_default_pgp_num` from all example configs
- Adds mentions of the autoscaling and how it interacts with the default
values in various places
For each file that was touched, the following maintenance was also
performed:
- Change interternal spaces to underscores for config values
- Remove mentions of filestore or any of its settings
- Fix minor inconsistencies, like indentation etc.
There is also a ticket which I think is very relevant and fixed by this,
though it only captures part of the broader issue addressed here:
qa/suites/rbd: install qemu-utils in addition to qemu-block-extra on Ubuntu
qemu-utils is usually pre-installed but, due to what appears to be
a Ubuntu packaging bug, it's not upgraded when qemu-block-extra is
installed:
The following NEW packages will be installed:
qemu-block-extra
The following packages will be upgraded:
qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86
However, the version of the block driver must match exactly the version
of the qemu-img tool, so the above leads to:
$ qemu-img convert -f qcow2 -O raw /home/ubuntu/cephtest/qemu/base.client.0.0.qcow2 rbd:rbd/client.0.0
Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so
Note: only modules from the same build can be loaded.
qemu: module block-block-rbd not found, do you want to install qemu-block-extra package?
qemu-img: Unknown protocol 'rbd'
```
error: /var/cache/dnf/baseos-00fe51d07def85f0/packages/kernel-core-4.18.0-483.el8.x86_64.rpm: signature hdr data: BAD, no. of bytes(459772) out of range
```
ceph-volume tests are failing, OSDs never get up and running.
For some reason, updating the OS early in the testing workflow
addresses that issue in the CI.
Remove confusing parentheses and add a clearer (as compared to the
parentheses) hyphen (actually an em-dash, or at least it is intended
to be an em-dash) to doc/rados/operations/monitoring-osd-pg.rst
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0c965c18d0e6ab1461b5fad42d481f25e4207940)
Ilya Dryomov [Tue, 28 Mar 2023 18:03:05 +0000 (20:03 +0200)]
librbd: avoid generating ESHUTDOWN in ManagedLock
EBLOCKLISTED has a very special meaning but happens to be an alias for
ESHUTDOWN. If the client gets blocklisted, we always want to propagate
EBLOCKLISTED error code since it's generated by the OSD.
For ManagedLock use case of indicating that an operation on the lock
raced with lock shut down, meaning that a higher level request can just
be restarted, ERESTART should do.
projects / ceph-ci.git / log