]> git.apps.os.sepia.ceph.com Git - ceph.git/commit
mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands 50722/head
authorAdam King <adking@redhat.com>
Mon, 20 Mar 2023 19:31:12 +0000 (15:31 -0400)
committerAdam King <adking@redhat.com>
Tue, 25 Apr 2023 14:13:50 +0000 (10:13 -0400)
commitadf13c82ebb75a132709aa0b0bbc1d4fef7643a7
tree725ab5aefb89ca18f1de93ea2c68b916d57f2309
parent00d4203ec60ed42840b1b318af49e4e697bd04f2
mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Since we already have make use of asyncio for our ssh commands,
we can use asyncio's timeout on waiting for concurrent futures to complete
as a way to have universal timeouts on our cephadm commands.
This change also creates a contextmanager that will catch any asyncio.TimeoutError.
Using the contextmanager along with calls to the wait_async function
will catch any timeout exception raised and convert it into an appropriate
OrchetratorError including information about what and where for the timeout
if it was provided (host where run, what command). This allows us to guarantee a
background ssh command eventually returns and inform users of any
timeouts by raising a health warning or logging the error instead
of sitting idle indefinitely

Fixes: https://tracker.ceph.com/issues/54024
Signed-off-by: Adam King <adking@redhat.com>
src/pybind/mgr/cephadm/module.py
src/pybind/mgr/cephadm/serve.py
src/pybind/mgr/cephadm/services/osd.py
src/pybind/mgr/cephadm/ssh.py
src/pybind/mgr/cephadm/tests/fixtures.py
src/pybind/mgr/cephadm/tests/test_cephadm.py
src/pybind/mgr/cephadm/upgrade.py