]> git.apps.os.sepia.ceph.com Git - ceph.git/commit
mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands 51613/head
authorAdam King <adking@redhat.com>
Mon, 20 Mar 2023 19:31:12 +0000 (15:31 -0400)
committerAdam King <adking@redhat.com>
Mon, 22 May 2023 20:42:36 +0000 (16:42 -0400)
commitc3e12a319661ee4422160352ec0bb236900cda51
treecb7863ae5d0421a0062b471577e9e2f6dd9e0f13
parent69232d72828b3b960b7d4cacedd5cf25c23fc733
mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Since we already have make use of asyncio for our ssh commands,
we can use asyncio's timeout on waiting for concurrent futures to complete
as a way to have universal timeouts on our cephadm commands.
This change also creates a contextmanager that will catch any asyncio.TimeoutError.
Using the contextmanager along with calls to the wait_async function
will catch any timeout exception raised and convert it into an appropriate
OrchetratorError including information about what and where for the timeout
if it was provided (host where run, what command). This allows us to guarantee a
background ssh command eventually returns and inform users of any
timeouts by raising a health warning or logging the error instead
of sitting idle indefinitely

Fixes: https://tracker.ceph.com/issues/54024
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit adf13c82ebb75a132709aa0b0bbc1d4fef7643a7)
src/pybind/mgr/cephadm/module.py
src/pybind/mgr/cephadm/serve.py
src/pybind/mgr/cephadm/services/osd.py
src/pybind/mgr/cephadm/ssh.py
src/pybind/mgr/cephadm/tests/fixtures.py
src/pybind/mgr/cephadm/tests/test_cephadm.py
src/pybind/mgr/cephadm/upgrade.py