]> git.apps.os.sepia.ceph.com Git - ceph.git/commit
mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands 51218/head
authorAdam King <adking@redhat.com>
Mon, 20 Mar 2023 19:31:12 +0000 (15:31 -0400)
committerAdam King <adking@redhat.com>
Wed, 24 May 2023 17:55:01 +0000 (13:55 -0400)
commitcb2b1f24d3c5413ab0667d77508c9c95a6435b79
tree0f7c2603fcff786b294cf25c3fdad67a386861f9
parent137bd6bb57259376ef0419731c26782d09dd1d2f
mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Since we already have make use of asyncio for our ssh commands,
we can use asyncio's timeout on waiting for concurrent futures to complete
as a way to have universal timeouts on our cephadm commands.
This change also creates a contextmanager that will catch any asyncio.TimeoutError.
Using the contextmanager along with calls to the wait_async function
will catch any timeout exception raised and convert it into an appropriate
OrchetratorError including information about what and where for the timeout
if it was provided (host where run, what command). This allows us to guarantee a
background ssh command eventually returns and inform users of any
timeouts by raising a health warning or logging the error instead
of sitting idle indefinitely

Fixes: https://tracker.ceph.com/issues/54024
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit adf13c82ebb75a132709aa0b0bbc1d4fef7643a7)

Conflicts:
src/pybind/mgr/cephadm/services/osd.py
src/pybind/mgr/cephadm/tests/test_cephadm.py
src/pybind/mgr/cephadm/module.py
src/pybind/mgr/cephadm/serve.py
src/pybind/mgr/cephadm/services/osd.py
src/pybind/mgr/cephadm/ssh.py
src/pybind/mgr/cephadm/tests/fixtures.py
src/pybind/mgr/cephadm/tests/test_cephadm.py
src/pybind/mgr/cephadm/upgrade.py