mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands
Since we already have make use of asyncio for our ssh commands,
we can use asyncio's timeout on waiting for concurrent futures to complete
as a way to have universal timeouts on our cephadm commands.
This change also creates a contextmanager that will catch any asyncio.TimeoutError.
Using the contextmanager along with calls to the wait_async function
will catch any timeout exception raised and convert it into an appropriate
OrchetratorError including information about what and where for the timeout
if it was provided (host where run, what command). This allows us to guarantee a
background ssh command eventually returns and inform users of any
timeouts by raising a health warning or logging the error instead
of sitting idle indefinitely
Fixes: https://tracker.ceph.com/issues/54024
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit
adf13c82ebb75a132709aa0b0bbc1d4fef7643a7)
Conflicts:
src/pybind/mgr/cephadm/services/osd.py
src/pybind/mgr/cephadm/tests/test_cephadm.py