cephadm: Detect stale and then recreate connections
Currently we make and cache connections to nodes during a check_host.
If a cached connection is disconnect from the other end the remoto
connection object doesn't track this, so further checks to the host
fail.
I have pushed up a PR[0] to remoto to add a `has_connection` method to
their `BaseConnection` class, which we now use in this patch to check to
see if the connection is stale. If it is it is then recreated.
There is some monkey patching happening so we can add the required
`has_connection` to remoto in this patch which we can remove as soon as
the other PR have landed and a new version of remoto is released.
[0] https://github.com/alfredodeza/remoto/pull/56
Fixes: https://tracker.ceph.com/issues/45627
Fixes: https://tracker.ceph.com/issues/45032
Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit
afabf82395361f5d2209a980578b4b6715d80f3d)