msg/simple: do not stop_and_wait on mark_down

msg/simple: do not stop_and_wait on mark_down

We originally blocked in mark_down for fast dispatch threads
to complete to avoid various races in the code.  Most of these
were in the OSD itself, where we were not prepared to get
messges on connections that had no attached session.  Since
then, the OSD checks have been cleaned up to handle this.
There were other races we were worried about too, but the
details have been lost in the depths of time.

Instead, take the other route: make mark_down never block on
dispatch.  This lets us remove the special case that
was added in order to cope with fast dispatch calling
mark_down on itself.

Now, the only stop_and_wait() user is the shutdown sequence.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 00907e032011b9d2acd16ea588555cf379830814)