os/bluestore: allow multiple DeferredBatches in flight at once

os/bluestore: allow multiple DeferredBatches in flight at once

The current code only allows two DeferredBatches per osr: one that is
accumulating new writes and one that is currently in flight to disk.  If
the previous batch is in flight ot disk, then the currently accumulating
one can't also be queued for disk.

This can cause problems, notably that described in
http://tracker.ceph.com/issues/20295, where one transaction is trying to
grab deferred_throttle but cannot due to other in-progress txcs that
include deferred IO.  The short version is that it cannot queue all of the
IO needed, and that later when the IO does complete it is awkward to
determine whether other IO queued behind it also needs to be queued
immediately.  And since it's not, this leads to a deadlock/stall.

Simply allowing multiple batches of IO to be in flight at once is a simple
fix.  Specifically, in queue_transactions(), if throttle_deferred_bytes
get_or_fail() fails, we can now deferred_submit_all() and be sure that
other IO will be submitted and thus complete and release the throttle that
we need to continue.

It is possible that the deferred_aggressive behavior could be simplified
now that the old restriction is dropped, but that needs a closer review
of the code.

Fixes: http://tracker.ceph.com/issues/20295
Signed-off-by: Sage Weil <sage@redhat.com>