If we have straggler pgs and blackhole osd.1, we can deadlock because we
need info from that osd to repeer and continue. Make sure we're clean, and
then start the write + blackhole + kill test.
manager.kill_osd(0)
manager.mark_down_osd(0)
+ # wait for everything to peer and be happy...
+ manager.raw_cluster_cmd('tell', 'osd.1', 'flush_pg_stats')
+ manager.raw_cluster_cmd('tell', 'osd.2', 'flush_pg_stats')
+ manager.wait_till_clean()
+
# write some new data
p = rados_start(mon, ['-p', 'data', 'bench', '30', 'write', '-b', '4096'])