PG,OSD: delay ops for map prior to queueing in the OpWQ
Previously, we simply queued ops in the OpWQ without checking. The PG
would then check in do_request whether the message should wait for a new
map. Unfortunately, this has the side effect that any op requeued for
any reason must also requeue the waiting_for_map queue.
Now, we will check before queueing the op whether it must wait on a map.
To avoid contention, there is now a map_lock which must be held along
with the PG lock in order to update the osdmap_ref. The map_lock also
protects the waiting_for_map list and queueing PG ops at the back of
the OpWQ. A few details:
1) It is no longer necessary to requeue waiting_for_map in on_change()
since the other ops are queued at the front.
2) Once waiting_for_map is non-empty, all ops are delayed to simplify
ordering.
3) waiting_for_map may now be non-empty during split, so we must split
waiting_for_map along with waiting_for_active. This must be done
under the map_lock.
The bug which uncovered this involved an out of order op as follows:
client.4208.0:2378 (e252) arrives, object is degraded
client.4208.0:2379 (e253) arrives, waits for map
client.4208.0:2378 (e252) is requeued after recovery
client.4208.0:2379 (e253) is requeued on map arrival
client.4208.0:2379 is processed
client.4208.0:2378 is processed
Fixes: #4955 Signed-off-by: Samuel Just <sam.just@inktank.com>