The wait_for_ondisk handling fixed COMMIT ordering, but the ACKs need to
go back in the same order too. For example:
- op A is queued
- client disconnects, both ACK and COMMIT replies are lost
- client reconnects
- op A and B are sent
- op A is queued
- op B is applied, ACK is sent
- op A and B COMMITs are sent
-> client's ack callbacks will see B and then A.
Fix this by creating a waiting_for_ack queue as well, and sending ACK
responses as needed. Also handle the case where the ACK should be sent
immediately when the retry event is received.
Fixes: #2823 Signed-off-by: Sage Weil <sage@inktank.com> Reviewed-by: Mike Ryan <mike.ryan@inktank.com>