crimson/osd: clear peer_missing entries for deleted objects
This commit fixes the abort in Recovered::Recovered.
There is a race to acquire the OBC lock between backfill and
client delete for the same object.
When the lock is acquired first by the backfill, the object is
recovered first, and then deleted by the client delete request.
When recovering the object, the corresponding peer_missing entry
is cleared and we are able to transition to Recovered state
successfully.
When the lock is acquired first by client delete request, the
object is deleted. Then backfill tries to recover the object,
finds it deleted and exists early. The stale peer_missing
entry is not cleared. In Recovered::Recovered, needs_recovery()
sees this stale peer_missing entry and calls abort.
This issue is fixed by clearing out the stale peer_missing
entries for the deleted object when object is being recovered
by recover_object().
Earlier, an alternative fix was to clear the stale
peer_missing entries in enqueue_delete_for_backfill called by
the client delete. This approach was abandoned as this can also
lead to a race condition which might result in this same error,
if the bckfill completes before the entries are cleared in
enqueue_delete_for_backfill().