osd/ReplicatedPG: allow whiteout deletion with IGNORE_CACHE flag

osd/ReplicatedPG: allow whiteout deletion with IGNORE_CACHE flag

If the client specifies IGNORE_CACHE, allow a regular DELETE to zap a
whiteout.  Expand test case to verify this works.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 34e4d24)

    Conflicts:
        src/test/librados/tier.cc
            replaced NObjectIterator -> ObjectIterator
            replaced cache_ioctx.nobjects_begin -> cache_ioctx.objects_begin
            replaced cache_ioctx.nobjects_end -> cache_ioctx.objects_end
            replace it->get_oid() with it->first for it of type ObjectIterator