test/librbd: avoid caching effects TestGroup.add_snapshot{,PP}

test/librbd: avoid caching effects TestGroup.add_snapshot{,PP}

After rollback started being tested in commit b3977c53c930
("test/librbd: make rollback in TestGroup.add_snapshot{,PP}
meaningful"), these tests can fail on comparing post-rollback
data to expected data if run with exclusive lock disabled.
This doesn't occur with exclusive lock enabled because the RBD
cache gets invalidated implicitly before releasing the lock.

While at it, pass LIBRADOS_OP_FLAG_FADVISE_FUA to avoid relying
on any cache settings that happen to be in effect.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit a7e314c2ef899406eb3ade6c5f862622ce1e698f)