qa/workunits/rbd: avoid caching effects in luks-encryption.sh

qa/workunits/rbd: avoid caching effects in luks-encryption.sh

Commit 40f6f5224bce ("qa/workunits/rbd: fix issues in
luks-encryption.sh") did the right thing for reads, which solved
most of the issue.  However, it actually made a step in the opposite
direction for writes -- depending on the RBD cache settings, rbd-nbd
virtual devices can behave as physical devices with a volatile write
cache, so fsync is required.

While at it, involving O_DIRECT for reads isn't needed outside of
test_encryption_format().

Fixes: https://tracker.ceph.com/issues/67051
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 9e1c0342e9931e898d795d47f923edd3b4324f4a)

Conflicts:
	qa/workunits/rbd/luks-encryption.sh [ commit 1d3de19c4005
	  ("tools/rbd: add encryption format support for cloned image")
	  not in quincy ]