os/bluestore/bluefs: Remove possibility of bluefs replay log containing files without...

os/bluestore/bluefs: Remove possibility of bluefs replay log containing files without data

It had been possible to have a bluefs replay log to serialize file metadata (size, allocations),
but actual data stored in these allocations is not yet synced to disk.

This could happen if _flush_range(h1) allocated space for file h1 on device (like SLOW) that will not
be used when flushing future replay log. Such thing can happen when we have h2 that wrote to WAL and
out replay log is on DB. After fsync(h2) we write to replay log, wait for fdatasync on WAL and DB.
There is no waiting on SLOW, but h1 was dirty and has been serialized to replay log.

Solution is to delay notifying replay log that it has to include h1 after finishing fdatasync.

Cherry-picked from: 03ac53f7d4c83e56f664ad371ffe3bc2d40e1837
Fixes: https://tracker.ceph.com/issues/51129
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Conflicts:
	src/os/bluestore/BlueFS.cc
	src/os/bluestore/BlueRocksEnv.cc