Rely on PurgeObsoleteFiles Only for Options file clean up when remote compaction is enabled (#13139)
Summary:
In PR https://github.com/facebook/rocksdb/issues/13074 , we added a logic to prevent stale OPTIONS file from getting deleted by `PurgeObsoleteFiles()` if the OPTIONS file is being referenced by any of the scheduled the remote compactions.
`PurgeObsoleteFiles()` was not the only place that we were cleaning up the old OPTIONS file. We've been also directly cleaning up the old OPTIONS file as part of `SetOptions()`: `RenameTempFileToOptionsFile()` -> `DeleteObsoleteOptionsFiles()` unless FileDeletion is disabled.
This was not caught by the UnitTest because we always preserve the last two OPTIONS file. A single call of `SetOptions()` was not enough to surface this issue in the previous PR.
To keep things simple, we are just skipping the old OPTIONS file clean up in `RenameTempFileToOptionsFile()` if remote compaction is enabled. We let `PurgeObsoleteFiles()` clean up the old options file later after the compaction is done.
Pull Request resolved: https://github.com/facebook/rocksdb/pull/13139
Test Plan:
Updated UnitTest to reproduce the scenario. It's now passing with the fix.
```
./compaction_service_test --gtest_filter="*PreservedOptionsRemoteCompaction*"
```
Reviewed By: cbi42
Differential Revision:
D65974726
Pulled By: jaykorean
fbshipit-source-id:
1907e8450d2ccbb42a93084f275e666648ef5b8c