]> git-server-git.apps.pok.os.sepia.ceph.com Git - rocksdb.git/commit
Preserve Options File (#13074)
authorJay Huh <jewoongh@meta.com>
Wed, 16 Oct 2024 16:22:51 +0000 (09:22 -0700)
committerJay Huh <jewoongh@meta.com>
Thu, 17 Oct 2024 18:01:51 +0000 (11:01 -0700)
commit11f21cf86b519a6f33b44f3cd3e3fe3fb1d1e315
tree337e0477839926e9921099238ad868149f5819f8
parenteca4f106bb9994045e31ca463187ecf5dc211b3f
Preserve Options File (#13074)

Summary:
In https://github.com/facebook/rocksdb/issues/13025 , we made a change to load the latest options file in the remote worker instead of serializing the entire set of options.

That was done under assumption that OPTIONS file do not get purged often. While testing, we learned that this happens more often than we want it to be, so we want to prevent the OPTIONS file from getting purged anytime between when the remote compaction is scheduled and the option is loaded in the remote worker.

Like how we are protecting new SST files from getting purged using `min_pending_output`, we are doing the same by keeping track of `min_options_file_number`. Any OPTIONS file with number greater than `min_options_file_number` will be protected from getting purged. Just like `min_pending_output`, `min_options_file_number` gets bumped when the compaction is done. This is only applicable when `options.compaction_service` is set.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13074

Test Plan:
```
./compaction_service_test --gtest_filter="*PreservedOptionsLocalCompaction*"
./compaction_service_test --gtest_filter="*PreservedOptionsRemoteCompaction*"
```

Reviewed By: anand1976

Differential Revision: D64433795

Pulled By: jaykorean

fbshipit-source-id: 0d902773f0909d9481dec40abf0b4c54ce5e86b2
db/compaction/compaction_job.h
db/compaction/compaction_service_job.cc
db/compaction/compaction_service_test.cc
db/db_impl/db_impl.cc
db/db_impl/db_impl.h
db/db_impl/db_impl_compaction_flush.cc
db/db_impl/db_impl_files.cc
db/db_impl/db_impl_secondary.cc
db/job_context.h
unreleased_history/behavior_changes/remote_compaction_preserve_options.md [new file with mode: 0644]