]> git-server-git.apps.pok.os.sepia.ceph.com Git - rocksdb.git/commit
Add file_checksum from FileChecksumGenFactory and Tests for corrupted output (#13060)
authorJay Huh <jewoongh@meta.com>
Tue, 15 Oct 2024 01:26:17 +0000 (18:26 -0700)
committerJay Huh <jewoongh@meta.com>
Thu, 17 Oct 2024 18:01:35 +0000 (11:01 -0700)
commiteca4f106bb9994045e31ca463187ecf5dc211b3f
treef3f2abf5ca14ffc3d24fa4cfbbdf9882ffc68c2a
parent5bb363edc72c31d57abe4c9eace5bb48d0e3bba3
Add file_checksum from FileChecksumGenFactory and Tests for corrupted output (#13060)

Summary:
- When `FileChecksumGenFactory` is set, include the `file_checksum` and `file_checksum_func_name` in the output file metadata
- ~~In Remote Compaction, try opening the output files in the temporary directory to do a quick sanity check before returning the result with status.~~
- After offline discussion, we decided to rely on Primary's existing Compaction flow to sanity check the output files. If the output file is corrupted, we will still be able to catch it and not installing it even after renaming them to cf_paths. The corrupted file in the cf_path won't be added to the MANIFEST and will be purged as part of the next `PurgeObsoleteFiles()` call.
- Unit Test has been added to validate above.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/13060

Test Plan:
Unit test added
```
./compaction_service_test --gtest_filter="*CorruptedOutput*"
./compaction_service_test --gtest_filter="*TruncatedOutput*"
./compaction_service_test --gtest_filter="*CustomFileChecksum*"
./compaction_job_test --gtest_filter="*ResultSerialization*"
```

Reviewed By: cbi42

Differential Revision: D64189645

Pulled By: jaykorean

fbshipit-source-id: 6cf28720169c960c80df257806bfee3c0d177159
db/compaction/compaction_job.h
db/compaction/compaction_job_test.cc
db/compaction/compaction_service_job.cc
db/compaction/compaction_service_test.cc