]> git-server-git.apps.pok.os.sepia.ceph.com Git - rocksdb.git/commit
Make it possible to enable blob files starting from a certain LSM tree level (#10077)
authorGang Liao <gangliao@fb.com>
Fri, 3 Jun 2022 03:04:33 +0000 (20:04 -0700)
committerFacebook GitHub Bot <facebook-github-bot@users.noreply.github.com>
Fri, 3 Jun 2022 03:04:33 +0000 (20:04 -0700)
commite6432dfd4c84382eb6a200ce780fef55cbfbe453
tree9a4102f9801f2b12e2e86c501b0e1c74ec71c157
parenta0200315525fbca26b7f11246ec3ab841241b05a
Make it possible to enable blob files starting from a certain LSM tree level (#10077)

Summary:
Currently, if blob files are enabled (i.e. `enable_blob_files` is true), large values are extracted both during flush/recovery (when SST files are written into level 0 of the LSM tree) and during compaction into any LSM tree level. For certain use cases that have a mix of short-lived and long-lived values, it might make sense to support extracting large values only during compactions whose output level is greater than or equal to a specified LSM tree level (e.g. compactions into L1/L2/... or above). This could reduce the space amplification caused by large values that are turned into garbage shortly after being written at the price of some write amplification incurred by long-lived values whose extraction to blob files is delayed.

In order to achieve this, we would like to do the following:
- Add a new configuration option `blob_file_starting_level` (default: 0) to `AdvancedColumnFamilyOptions` (and `MutableCFOptions` and extend the related logic)
- Instantiate `BlobFileBuilder` in `BuildTable` (used during flush and recovery, where the LSM tree level is L0) and `CompactionJob` iff `enable_blob_files` is set and the LSM tree level is `>= blob_file_starting_level`
- Add unit tests for the new functionality, and add the new option to our stress tests (`db_stress` and `db_crashtest.py` )
- Add the new option to our benchmarking tool `db_bench` and the BlobDB benchmark script `run_blob_bench.sh`
- Add the new option to the `ldb` tool (see https://github.com/facebook/rocksdb/wiki/Administration-and-Data-Access-Tool)
- Ideally extend the C and Java bindings with the new option
- Update the BlobDB wiki to document the new option.

Pull Request resolved: https://github.com/facebook/rocksdb/pull/10077

Reviewed By: ltamasi

Differential Revision: D36884156

Pulled By: gangliao

fbshipit-source-id: 942bab025f04633edca8564ed64791cb5e31627d
32 files changed:
HISTORY.md
db/blob/db_blob_compaction_test.cc
db/builder.cc
db/c.cc
db/c_test.c
db/compaction/compaction_job.cc
db_stress_tool/db_stress_common.h
db_stress_tool/db_stress_gflags.cc
db_stress_tool/db_stress_test_base.cc
include/rocksdb/advanced_options.h
include/rocksdb/c.h
include/rocksdb/utilities/ldb_cmd.h
java/rocksjni/options.cc
java/src/main/java/org/rocksdb/AdvancedMutableColumnFamilyOptionsInterface.java
java/src/main/java/org/rocksdb/ColumnFamilyOptions.java
java/src/main/java/org/rocksdb/MutableColumnFamilyOptions.java
java/src/main/java/org/rocksdb/Options.java
java/src/test/java/org/rocksdb/BlobOptionsTest.java
java/src/test/java/org/rocksdb/MutableColumnFamilyOptionsTest.java
java/src/test/java/org/rocksdb/MutableOptionsGetSetTest.java
options/cf_options.cc
options/cf_options.h
options/options.cc
options/options_helper.cc
options/options_settable_test.cc
options/options_test.cc
tools/db_bench_tool.cc
tools/db_bench_tool_test.cc
tools/db_crashtest.py
tools/ldb_cmd.cc
tools/ldb_test.py
tools/run_blob_bench.sh