]> git-server-git.apps.pok.os.sepia.ceph.com Git - rocksdb.git/commit
Refactor: group metadata needed to open an SST file to a separate copyable struct
authorsdong <siying.d@fb.com>
Fri, 13 Jun 2014 22:54:19 +0000 (15:54 -0700)
committersdong <siying.d@fb.com>
Mon, 16 Jun 2014 23:10:52 +0000 (16:10 -0700)
commitcadc1adffa447f02eb65bd848cf26c13142a74bb
tree052f411bc99d21b5836c2842d8fe5e5fc01ba2a3
Refactor: group metadata needed to open an SST file to a separate copyable struct

Summary:
We added multiple fields to FileMetaData recently and are planning to add more.
This refactoring separate the minimum information for accessing the file. This object is copyable (FileMetaData is not copyable since the ref counter). I hope this refactoring can enable further improvements:

(1) use it to design a more efficient data structure to speed up read queries.
(2) in the future, when we add information of storage level, we can easily do the encoding, instead of enlarge this structure, which might expand memory work set for file meta data.

The definition is same as current EncodedFileMetaData used in two level iterator, so now the logic in two level iterator is easier to understand.

Test Plan: make all check

Reviewers: haobo, igor, ljin

Reviewed By: ljin

Subscribers: leveldb, dhruba, yhchiang

Differential Revision: https://reviews.facebook.net/D18933
384 files changed:
.arcconfig [new file with mode: 0644]
.clang-format [new file with mode: 0644]
.gitignore [new file with mode: 0644]
.travis.yml [new file with mode: 0644]
CONTRIBUTING.md [new file with mode: 0644]
HISTORY.md [new file with mode: 0644]
INSTALL.md [new file with mode: 0644]
LICENSE [new file with mode: 0644]
Makefile [new file with mode: 0644]
PATENTS [new file with mode: 0644]
README.md [new file with mode: 0644]
ROCKSDB_LITE.md [new file with mode: 0644]
build_tools/build_detect_platform [new file with mode: 0755]
build_tools/build_detect_version [new file with mode: 0755]
build_tools/fbcode.clang31.sh [new file with mode: 0644]
build_tools/fbcode.gcc471.sh [new file with mode: 0644]
build_tools/fbcode.gcc481.sh [new file with mode: 0644]
build_tools/format-diff.sh [new file with mode: 0755]
build_tools/mac-install-gflags.sh [new file with mode: 0755]
build_tools/make_new_version.sh [new file with mode: 0755]
build_tools/regression_build_test.sh [new file with mode: 0755]
build_tools/valgrind_test.sh [new file with mode: 0755]
coverage/coverage_test.sh [new file with mode: 0755]
coverage/parse_gcov_output.py [new file with mode: 0644]
db/builder.cc [new file with mode: 0644]
db/builder.h [new file with mode: 0644]
db/c.cc [new file with mode: 0644]
db/c_test.c [new file with mode: 0644]
db/column_family.cc [new file with mode: 0644]
db/column_family.h [new file with mode: 0644]
db/column_family_test.cc [new file with mode: 0644]
db/compaction.cc [new file with mode: 0644]
db/compaction.h [new file with mode: 0644]
db/compaction_picker.cc [new file with mode: 0644]
db/compaction_picker.h [new file with mode: 0644]
db/corruption_test.cc [new file with mode: 0644]
db/db_bench.cc [new file with mode: 0644]
db/db_filesnapshot.cc [new file with mode: 0644]
db/db_impl.cc [new file with mode: 0644]
db/db_impl.h [new file with mode: 0644]
db/db_impl_debug.cc [new file with mode: 0644]
db/db_impl_readonly.cc [new file with mode: 0644]
db/db_impl_readonly.h [new file with mode: 0644]
db/db_iter.cc [new file with mode: 0644]
db/db_iter.h [new file with mode: 0644]
db/db_stats_logger.cc [new file with mode: 0644]
db/db_test.cc [new file with mode: 0644]
db/dbformat.cc [new file with mode: 0644]
db/dbformat.h [new file with mode: 0644]
db/dbformat_test.cc [new file with mode: 0644]
db/deletefile_test.cc [new file with mode: 0644]
db/file_indexer.cc [new file with mode: 0644]
db/file_indexer.h [new file with mode: 0644]
db/file_indexer_test.cc [new file with mode: 0644]
db/filename.cc [new file with mode: 0644]
db/filename.h [new file with mode: 0644]
db/filename_test.cc [new file with mode: 0644]
db/forward_iterator.cc [new file with mode: 0644]
db/forward_iterator.h [new file with mode: 0644]
db/internal_stats.cc [new file with mode: 0644]
db/internal_stats.h [new file with mode: 0644]
db/log_and_apply_bench.cc [new file with mode: 0644]
db/log_format.h [new file with mode: 0644]
db/log_reader.cc [new file with mode: 0644]
db/log_reader.h [new file with mode: 0644]
db/log_test.cc [new file with mode: 0644]
db/log_writer.cc [new file with mode: 0644]
db/log_writer.h [new file with mode: 0644]
db/memtable.cc [new file with mode: 0644]
db/memtable.h [new file with mode: 0644]
db/memtable_list.cc [new file with mode: 0644]
db/memtable_list.h [new file with mode: 0644]
db/merge_context.h [new file with mode: 0644]
db/merge_helper.cc [new file with mode: 0644]
db/merge_helper.h [new file with mode: 0644]
db/merge_operator.cc [new file with mode: 0644]
db/merge_test.cc [new file with mode: 0644]
db/perf_context_test.cc [new file with mode: 0644]
db/plain_table_db_test.cc [new file with mode: 0644]
db/prefix_test.cc [new file with mode: 0644]
db/repair.cc [new file with mode: 0644]
db/simple_table_db_test.cc [new file with mode: 0644]
db/skiplist.h [new file with mode: 0644]
db/skiplist_test.cc [new file with mode: 0644]
db/snapshot.h [new file with mode: 0644]
db/table_cache.cc [new file with mode: 0644]
db/table_cache.h [new file with mode: 0644]
db/table_properties_collector.cc [new file with mode: 0644]
db/table_properties_collector.h [new file with mode: 0644]
db/table_properties_collector_test.cc [new file with mode: 0644]
db/tailing_iter.cc [new file with mode: 0644]
db/tailing_iter.h [new file with mode: 0644]
db/transaction_log_impl.cc [new file with mode: 0644]
db/transaction_log_impl.h [new file with mode: 0644]
db/version_edit.cc [new file with mode: 0644]
db/version_edit.h [new file with mode: 0644]
db/version_edit_test.cc [new file with mode: 0644]
db/version_set.cc [new file with mode: 0644]
db/version_set.h [new file with mode: 0644]
db/version_set_test.cc [new file with mode: 0644]
db/write_batch.cc [new file with mode: 0644]
db/write_batch_internal.h [new file with mode: 0644]
db/write_batch_test.cc [new file with mode: 0644]
doc/doc.css [new file with mode: 0644]
doc/index.html [new file with mode: 0644]
doc/log_format.txt [new file with mode: 0644]
doc/rockslogo.jpg [new file with mode: 0644]
doc/rockslogo.png [new file with mode: 0644]
examples/.gitignore [new file with mode: 0644]
examples/Makefile [new file with mode: 0644]
examples/README.md [new file with mode: 0644]
examples/column_families_example.cc [new file with mode: 0644]
examples/simple_example.cc [new file with mode: 0644]
hdfs/README [new file with mode: 0644]
hdfs/env_hdfs.h [new file with mode: 0644]
hdfs/setup.sh [new file with mode: 0644]
helpers/memenv/memenv.cc [new file with mode: 0644]
helpers/memenv/memenv_test.cc [new file with mode: 0644]
include/rocksdb/c.h [new file with mode: 0644]
include/rocksdb/cache.h [new file with mode: 0644]
include/rocksdb/compaction_filter.h [new file with mode: 0644]
include/rocksdb/comparator.h [new file with mode: 0644]
include/rocksdb/db.h [new file with mode: 0644]
include/rocksdb/env.h [new file with mode: 0644]
include/rocksdb/filter_policy.h [new file with mode: 0644]
include/rocksdb/flush_block_policy.h [new file with mode: 0644]
include/rocksdb/iterator.h [new file with mode: 0644]
include/rocksdb/ldb_tool.h [new file with mode: 0644]
include/rocksdb/memtablerep.h [new file with mode: 0644]
include/rocksdb/merge_operator.h [new file with mode: 0644]
include/rocksdb/options.h [new file with mode: 0644]
include/rocksdb/perf_context.h [new file with mode: 0644]
include/rocksdb/slice.h [new file with mode: 0644]
include/rocksdb/slice_transform.h [new file with mode: 0644]
include/rocksdb/statistics.h [new file with mode: 0644]
include/rocksdb/status.h [new file with mode: 0644]
include/rocksdb/table.h [new file with mode: 0644]
include/rocksdb/table_properties.h [new file with mode: 0644]
include/rocksdb/transaction_log.h [new file with mode: 0644]
include/rocksdb/types.h [new file with mode: 0644]
include/rocksdb/universal_compaction.h [new file with mode: 0644]
include/rocksdb/version.h [new file with mode: 0644]
include/rocksdb/write_batch.h [new file with mode: 0644]
include/utilities/backupable_db.h [new file with mode: 0644]
include/utilities/db_ttl.h [new file with mode: 0644]
include/utilities/geo_db.h [new file with mode: 0644]
include/utilities/stackable_db.h [new file with mode: 0644]
include/utilities/utility_db.h [new file with mode: 0644]
java/Makefile [new file with mode: 0644]
java/RocksDBSample.java [new file with mode: 0644]
java/jdb_bench.sh [new file with mode: 0755]
java/org/rocksdb/BackupableDB.java [new file with mode: 0644]
java/org/rocksdb/BackupableDBOptions.java [new file with mode: 0644]
java/org/rocksdb/BloomFilter.java [new file with mode: 0644]
java/org/rocksdb/Filter.java [new file with mode: 0644]
java/org/rocksdb/HashLinkedListMemTableConfig.java [new file with mode: 0644]
java/org/rocksdb/HashSkipListMemTableConfig.java [new file with mode: 0644]
java/org/rocksdb/HistogramData.java [new file with mode: 0644]
java/org/rocksdb/HistogramType.java [new file with mode: 0644]
java/org/rocksdb/MemTableConfig.java [new file with mode: 0644]
java/org/rocksdb/Options.java [new file with mode: 0644]
java/org/rocksdb/PlainTableConfig.java [new file with mode: 0644]
java/org/rocksdb/ReadOptions.java [new file with mode: 0644]
java/org/rocksdb/RestoreBackupableDB.java [new file with mode: 0644]
java/org/rocksdb/RestoreOptions.java [new file with mode: 0644]
java/org/rocksdb/RocksDB.java [new file with mode: 0644]
java/org/rocksdb/RocksDBException.java [new file with mode: 0644]
java/org/rocksdb/RocksEnv.java [new file with mode: 0644]
java/org/rocksdb/RocksIterator.java [new file with mode: 0644]
java/org/rocksdb/RocksObject.java [new file with mode: 0644]
java/org/rocksdb/SkipListMemTableConfig.java [new file with mode: 0644]
java/org/rocksdb/Statistics.java [new file with mode: 0644]
java/org/rocksdb/TableFormatConfig.java [new file with mode: 0644]
java/org/rocksdb/TickerType.java [new file with mode: 0644]
java/org/rocksdb/VectorMemTableConfig.java [new file with mode: 0644]
java/org/rocksdb/WriteBatch.java [new file with mode: 0644]
java/org/rocksdb/WriteBatchTest.java [new file with mode: 0644]
java/org/rocksdb/WriteOptions.java [new file with mode: 0644]
java/org/rocksdb/benchmark/DbBenchmark.java [new file with mode: 0644]
java/org/rocksdb/test/BackupableDBTest.java [new file with mode: 0644]
java/org/rocksdb/test/OptionsTest.java [new file with mode: 0644]
java/org/rocksdb/test/ReadOptionsTest.java [new file with mode: 0644]
java/org/rocksdb/util/Environment.java [new file with mode: 0644]
java/org/rocksdb/util/SizeUnit.java [new file with mode: 0644]
java/rocksjni/backupablejni.cc [new file with mode: 0644]
java/rocksjni/env.cc [new file with mode: 0644]
java/rocksjni/filter.cc [new file with mode: 0644]
java/rocksjni/iterator.cc [new file with mode: 0644]
java/rocksjni/memtablejni.cc [new file with mode: 0644]
java/rocksjni/options.cc [new file with mode: 0644]
java/rocksjni/portal.h [new file with mode: 0644]
java/rocksjni/restorejni.cc [new file with mode: 0644]
java/rocksjni/rocksjni.cc [new file with mode: 0644]
java/rocksjni/statistics.cc [new file with mode: 0644]
java/rocksjni/table.cc [new file with mode: 0644]
java/rocksjni/write_batch.cc [new file with mode: 0644]
linters/__phutil_library_init__.php [new file with mode: 0644]
linters/__phutil_library_map__.php [new file with mode: 0644]
linters/cpp_linter/ArcanistCpplintLinter.php [new file with mode: 0644]
linters/cpp_linter/FbcodeCppLinter.php [new file with mode: 0644]
linters/cpp_linter/PfffCppLinter.php [new file with mode: 0644]
linters/cpp_linter/cpplint.py [new file with mode: 0755]
linters/lint_engine/FacebookFbcodeLintEngine.php [new file with mode: 0644]
port/README [new file with mode: 0644]
port/atomic_pointer.h [new file with mode: 0644]
port/likely.h [new file with mode: 0644]
port/port.h [new file with mode: 0644]
port/port_example.h [new file with mode: 0644]
port/port_posix.cc [new file with mode: 0644]
port/port_posix.h [new file with mode: 0644]
port/stack_trace.cc [new file with mode: 0644]
port/stack_trace.h [new file with mode: 0644]
port/win/stdint.h [new file with mode: 0644]
table/block.cc [new file with mode: 0644]
table/block.h [new file with mode: 0644]
table/block_based_table_builder.cc [new file with mode: 0644]
table/block_based_table_builder.h [new file with mode: 0644]
table/block_based_table_factory.cc [new file with mode: 0644]
table/block_based_table_factory.h [new file with mode: 0644]
table/block_based_table_reader.cc [new file with mode: 0644]
table/block_based_table_reader.h [new file with mode: 0644]
table/block_builder.cc [new file with mode: 0644]
table/block_builder.h [new file with mode: 0644]
table/block_hash_index.cc [new file with mode: 0644]
table/block_hash_index.h [new file with mode: 0644]
table/block_hash_index_test.cc [new file with mode: 0644]
table/block_test.cc [new file with mode: 0644]
table/filter_block.cc [new file with mode: 0644]
table/filter_block.h [new file with mode: 0644]
table/filter_block_test.cc [new file with mode: 0644]
table/flush_block_policy.cc [new file with mode: 0644]
table/format.cc [new file with mode: 0644]
table/format.h [new file with mode: 0644]
table/iter_heap.h [new file with mode: 0644]
table/iterator.cc [new file with mode: 0644]
table/iterator_wrapper.h [new file with mode: 0644]
table/merger.cc [new file with mode: 0644]
table/merger.h [new file with mode: 0644]
table/meta_blocks.cc [new file with mode: 0644]
table/meta_blocks.h [new file with mode: 0644]
table/plain_table_builder.cc [new file with mode: 0644]
table/plain_table_builder.h [new file with mode: 0644]
table/plain_table_factory.cc [new file with mode: 0644]
table/plain_table_factory.h [new file with mode: 0644]
table/plain_table_reader.cc [new file with mode: 0644]
table/plain_table_reader.h [new file with mode: 0644]
table/table_builder.h [new file with mode: 0644]
table/table_properties.cc [new file with mode: 0644]
table/table_reader.h [new file with mode: 0644]
table/table_reader_bench.cc [new file with mode: 0644]
table/table_test.cc [new file with mode: 0644]
table/two_level_iterator.cc [new file with mode: 0644]
table/two_level_iterator.h [new file with mode: 0644]
third-party/rapidjson/document.h [new file with mode: 0644]
third-party/rapidjson/filestream.h [new file with mode: 0644]
third-party/rapidjson/internal/pow10.h [new file with mode: 0644]
third-party/rapidjson/internal/stack.h [new file with mode: 0644]
third-party/rapidjson/internal/strfunc.h [new file with mode: 0644]
third-party/rapidjson/license.txt [new file with mode: 0644]
third-party/rapidjson/prettywriter.h [new file with mode: 0644]
third-party/rapidjson/rapidjson.h [new file with mode: 0644]
third-party/rapidjson/reader.h [new file with mode: 0644]
third-party/rapidjson/stringbuffer.h [new file with mode: 0644]
third-party/rapidjson/writer.h [new file with mode: 0644]
tools/auto_sanity_test.sh [new file with mode: 0755]
tools/blob_store_bench.cc [new file with mode: 0644]
tools/db_crashtest.py [new file with mode: 0644]
tools/db_crashtest2.py [new file with mode: 0644]
tools/db_repl_stress.cc [new file with mode: 0644]
tools/db_sanity_test.cc [new file with mode: 0644]
tools/db_stress.cc [new file with mode: 0644]
tools/ldb.cc [new file with mode: 0644]
tools/ldb_test.py [new file with mode: 0644]
tools/reduce_levels_test.cc [new file with mode: 0644]
tools/sst_dump.cc [new file with mode: 0644]
util/arena.cc [new file with mode: 0644]
util/arena.h [new file with mode: 0644]
util/arena_test.cc [new file with mode: 0644]
util/auto_roll_logger.cc [new file with mode: 0644]
util/auto_roll_logger.h [new file with mode: 0644]
util/auto_roll_logger_test.cc [new file with mode: 0755]
util/autovector.h [new file with mode: 0644]
util/autovector_test.cc [new file with mode: 0644]
util/benchharness.cc [new file with mode: 0644]
util/benchharness.h [new file with mode: 0644]
util/benchharness_test.cc [new file with mode: 0644]
util/blob_store.cc [new file with mode: 0644]
util/blob_store.h [new file with mode: 0644]
util/blob_store_test.cc [new file with mode: 0644]
util/bloom.cc [new file with mode: 0644]
util/bloom_test.cc [new file with mode: 0644]
util/build_version.h [new file with mode: 0644]
util/cache.cc [new file with mode: 0644]
util/cache_test.cc [new file with mode: 0644]
util/coding.cc [new file with mode: 0644]
util/coding.h [new file with mode: 0644]
util/coding_test.cc [new file with mode: 0644]
util/comparator.cc [new file with mode: 0644]
util/crc32c.cc [new file with mode: 0644]
util/crc32c.h [new file with mode: 0644]
util/crc32c_test.cc [new file with mode: 0644]
util/dynamic_bloom.cc [new file with mode: 0644]
util/dynamic_bloom.h [new file with mode: 0644]
util/dynamic_bloom_test.cc [new file with mode: 0644]
util/env.cc [new file with mode: 0644]
util/env_hdfs.cc [new file with mode: 0644]
util/env_posix.cc [new file with mode: 0644]
util/env_test.cc [new file with mode: 0644]
util/filelock_test.cc [new file with mode: 0644]
util/filter_policy.cc [new file with mode: 0644]
util/hash.cc [new file with mode: 0644]
util/hash.h [new file with mode: 0644]
util/hash_cuckoo_rep.cc [new file with mode: 0644]
util/hash_cuckoo_rep.h [new file with mode: 0644]
util/hash_linklist_rep.cc [new file with mode: 0644]
util/hash_linklist_rep.h [new file with mode: 0644]
util/hash_skiplist_rep.cc [new file with mode: 0644]
util/hash_skiplist_rep.h [new file with mode: 0644]
util/histogram.cc [new file with mode: 0644]
util/histogram.h [new file with mode: 0644]
util/histogram_test.cc [new file with mode: 0644]
util/ldb_cmd.cc [new file with mode: 0644]
util/ldb_cmd.h [new file with mode: 0644]
util/ldb_cmd_execute_result.h [new file with mode: 0644]
util/ldb_tool.cc [new file with mode: 0644]
util/log_buffer.cc [new file with mode: 0644]
util/log_buffer.h [new file with mode: 0644]
util/log_write_bench.cc [new file with mode: 0644]
util/logging.cc [new file with mode: 0644]
util/logging.h [new file with mode: 0644]
util/manual_compaction_test.cc [new file with mode: 0644]
util/murmurhash.cc [new file with mode: 0644]
util/murmurhash.h [new file with mode: 0644]
util/mutexlock.h [new file with mode: 0644]
util/options.cc [new file with mode: 0644]
util/perf_context.cc [new file with mode: 0644]
util/perf_context_imp.h [new file with mode: 0644]
util/posix_logger.h [new file with mode: 0644]
util/random.h [new file with mode: 0644]
util/signal_test.cc [new file with mode: 0644]
util/skiplistrep.cc [new file with mode: 0644]
util/slice.cc [new file with mode: 0644]
util/statistics.cc [new file with mode: 0644]
util/statistics.h [new file with mode: 0644]
util/stats_logger.h [new file with mode: 0644]
util/status.cc [new file with mode: 0644]
util/stl_wrappers.h [new file with mode: 0644]
util/stop_watch.h [new file with mode: 0644]
util/string_util.cc [new file with mode: 0644]
util/string_util.h [new file with mode: 0644]
util/sync_point.cc [new file with mode: 0644]
util/sync_point.h [new file with mode: 0644]
util/testharness.cc [new file with mode: 0644]
util/testharness.h [new file with mode: 0644]
util/testutil.cc [new file with mode: 0644]
util/testutil.h [new file with mode: 0644]
util/thread_local.cc [new file with mode: 0644]
util/thread_local.h [new file with mode: 0644]
util/thread_local_test.cc [new file with mode: 0644]
util/vectorrep.cc [new file with mode: 0644]
util/xxhash.cc [new file with mode: 0644]
util/xxhash.h [new file with mode: 0644]
utilities/backupable/backupable_db.cc [new file with mode: 0644]
utilities/backupable/backupable_db_test.cc [new file with mode: 0644]
utilities/geodb/geodb_impl.cc [new file with mode: 0644]
utilities/geodb/geodb_impl.h [new file with mode: 0644]
utilities/geodb/geodb_test.cc [new file with mode: 0644]
utilities/merge_operators.h [new file with mode: 0644]
utilities/merge_operators/put.cc [new file with mode: 0644]
utilities/merge_operators/string_append/stringappend.cc [new file with mode: 0644]
utilities/merge_operators/string_append/stringappend.h [new file with mode: 0644]
utilities/merge_operators/string_append/stringappend2.cc [new file with mode: 0644]
utilities/merge_operators/string_append/stringappend2.h [new file with mode: 0644]
utilities/merge_operators/string_append/stringappend_test.cc [new file with mode: 0644]
utilities/merge_operators/uint64add.cc [new file with mode: 0644]
utilities/redis/README [new file with mode: 0644]
utilities/redis/redis_list_exception.h [new file with mode: 0644]
utilities/redis/redis_list_iterator.h [new file with mode: 0644]
utilities/redis/redis_lists.cc [new file with mode: 0644]
utilities/redis/redis_lists.h [new file with mode: 0644]
utilities/redis/redis_lists_test.cc [new file with mode: 0644]
utilities/ttl/db_ttl_impl.cc [new file with mode: 0644]
utilities/ttl/db_ttl_impl.h [new file with mode: 0644]
utilities/ttl/ttl_test.cc [new file with mode: 0644]