os/bluestore/BlueStore: fix error path if label set fails

Reported-by: David Zafman <dzafman@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: fix recycle replay

Signed-off-by: Sage Weil <sage@redhat.com>

Makefile-rocksdb.am: update

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: default to 64k min_alloc_size

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix _open_bdev() failure path

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: behave if options string is empty

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clear coll_map on umount, fsck finish

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore/KStore: fix object key decode with key

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix object key decode with key

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_objectstore_test: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>

os/KeyValueStore: drop kinetic #include

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: add new KStore backend

This is based on BlueStore, but with all of the block-related code
and complexity ripped out, and a simple striping strategy added
in its place.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: localize types

Prefix with bluestore_

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add extent_ref_map_t

This will be used to refcount extents for some subset
of the store (objects with same name or hash value?).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/FreelistManager: drop unused db ref

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: record kv backend

Record kv backend at mkfs time instead of relying on current value
of config option.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: statfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: inject block failures

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: clean up synthetic collections

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: block.db support

Support a mid- to fast device that will preferentially
store the rocksdb data (and wal, if block.wal is not
present).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: less debug noise

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: all overwrites on open_for_write

rocksdb will occasionally overwrite an existing file
if it is not present/valid in the manifest.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: drop internal EnvMirror

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: pull up to master, include EnvMirror

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: label all block devices

Label all of our block devices with a simple label
that includes the osd_uuid.  Wire this into the
ObjectStore and OSD probe mechanism.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: flush log if needed

If a file has dirty metadata (but no dirty data), we
still need to flush the log when it is flushed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix replay of unlink

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: support second block.wal device

Use this device for the bluefs log.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix zero gap bug

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: disable overlay for now

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: restructure interface

use atomics, do not track in-flight extents or magically cope
with racing ios (that is the users responsibility).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix overwrite

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix writes spanning extents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reenable rocksdb recycling

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: lock device while open

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: debug read result

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix alignment check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: check aio return values

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: avoid lock during reads

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: prevent read+write sharing

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: debug bluefs and rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: periodically compact log

Rewrite only the current metadata in a fresh log
periodically to free log space.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: simplify extent list

Merge contiguous extents.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix read

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: trivial init fix

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: rocksdb_separate_wal_dir option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: ref count BlueFS::File *

There are FileWriters that exist when the file is
deleted.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: readdir list dirs, too

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluefs-tool: simple tool to export bluefs content

Currently we just do a dump.  We'll add more
functionality later.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: many fixes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: share space with BlueFS

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: move to simple mutex model

Just for now, while we get the rest of this working.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: simple file system to back rocksdb

BlueFS is a simple file system that will back rocksdb.
BlueRocksEnv is the rocksdb::Env implementation that
glues them together.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: less verbose

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: less verbose on hash collision test

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix _do_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: fix locking

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: fix misc bugs

Can't use invalid iterator; fix init_rm_free.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/Allocator: init_rm_free

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: take custom Env

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read return value

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix read return value

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: separate Allocator from freelist storage

FreelistManager perists our freelist.  Allocator is a policy that
allocates it.

Signed-off-by: Sage Weil <sage@redhat.com>

newstore -> bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: always create db.wal

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: create db dir

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: consume a raw block device

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: make collection_list tolerate sloppy start position

Because of this change (#6076), the hobject_t will contain pool id, hence
the ghobject_t having this hobject_t will be not equal to ghobject_t().

In newstore, this will cause assertion failure:
FAILED assert(k >= start_key && k < end_key)

The fix is to make compatible with previous change to create a
ghobject_t object with pool id and shard id in newstore.

Fixes: #13801
Reported-by: Zhi Zhang <zhangz.david@outlook.com>
Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: make key names more efficient

- pack u32 and u64 in binary (instead of in hex)
- avoid duplicating the object name while making things still
  sort by (key,name).  Use < when key < name, = when key == name,
  > when key > name) as a prefix.  And in the = case (which is
  basically always) include the name just once.

Note that this breaks on-disk compatibility.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix collection_list vs max entries

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: do not set/change frag_size if there are overlays

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: define a fid_backpointer_t type

Signed-off-by: Sage Weil <sage@redhat.com>
fix wal_oP_t

os/newstoer: add newstore types to ceph-dencoder

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: set alloc hint on new frags

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: dump onode contents

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fixed fragment size

Instead of a single, variable-length fragment for each object,
set a fixed size (newstore_min_frag_size = 1 MB) and stripe the
object over these.  The last fragment will be smaller
than 1 MB if the object is not a multiple of 1 MB.

On write, this is basically free: we can just as cheaply write
4 inodes created together and fsync them than we can one.  On
overwrite, it allows us to replace individual fragments and avoid
write-ahead many cases.

On read it is a bit slower because of inode lookups and disk
seeks.  In the common case (big object written sequentially) we
hope that fs prefetching will hide most of it (e.g., all inodes
will be loaded together in the same metadata btree node, and the
files' data is written sequentially on disk).

Allowing for a singe large fragment in the case of a sequentially
written large object may save us something, but it complicates
the code significantly.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: recycle rocksdb log files

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: latest master

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6649 from majianpeng/filesstore-lfnunlink

osd: FileStore:: optimize lfn_unlink

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7017 from efirs/ef_atomic_ceph_tid

osd: use atomic to generate ceph_tid

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7077 from XinzeChi/wip-fix-wip-perf

osd: fix wip (l_osd_op_wip) perf counter and remove repop_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6893 from kylinstorage/wip-osd_command

librados: add c++ style osd/pg command interface

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5630 from wonzhq/evict-after-flush

osd: try evicting after flushing is done

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6639 from xiexingguo/xxg-wip-13822

librados: potential null pointer access in list_(n)objects

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6702 from liewegas/wip-fix-recency

osd/ReplicatedPG: fix promotion recency logic

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6824 from Sandy4999/wip-crushtool-build

crushtool: set type 0 name "device" for --build option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6962 from liewegas/wip-buffer-lastp

buffer: fix internal iterator invalidation on rebuild, get_contiguous

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6970 from aiicore/drop_removal_pg_type

osd: drop deprecated removal pg type

Reviewed-by: Sage Weil <sage@redhat.com>

osd: remove repop_map in osd

If I do not misread, repop_map is useless.

Signed-off-by: Xinze Chi <xinze@xsky.com>

osd: fix wip (l_osd_op_wip) perf counter

The l_osd_op_wip is for osd, so it should be the sum of all pgs in osd

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #6987 from H3C/wip-addr-bugfix

common/address_help.cc: fix the leak in entity_addr_from_url()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge remote-tracking branch 'origin/jewel'

common/address_help.cc: fix the leak in entity_addr_from_url()

Fixes: #14132
Signed-off-by: Qiankun Zheng <zheng.qiankun@h3c.com>

Merge pull request #7026 from xdonghai/master

rbd: must specify both of stripe-unit and stripe-count when specifying stripingv2 feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6998 from xiexingguo/xxg-wip-clsrbd

stringify outputted error code and fix unmatched parentheses.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6983 from xiexingguo/xxg-wip-14126

librbd: small fixes for error messages and readahead counter

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6298 from guangyy/wip-13441

osd: make list_missing query missing_loc.needs_recovery_map

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6572 from liewegas/wip-crush-chooseleaf-stable

crush: add chooseleaf_stable tunable

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7043 from dillaman/wip-14170-jewel

librbd: do not ignore self-managed snapshot release result

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: do not ignore self-managed snapshot release result

Fixes: #14170
Signed-off-by: Jason Dillaman <dillaman@redhat.com>