qa/workunits/rados/test_rados_tool.sh: fix path

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_rados_tool: adapt to new buffer hexdump

Signed-off-by: Sage Weil <sage@redhat.com>

uniform the type format

Signed-off-by: Ning Yao <zay11022@gmail.com>

os/bluestore/BlueStore: fix overlay keys

Signed-off-by: YiQiang Chen <cyqsign@163.com>

fs: fix unclosed formatter session

Fixes: #14210
Signed-off-by: xie.xingguo <xie.xingguo@zte.com.cn>

qa/workunits/cephtool/test.sh: leave sortbitwise set

New backends don't work if it's off.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: remove deleted file from dirty list

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Don't forget close fd in _setup_block_symlink_or_file.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: write 'type' file

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: don't require journal path

Assume journal symlink is present.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: get store type from 'type' file, if present

Just like ceph-osd.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: wait for flush aios to complete before returning

This makes all the IOs effectively synchronous (while holding the lock),
which isn't great, but at least it's correct.  We can contemplate async
later..

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: clean up aio completion a bit

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: delay IOContext dtor until after io completes

It's possible for the IO to be in flight when the caller closes the
writer handle (although dangerous of them).  Queue the IOContext for
async cleanup when we sync everything to disk.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix a few leaks

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: close cct on shutdown

This lets us do leak checking.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueRocksEnv: fix several leaks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix leak of iocs in writer

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: fix leak of custom Env

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix FS leak

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: no need for 'quit' to ceph cli

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: k=2 for ec

It didn't like k=1 with the default profile.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: be tolerant of collection_list bounds

In particular, we may get a shard specified along with hobject_t min or
max (from PGBackend::objects_list_range()).

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists: update for os/ reorg

Signed-off-by: Sage Weil <sage@redhat.com>

os/Makefile: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: cache in-flight stripe updates in memory

The write process may do a read/modify/write on a stripe.  In order to
allow multiple writes to coexist within the same transaction, we need to
be able to "see" our writes.

Clear the "cached" stripe values when the last TransContext touching an
onode is finished.  In theory we could pin memory with a constant stream
of updates to an object; we may need to address that later.

Signed-off-by: Sage Weil <sage@redhat.com>

common/buffer: make hexdump look like 'hexdump -C ...'

...the better to diff them with.

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: debug nid

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: fix bugs in read stripe logic

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: clean out test dir for each test

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: clear coll_map on umount

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not catch mkfs exceptions

We either want to take a clean error path, or dump core.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: be less stupid about alloc_unit

Make sure the extent we return is aligned to alloc_unit.  If the first
extent isn't suitable, try the next one.

Keep bin 0 for things that are smaller than min_alloc_size.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: fix warnings

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: fix bluestore_extent_ref_map_t test inst

Signed-off-by: Sage Weil <sage@redhat.com>

os/Makefile: cosmetic

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: move btrfs_ioctl.h to os/fs

Signed-off-by: Sage Weil <sage@redhat.com>

os/ZFS: move to os/fs

Signed-off-by: Sage Weil <sage@redhat.com>

os/keyvaluestore: move KeyValueStore into os/keyvaluestore/*

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: move MemStore into os/memstore/*

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: move FileStore to os/filestore/*

Signed-off-by: Sage Weil <sage@redhat.com>

ceph:small_io_bench_dumb: drop unused include

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: drop unused FileSTore include

Signed-off-by: Sage Weil <sage@redhat.com>

remove ceph_streamtest

This is an ancient obsolete test that is never used.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allocate shouldn't fail if reserve succeeded

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/Allocator: add unreserve()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: fix reservation release on allocate

Use up the amount we allocated, not the amount we asked for.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: add reservation assertions

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: do not include header in omap iterator result

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not include header in iterator result

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: print omap_digest in debug log

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add omap iterator test

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: add fsck command

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: add fsck to interface

Only bluestore and kstore implement this currently.

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: clear onode on _do_remove

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clear onode in _do_remove

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGBackend: fix omap digest error message

Print the *omap* digest, not *data* digest.

Signed-off-by: Sage Weil <sage@redhat.com>

os: remove {get,set}_allow_sharded_objects from interface

We've already forced everyone to upgrade through hammer, so everyone
supports this.  Just unconditionally set the feature if it is not set
(for consistency's sake).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not flush metadata on flush()

That's what fsync is for.  Moreover, this can lead to some squirreliness
if we trigger this from _flush_log().

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: helpful error when aio cannot init

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/FreelistManager: audit

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: implement invalidate_cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: add invalidate_cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify _do_remove

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: debug msg on statfs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: optimize _dump_onode slightly

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: enable bluestore debug options

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fsck_on_umount

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: bluestore_debug_small_allocations

Force small allocations for debugging purposes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix _zero when previous extent partially unwritten

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: support copy-on-write clones

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: only allow clone if hash matches

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Enode infrastructure

Enodes will track extent ref counts for any extent
that is marked shared.  There will be an enode for
any unique hash value that has any refs.  We will keep
in-memory copies of only those Enodes that are
referenced by in-memory Onodes, and only if the enode
is requested (e.g., the enode won't be loaded as a
result of an object read because we never need to
call get_enode.).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add extent FLAG_COW_{HEAD,TAIL}

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluefs, unittest_bluestore_types

These should run during make check.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add contains(), clear(), empty() to extent_ref_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: wal_op_t::OP_COPY

Assume block-aligned.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix waiter wakeup use-after-free race

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add bluestore_debug_no_reuse_blocks

This makes debugging a bit easier because we never use the same
extent of the disk twice, leaving useful evidence behind.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: do Synthetic tests over larger objects

400k for objects.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: use a few hash values for objects; clone between them

We only guarantee support for clone between objects with the same hash.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: dump actual vs expected on read data mismatch

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add many clone tests

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: validate full object contents after writes

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: debug enter/exit points

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: save map lookups for a few ops

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix locking for a few ops

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix clone

Copy the buffer, in case other threads modify it in place.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: simplify object name generation

The long names don't exercise useful code paths, and having
consistent naming makes it easier to grep through logs.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: clone non-empty objects, not empty ones

This condition was backwards.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: clone objects with same hash

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add some slow debug path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clean up comments a bit

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: note wal releases in fsck

Include these in used_blocks (they are about to be released but
not reflected in the onode).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix read bug when there is a hole

Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: fix rename

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: fix rename

Install a negative onode entry at the old name position.
Otherwise, a simple transaction like

 rename a -> b
 touch b

will re-read the old b onode key on the second op, and chaos will
ensue (e.g., because it'll reference the same extents from a
different object).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: remove unused OnodeMap::remove

We install negative entries instead.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: adjust debug output

5 helpful (read/write offsets)
10 more, with aio completions
20 everything
30 fire hose
40 data hexdumps

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: fix path

Signed-off-by: Sage Weil <sage@redhat.com>