os/bluestore: _dump_onode crcs in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove obsolete tail cache

The buffer cache will cover this in a much more general way.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Fixes improper length calculation in BufferSpace::read + adds simplified test case to highlight an issue for append to existing blob

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: drop min_alloc_size locals

We have this in the class, now.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix min_alloc_size global

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: release partial extents

Use the blob put_ref helper so that we can deallocate blobs partially
(instead of always waiting until they are completely unused).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only write into a blob region that is allocated

We're only worried about direct writes and wal overwrites; the other write
paths are to freshly allocated blobs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t: add tracking for released extents

We reference count which parts of the blob are used (by lextents), but
currently we only release our space back to the system when all references
go away.  That is a problem if the blob is large (say, 4MB), and we, say,
truncate off most (but not all) of it.

Unfortunately, we can't simply deallocate anything that doesn't have a
reference, because the logical refs are on byte boundaries, and allocation
happens in larger units (min_alloc_size).  A one byte logical punch_hole
might be responsible for the release of a larger block of storage.

To resolve this, we keep track of which portions of the blob have been
released by poisoning the offset in the extents vector.  We expect that
this vector will be almost always short, so we do not bother with a
indexed structure, since iterating a blob offset to determine if it is
still allocated is likely faster.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add poison offset to pextent_t

This is a "magic" offset that we can use to indicate an invalid extent
(vs, say, an extent at offset 0 that might clobber real data if it were
used).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove dead _txc_release

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only direct write into unused blob space

We can only do a direct write into an already-allocated blob once, if that
range hasn't yet been used.  Once it has been used, it is much to complex
to keep track of when all references to it have committed to disk before
reusing it, so we don't try to handle that case at all.

Since the range has never been used, we can assert that there are no
references to it.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: mark used range on partial blob writes

- writing into unreferenced blob space
- wal blob writes

both need to update the blob used map.  The full blob writes generates
blobs that are always full, so no change is needed there.  New partial
blob creations need to indicate which parts aren't yet used.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add blob_t unused

Keep track of which ranges of this blob have *never* been used.  We do
this as a negative so that the common case of a fully-written blob is an
empty set.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: benchmark different csum methods

crc32c wins on my laptop.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: run csum tests on all algorithms

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t: add xxhash64

Signed-off-by: Sage Weil <sage@redhat.com>

common/Checksummer: add xxhash64

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop old Checksummer

blob_t uses it directly via the static methods.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use blob_t csum methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: simpler {calc,verify}_csum methods

This keeps the CSUM_* definitions local to blob_t, and avoids passing
arguments around.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: defer csum calcuations sometimes

When we are doing a partial chunk overwrite, we need to defer the csum_data
update.  Otherwise, another write in the same transaction might need to
read part of the chunk, not find the data in the buffer cache, read it
from disk, and fail the csum check.

This patch defers the calculation until after we've build the transaction
and are about to commit to the kv store.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/bluestore: update based on Igor's feedback

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Fixes some issues when using Buffer Cache from _do_read and improves test coverage

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes invalid assert in  Buffer::truncate

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/objectstore: Adds trivial test case to verify buffer cache use in bluestore

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Adds cached buffer processing for _do_read

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add a very simple (incomplete) buffer cache

Attach it to each onode.

There is no trimming yet.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make tail cache a bit smarter

This is really a stop-gap.  Since we are doing reads in the pre-commit
write path, we need to have some sort of buffer cache so that a sequence
of writes in the same transaction can remain coherent (the second write
must "read" the first write in order to fill out the chunk).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump tail_bl state in _dump_onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Adds lacking methods in bluestore_compression_header_t to fix encoder test build

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add decompressor call to read path

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore/bluestore_types: add bluestore_compression_header_t

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

compressor: Refactor to allow bufferlist::iterator as an input

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: new write path

- simplified wal_op_t.  we still have overlays in there, although that
  might need to get removed soon too.
- init_csum cleanup
- totally new write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: verify blob ref_maps during fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: return EIO on csum verification error

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify _verify_csum

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cleanup _read_extent_sparse

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluetores: fix checksum verification

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: read patch code cleanup

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: replace _do_read with ExtentManager version

Replaces _do_read with he corresponding code from ExtentManager to support csum verification & decompression.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

doc/dev/bluestore: write path notes

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: new test SmallBlockWrites

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add BigWriteBigZero test

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: expand SimpleObjectTest a bit

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define min_alloc_size globally

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define block_size, block_mask, block_order globally

These don't change, no need to declare them all over the place.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: change block_size -> chunk_size for _pad_zero* methods

The granularity is not the same as block_size.. depends on what the caller
wants.  Use a more generic name.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use _txc_finalize_kv for wal allocation updates

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass txc to _do_wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename _txc_update_fm -> _txc_finalize_kv

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: break _txc_finalize into _txc_write_nodes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump csums in _dump_onode/bnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_bnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: nicer operator<< for blob_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add onode_t::compress_extent_map()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t convenience methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: FLAG_IMMUTABLE -> FLAG_MUTABLE

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t::is_unreferenced() method

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add blob_t::has_any_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: test {find,seek}_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t csum additions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: blob_t::init_csum() helper

Set up the checksum parameters and size csum_data appropriately.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track csum config options

Use global csum_type member to drive new checksum calcuations.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instantiate a Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prototype Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add buffer::list::iterator::get_ptr_and_advance

Efficiently iterate over buffers in a list, via an iterator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: extent_ref_map_t::intersects

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: change extent_ref_map_t

- no more add, or weird add vs get assumptions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: drop num_refs from blob_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: onode_t::punch_hole()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: disable clone cow until it works again

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix bad iterator deref in _do_zero_tail_extent

The extent_map may be empty, or not contain anything at this offset.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename enode -> bnode

We're not refcounting extents anymore; we're storing shared blobs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use hex almost everywhere

It's like going metric, but even better!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove block_map from onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: convert io path to extent_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add get_blob_ptr helpers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: show start and end offsets for buffer mismatch

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to enode

We'll remove the old ref_map once the users go away.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass c to _do_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to onode_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent_t for releases

Trying to remove the old extent_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent for wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix blob operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep an extent_ref_map for each blob

This lets us track which portions of a blob are still in use.  In some
cases, we may be able to split the blob to deallocate a portion of it.
In other cases, we will want this information to know whether to
recompress the blob (or whatever).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add map and map_bl methods to blob

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: data structure to support compression/checksum verification. Read handler prototype implementation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9183 from yehudasa/wip-rgw-sync-retry

rgw: sync retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9073 from ukernel/wip-posixacl-fix

client: fix setting empty default ACL

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9422 from liewegas/wip-bluefs

os/bluestore/BlueFS: wait for aios on shutdown

Merge pull request #9343 from dachary/wip-13522-ceph-disk

ceph-disk: workaround gperftool hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: wait for aios on shutdown

This fixes segv from unittest_bluefs.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: workaround gperftool hang

Temporary workaround: if ceph-osd --mkfs does not
complete within 5 minutes, assume it is blocked
because of https://github.com/gperftools/gperftools/issues/786

References http://tracker.ceph.com/issues/13522

Signed-off-by: Loic Dachary <loic@dachary.org>

qa/workunits/fs/misc: test for setting empty ACL

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9299 from cbodley/wip-rgw-vstart-valgrind

vstart: add --valgrind_rgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9375 from dillaman/wip-14937

rbd-mirror: replicate cloned images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9162 from xiexingguo/xxg-wip-fixbmap

os/bluestore: fix bmap allocator relevant issues

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>