unittest_bluestore_types: run csum tests on all algorithms

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t: add xxhash64

Signed-off-by: Sage Weil <sage@redhat.com>

common/Checksummer: add xxhash64

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop old Checksummer

blob_t uses it directly via the static methods.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use blob_t csum methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: simpler {calc,verify}_csum methods

This keeps the CSUM_* definitions local to blob_t, and avoids passing
arguments around.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: defer csum calcuations sometimes

When we are doing a partial chunk overwrite, we need to defer the csum_data
update.  Otherwise, another write in the same transaction might need to
read part of the chunk, not find the data in the buffer cache, read it
from disk, and fail the csum check.

This patch defers the calculation until after we've build the transaction
and are about to commit to the kv store.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/bluestore: update based on Igor's feedback

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Fixes some issues when using Buffer Cache from _do_read and improves test coverage

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes invalid assert in  Buffer::truncate

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/objectstore: Adds trivial test case to verify buffer cache use in bluestore

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Adds cached buffer processing for _do_read

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add a very simple (incomplete) buffer cache

Attach it to each onode.

There is no trimming yet.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make tail cache a bit smarter

This is really a stop-gap.  Since we are doing reads in the pre-commit
write path, we need to have some sort of buffer cache so that a sequence
of writes in the same transaction can remain coherent (the second write
must "read" the first write in order to fill out the chunk).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump tail_bl state in _dump_onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Adds lacking methods in bluestore_compression_header_t to fix encoder test build

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add decompressor call to read path

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore/bluestore_types: add bluestore_compression_header_t

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

compressor: Refactor to allow bufferlist::iterator as an input

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: new write path

- simplified wal_op_t.  we still have overlays in there, although that
  might need to get removed soon too.
- init_csum cleanup
- totally new write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: verify blob ref_maps during fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: return EIO on csum verification error

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify _verify_csum

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cleanup _read_extent_sparse

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluetores: fix checksum verification

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: read patch code cleanup

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: replace _do_read with ExtentManager version

Replaces _do_read with he corresponding code from ExtentManager to support csum verification & decompression.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

doc/dev/bluestore: write path notes

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: new test SmallBlockWrites

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add BigWriteBigZero test

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: expand SimpleObjectTest a bit

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define min_alloc_size globally

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define block_size, block_mask, block_order globally

These don't change, no need to declare them all over the place.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: change block_size -> chunk_size for _pad_zero* methods

The granularity is not the same as block_size.. depends on what the caller
wants.  Use a more generic name.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use _txc_finalize_kv for wal allocation updates

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass txc to _do_wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename _txc_update_fm -> _txc_finalize_kv

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: break _txc_finalize into _txc_write_nodes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump csums in _dump_onode/bnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_bnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: nicer operator<< for blob_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add onode_t::compress_extent_map()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t convenience methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: FLAG_IMMUTABLE -> FLAG_MUTABLE

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t::is_unreferenced() method

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add blob_t::has_any_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: test {find,seek}_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t csum additions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: blob_t::init_csum() helper

Set up the checksum parameters and size csum_data appropriately.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track csum config options

Use global csum_type member to drive new checksum calcuations.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instantiate a Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prototype Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add buffer::list::iterator::get_ptr_and_advance

Efficiently iterate over buffers in a list, via an iterator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: extent_ref_map_t::intersects

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: change extent_ref_map_t

- no more add, or weird add vs get assumptions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: drop num_refs from blob_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: onode_t::punch_hole()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: disable clone cow until it works again

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix bad iterator deref in _do_zero_tail_extent

The extent_map may be empty, or not contain anything at this offset.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename enode -> bnode

We're not refcounting extents anymore; we're storing shared blobs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use hex almost everywhere

It's like going metric, but even better!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove block_map from onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: convert io path to extent_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add get_blob_ptr helpers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: show start and end offsets for buffer mismatch

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to enode

We'll remove the old ref_map once the users go away.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass c to _do_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to onode_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent_t for releases

Trying to remove the old extent_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent for wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix blob operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep an extent_ref_map for each blob

This lets us track which portions of a blob are still in use.  In some
cases, we may be able to split the blob to deallocate a portion of it.
In other cases, we will want this information to know whether to
recompress the blob (or whatever).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add map and map_bl methods to blob

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: data structure to support compression/checksum verification. Read handler prototype implementation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9183 from yehudasa/wip-rgw-sync-retry

rgw: sync retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9073 from ukernel/wip-posixacl-fix

client: fix setting empty default ACL

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9422 from liewegas/wip-bluefs

os/bluestore/BlueFS: wait for aios on shutdown

Merge pull request #9343 from dachary/wip-13522-ceph-disk

ceph-disk: workaround gperftool hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: wait for aios on shutdown

This fixes segv from unittest_bluefs.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: workaround gperftool hang

Temporary workaround: if ceph-osd --mkfs does not
complete within 5 minutes, assume it is blocked
because of https://github.com/gperftools/gperftools/issues/786

References http://tracker.ceph.com/issues/13522

Signed-off-by: Loic Dachary <loic@dachary.org>

qa/workunits/fs/misc: test for setting empty ACL

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9299 from cbodley/wip-rgw-vstart-valgrind

vstart: add --valgrind_rgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9375 from dillaman/wip-14937

rbd-mirror: replicate cloned images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9162 from xiexingguo/xxg-wip-fixbmap

os/bluestore: fix bmap allocator relevant issues

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9415 from ceph/wip-16095-parted

packaging: move parted requirement to -osd subpkg

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #9406 from tchaikov/wip-doc-hyperlink-of-rados_t

doc: add breathe_domain_by_extension to conf.py

os/bluestore/BitAllocator: set_bits() in batch

Which is simpler and faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix race condition for check_bit()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: use std namespace for atomic_*

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix is_allocated() may return false positive

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: basic cloned image test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

packaging: move parted requirement to -osd subpkg

Prior to this change, ceph-base required the "parted" package, which
meant that any installation of ceph-osd, ceph-mon, or ceph-mds would
pull in the parted package.

Move the parted dependency to ceph-osd, since ceph-disk is the only
thing that uses parted.

The advantage of this change is that Mon and MDS systems will not need
to install the parted package.

Fixes: http://tracker.ceph.com/issues/16095
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rbd-mirror: copy snapshot parent image settings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper method to set parent

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: add support for cloning images from mirrored parent

Fixes: http://tracker.ceph.com/issues/14937
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: extend internal API to clone from open parent image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper methods to retrieve mirror image metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine to open remote/read-only images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>