os/bluestore: add decompressor call to read path

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore/bluestore_types: add bluestore_compression_header_t

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

compressor: Refactor to allow bufferlist::iterator as an input

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: new write path

- simplified wal_op_t.  we still have overlays in there, although that
  might need to get removed soon too.
- init_csum cleanup
- totally new write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: verify blob ref_maps during fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: return EIO on csum verification error

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify _verify_csum

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cleanup _read_extent_sparse

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluetores: fix checksum verification

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: read patch code cleanup

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: replace _do_read with ExtentManager version

Replaces _do_read with he corresponding code from ExtentManager to support csum verification & decompression.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

doc/dev/bluestore: write path notes

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: new test SmallBlockWrites

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add BigWriteBigZero test

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: expand SimpleObjectTest a bit

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define min_alloc_size globally

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: define block_size, block_mask, block_order globally

These don't change, no need to declare them all over the place.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: change block_size -> chunk_size for _pad_zero* methods

The granularity is not the same as block_size.. depends on what the caller
wants.  Use a more generic name.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use _txc_finalize_kv for wal allocation updates

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass txc to _do_wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename _txc_update_fm -> _txc_finalize_kv

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: break _txc_finalize into _txc_write_nodes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump csums in _dump_onode/bnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_bnode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: nicer operator<< for blob_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add onode_t::compress_extent_map()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t convenience methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: FLAG_IMMUTABLE -> FLAG_MUTABLE

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t::is_unreferenced() method

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add blob_t::has_any_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: test {find,seek}_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t csum additions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: blob_t::init_csum() helper

Set up the checksum parameters and size csum_data appropriately.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track csum config options

Use global csum_type member to drive new checksum calcuations.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instantiate a Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prototype Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add buffer::list::iterator::get_ptr_and_advance

Efficiently iterate over buffers in a list, via an iterator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: extent_ref_map_t::intersects

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: change extent_ref_map_t

- no more add, or weird add vs get assumptions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: drop num_refs from blob_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: onode_t::punch_hole()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: disable clone cow until it works again

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix bad iterator deref in _do_zero_tail_extent

The extent_map may be empty, or not contain anything at this offset.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename enode -> bnode

We're not refcounting extents anymore; we're storing shared blobs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use hex almost everywhere

It's like going metric, but even better!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove block_map from onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: convert io path to extent_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add get_blob_ptr helpers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: show start and end offsets for buffer mismatch

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to enode

We'll remove the old ref_map once the users go away.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass c to _do_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to onode_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent_t for releases

Trying to remove the old extent_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent for wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix blob operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep an extent_ref_map for each blob

This lets us track which portions of a blob are still in use.  In some
cases, we may be able to split the blob to deallocate a portion of it.
In other cases, we will want this information to know whether to
recompress the blob (or whatever).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add map and map_bl methods to blob

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: data structure to support compression/checksum verification. Read handler prototype implementation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9183 from yehudasa/wip-rgw-sync-retry

rgw: sync retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9073 from ukernel/wip-posixacl-fix

client: fix setting empty default ACL

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9422 from liewegas/wip-bluefs

os/bluestore/BlueFS: wait for aios on shutdown

Merge pull request #9343 from dachary/wip-13522-ceph-disk

ceph-disk: workaround gperftool hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: wait for aios on shutdown

This fixes segv from unittest_bluefs.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: workaround gperftool hang

Temporary workaround: if ceph-osd --mkfs does not
complete within 5 minutes, assume it is blocked
because of https://github.com/gperftools/gperftools/issues/786

References http://tracker.ceph.com/issues/13522

Signed-off-by: Loic Dachary <loic@dachary.org>

qa/workunits/fs/misc: test for setting empty ACL

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9299 from cbodley/wip-rgw-vstart-valgrind

vstart: add --valgrind_rgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9375 from dillaman/wip-14937

rbd-mirror: replicate cloned images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9162 from xiexingguo/xxg-wip-fixbmap

os/bluestore: fix bmap allocator relevant issues

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9415 from ceph/wip-16095-parted

packaging: move parted requirement to -osd subpkg

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #9406 from tchaikov/wip-doc-hyperlink-of-rados_t

doc: add breathe_domain_by_extension to conf.py

os/bluestore/BitAllocator: set_bits() in batch

Which is simpler and faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix race condition for check_bit()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: use std namespace for atomic_*

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix is_allocated() may return false positive

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: basic cloned image test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

packaging: move parted requirement to -osd subpkg

Prior to this change, ceph-base required the "parted" package, which
meant that any installation of ceph-osd, ceph-mon, or ceph-mds would
pull in the parted package.

Move the parted dependency to ceph-osd, since ceph-disk is the only
thing that uses parted.

The advantage of this change is that Mon and MDS systems will not need
to install the parted package.

Fixes: http://tracker.ceph.com/issues/16095
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rbd-mirror: copy snapshot parent image settings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper method to set parent

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: add support for cloning images from mirrored parent

Fixes: http://tracker.ceph.com/issues/14937
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: extend internal API to clone from open parent image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper methods to retrieve mirror image metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine to open remote/read-only images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: support size suffixes for --max-size in radosgw-admin command

rgw: support size suffixes for --max-size in radosgw-admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

remove references to `mds destroy` in ceph-deployReviewed-by: John Wilkins

remove references to `mds destroy` in ceph-deploy

Reviewed-by: John Wilkins jowilkin@redhat.com

unittest_bluefs: debug bdev, bluefs

Hopefully this will catch the intermittent segv we are
seeing.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: ceph-deploy man: remove references to mds destroy. Not implemented

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: style fixes for gen_state_diagram

Merge pull request #8915 from Vicente-Cheng/wip-clean-client-lock

client: correct the client_lock order when cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9214 from bniver/wip-bniver-9904

mds: log on bad seq

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9391 from Adirl/bug_fix_xcon

xio: clean xcon on connection teardown

Looks good--taking the new guidance as authoritative.

Merge pull request #9137 from majianpeng/common-blkdev-accept-symbolic-link

common/blkdev: accept symbolic link

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add breathe_domain_by_extension to conf.py

so it can render the doxygen generated objects with the proper domain by
checking the extension of rendered source file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9350 from rzarzynski/wip-rgw-remove-unused-token_tag

rgw, cleanup: drop unused variable token_tag in rgw_swift_auth.cc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

xio: clean xcon on connection teardown

Accelio could raise connection teardown event without
a disconnected/closed/refused/error event before it which will
leave the xcon reference in XioMessenger entity map and conn list
which will also cause an assertion from boost.

Signed-off-by: Roi Dayan <roid@mellanox.com>

Merge pull request #9360 from dillaman/wip-16045

rbd-mirror: support multiple replicated pools

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: cluster-level asok commands need to support multiple pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9284 from tchaikov/wip-cmake-doc

cmake,make: speed up man page generation

Reviewed-by: François Lafont <francois.lafont@ac-versailles.fr>

Merge pull request #8738 from xiexingguo/xxg-wip-fstool

tool/cephfs: make EventOutput::binary() idempotent

Reviewed-by: John Spray <john.spray@redhat.com>