os/bluestore/bluestore_types: add blob_t::has_any_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: test {find,seek}_lextents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t csum additions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: blob_t::init_csum() helper

Set up the checksum parameters and size csum_data appropriately.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track csum config options

Use global csum_type member to drive new checksum calcuations.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instantiate a Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prototype Checksummer

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add buffer::list::iterator::get_ptr_and_advance

Efficiently iterate over buffers in a list, via an iterator.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: extent_ref_map_t::intersects

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: change extent_ref_map_t

- no more add, or weird add vs get assumptions

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: drop num_refs from blob_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: onode_t::punch_hole()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: disable clone cow until it works again

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix bad iterator deref in _do_zero_tail_extent

The extent_map may be empty, or not contain anything at this offset.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename enode -> bnode

We're not refcounting extents anymore; we're storing shared blobs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use hex almost everywhere

It's like going metric, but even better!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove block_map from onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: convert io path to extent_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add get_blob_ptr helpers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: show start and end offsets for buffer mismatch

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to enode

We'll remove the old ref_map once the users go away.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass c to _do_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to onode_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent_t for releases

Trying to remove the old extent_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent for wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix blob operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep an extent_ref_map for each blob

This lets us track which portions of a blob are still in use.  In some
cases, we may be able to split the blob to deallocate a portion of it.
In other cases, we will want this information to know whether to
recompress the blob (or whatever).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add map and map_bl methods to blob

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: data structure to support compression/checksum verification. Read handler prototype implementation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9183 from yehudasa/wip-rgw-sync-retry

rgw: sync retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9073 from ukernel/wip-posixacl-fix

client: fix setting empty default ACL

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9422 from liewegas/wip-bluefs

os/bluestore/BlueFS: wait for aios on shutdown

Merge pull request #9343 from dachary/wip-13522-ceph-disk

ceph-disk: workaround gperftool hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: wait for aios on shutdown

This fixes segv from unittest_bluefs.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: workaround gperftool hang

Temporary workaround: if ceph-osd --mkfs does not
complete within 5 minutes, assume it is blocked
because of https://github.com/gperftools/gperftools/issues/786

References http://tracker.ceph.com/issues/13522

Signed-off-by: Loic Dachary <loic@dachary.org>

qa/workunits/fs/misc: test for setting empty ACL

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9299 from cbodley/wip-rgw-vstart-valgrind

vstart: add --valgrind_rgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9375 from dillaman/wip-14937

rbd-mirror: replicate cloned images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9162 from xiexingguo/xxg-wip-fixbmap

os/bluestore: fix bmap allocator relevant issues

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9415 from ceph/wip-16095-parted

packaging: move parted requirement to -osd subpkg

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #9406 from tchaikov/wip-doc-hyperlink-of-rados_t

doc: add breathe_domain_by_extension to conf.py

os/bluestore/BitAllocator: set_bits() in batch

Which is simpler and faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix race condition for check_bit()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: use std namespace for atomic_*

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix is_allocated() may return false positive

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: basic cloned image test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

packaging: move parted requirement to -osd subpkg

Prior to this change, ceph-base required the "parted" package, which
meant that any installation of ceph-osd, ceph-mon, or ceph-mds would
pull in the parted package.

Move the parted dependency to ceph-osd, since ceph-disk is the only
thing that uses parted.

The advantage of this change is that Mon and MDS systems will not need
to install the parted package.

Fixes: http://tracker.ceph.com/issues/16095
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rbd-mirror: copy snapshot parent image settings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper method to set parent

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: add support for cloning images from mirrored parent

Fixes: http://tracker.ceph.com/issues/14937
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: extend internal API to clone from open parent image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper methods to retrieve mirror image metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine to open remote/read-only images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: support size suffixes for --max-size in radosgw-admin command

rgw: support size suffixes for --max-size in radosgw-admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

remove references to `mds destroy` in ceph-deployReviewed-by: John Wilkins

remove references to `mds destroy` in ceph-deploy

Reviewed-by: John Wilkins jowilkin@redhat.com

unittest_bluefs: debug bdev, bluefs

Hopefully this will catch the intermittent segv we are
seeing.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: ceph-deploy man: remove references to mds destroy. Not implemented

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: style fixes for gen_state_diagram

Merge pull request #8915 from Vicente-Cheng/wip-clean-client-lock

client: correct the client_lock order when cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9214 from bniver/wip-bniver-9904

mds: log on bad seq

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9391 from Adirl/bug_fix_xcon

xio: clean xcon on connection teardown

Looks good--taking the new guidance as authoritative.

Merge pull request #9137 from majianpeng/common-blkdev-accept-symbolic-link

common/blkdev: accept symbolic link

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add breathe_domain_by_extension to conf.py

so it can render the doxygen generated objects with the proper domain by
checking the extension of rendered source file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9350 from rzarzynski/wip-rgw-remove-unused-token_tag

rgw, cleanup: drop unused variable token_tag in rgw_swift_auth.cc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

xio: clean xcon on connection teardown

Accelio could raise connection teardown event without
a disconnected/closed/refused/error event before it which will
leave the xcon reference in XioMessenger entity map and conn list
which will also cause an assertion from boost.

Signed-off-by: Roi Dayan <roid@mellanox.com>

Merge pull request #9360 from dillaman/wip-16045

rbd-mirror: support multiple replicated pools

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: cluster-level asok commands need to support multiple pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9284 from tchaikov/wip-cmake-doc

cmake,make: speed up man page generation

Reviewed-by: François Lafont <francois.lafont@ac-versailles.fr>

Merge pull request #8738 from xiexingguo/xxg-wip-fstool

tool/cephfs: make EventOutput::binary() idempotent

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8778 from ukernel/wip-open-symlink

mds: fix open symlink and truncate bugs

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #9187 from liewegas/wip-cppool

rados: refuse to cppool if there are snaps; warn about user_version

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9222 from xiexingguo/xxg-wip-scrubq-assert

mon: fix assert triggerred by the deprecated "scrubq" state

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9233 from vumrao/wip-vumrao-15809

mon : Display full flag in ceph status if full flag is set

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9234 from stiopaa1/osd_osdcap_preventCopyingVectorInConstructor

osd/OSDCap: prevent copying vector/strings

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9262 from songbaisen/song27

mon: remove the redundant judge on dispatch_op function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9298 from liewegas/wip-16012

osd: reject rados ops with bad PARALLELEXEC flag

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9379 from ceph/wip-cmake

cmake: add ObjectMapIterate.cc to librbd

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix build when WITH_RADOSGW is set OFF

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9382 from xiexingguo/xxg-wip-fixrbdut

test: fix ut failure caused by rbd usage update

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: fix ut failure caused by rbd usage update

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: librbd missing source file

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9347 from tchaikov/wip-cmake

cmake: some fixes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #8562 from ceph/wip-djf-object-map-check

rbd: add rbd object-map check

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9280 from zhuangzeqiang/wip15522

rbd: check value of config override when setting image-meta

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: check value of config override when setting image-meta

Fixes: http://tracker.ceph.com/issues/15522
Signed-off-by: zhuangzeqiang <zhuang.zeqiang@h3c.com>

Merge pull request #9326 from marcan/rbd-aio-completion

librbd: only mark async completion as done after callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind: misc fixesReviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind: misc fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9322 from trociny/wip-16011

test: failure in journal.sh workunit test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: remove cls_lock_client from libcephfs

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated src in osd_srcs

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add libcommon to libmds

MDSDaemon is using cpu_profiler_handle_command() offered by libcommon

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/ENABLE_SHARED/BUILD_SHARED_LIBS/

* BUILD_SHARED_LIB is a cmake variable respected by add_library().
  so we can simply expose it as an option instead of reinventing our own.
* and mark internal libraries as STATIC explicitly
* and add keyutil to librbd as secret.cc is using it. and secret.cc is
  linked into librbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add dl to libcommon

PluginRegistry.cc uses dlopen() functions, so added it to libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add missing obj to libosd

* add common_util_obj as OSD::_collect_metadata() is calling collect_sys_info(), which lives
  in util.c
* add global_common_objs as OSD::_committed_osd_maps() is calling
  queue_async_signal(), which lives in signal_handler.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use GNUInstallDirs for defining install dest directories

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install include/memory.h also

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install librbd headers

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install libradosstriper headers

also respect BUILD_SHARED_LIBS instead of always building this lib as a
static library. and we install .so in rpm/deb packaging scripts.

Signed-off-by: Kefu Chai <kchai@redhat.com>