os/bluestore: use hex almost everywhere

It's like going metric, but even better!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove block_map from onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: convert io path to extent_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add get_blob_ptr helpers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: show start and end offsets for buffer mismatch

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to enode

We'll remove the old ref_map once the users go away.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pass c to _do_read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add blob_map to onode_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent_t for releases

Trying to remove the old extent_t

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use pextent for wal_op

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix blob operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep an extent_ref_map for each blob

This lets us track which portions of a blob are still in use.  In some
cases, we may be able to split the blob to deallocate a portion of it.
In other cases, we will want this information to know whether to
recompress the blob (or whatever).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add map and map_bl methods to blob

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: data structure to support compression/checksum verification. Read handler prototype implementation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9183 from yehudasa/wip-rgw-sync-retry

rgw: sync retry

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9073 from ukernel/wip-posixacl-fix

client: fix setting empty default ACL

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9422 from liewegas/wip-bluefs

os/bluestore/BlueFS: wait for aios on shutdown

Merge pull request #9343 from dachary/wip-13522-ceph-disk

ceph-disk: workaround gperftool hang

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: wait for aios on shutdown

This fixes segv from unittest_bluefs.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: workaround gperftool hang

Temporary workaround: if ceph-osd --mkfs does not
complete within 5 minutes, assume it is blocked
because of https://github.com/gperftools/gperftools/issues/786

References http://tracker.ceph.com/issues/13522

Signed-off-by: Loic Dachary <loic@dachary.org>

qa/workunits/fs/misc: test for setting empty ACL

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9299 from cbodley/wip-rgw-vstart-valgrind

vstart: add --valgrind_rgw

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9375 from dillaman/wip-14937

rbd-mirror: replicate cloned images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9162 from xiexingguo/xxg-wip-fixbmap

os/bluestore: fix bmap allocator relevant issues

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9415 from ceph/wip-16095-parted

packaging: move parted requirement to -osd subpkg

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #9406 from tchaikov/wip-doc-hyperlink-of-rados_t

doc: add breathe_domain_by_extension to conf.py

os/bluestore/BitAllocator: set_bits() in batch

Which is simpler and faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix race condition for check_bit()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: use std namespace for atomic_*

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/BitAllocator: fix is_allocated() may return false positive

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: basic cloned image test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

packaging: move parted requirement to -osd subpkg

Prior to this change, ceph-base required the "parted" package, which
meant that any installation of ceph-osd, ceph-mon, or ceph-mds would
pull in the parted package.

Move the parted dependency to ceph-osd, since ceph-disk is the only
thing that uses parted.

The advantage of this change is that Mon and MDS systems will not need
to install the parted package.

Fixes: http://tracker.ceph.com/issues/16095
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rbd-mirror: copy snapshot parent image settings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper method to set parent

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: add support for cloning images from mirrored parent

Fixes: http://tracker.ceph.com/issues/14937
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: extend internal API to clone from open parent image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: asynchronous helper methods to retrieve mirror image metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: helper state machine to open remote/read-only images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: support size suffixes for --max-size in radosgw-admin command

rgw: support size suffixes for --max-size in radosgw-admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

remove references to `mds destroy` in ceph-deployReviewed-by: John Wilkins

remove references to `mds destroy` in ceph-deploy

Reviewed-by: John Wilkins jowilkin@redhat.com

unittest_bluefs: debug bdev, bluefs

Hopefully this will catch the intermittent segv we are
seeing.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: ceph-deploy man: remove references to mds destroy. Not implemented

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: style fixes for gen_state_diagram

Merge pull request #8915 from Vicente-Cheng/wip-clean-client-lock

client: correct the client_lock order when cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9214 from bniver/wip-bniver-9904

mds: log on bad seq

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9391 from Adirl/bug_fix_xcon

xio: clean xcon on connection teardown

Looks good--taking the new guidance as authoritative.

Merge pull request #9137 from majianpeng/common-blkdev-accept-symbolic-link

common/blkdev: accept symbolic link

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add breathe_domain_by_extension to conf.py

so it can render the doxygen generated objects with the proper domain by
checking the extension of rendered source file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9350 from rzarzynski/wip-rgw-remove-unused-token_tag

rgw, cleanup: drop unused variable token_tag in rgw_swift_auth.cc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

xio: clean xcon on connection teardown

Accelio could raise connection teardown event without
a disconnected/closed/refused/error event before it which will
leave the xcon reference in XioMessenger entity map and conn list
which will also cause an assertion from boost.

Signed-off-by: Roi Dayan <roid@mellanox.com>

Merge pull request #9360 from dillaman/wip-16045

rbd-mirror: support multiple replicated pools

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: cluster-level asok commands need to support multiple pools

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9284 from tchaikov/wip-cmake-doc

cmake,make: speed up man page generation

Reviewed-by: François Lafont <francois.lafont@ac-versailles.fr>

Merge pull request #8738 from xiexingguo/xxg-wip-fstool

tool/cephfs: make EventOutput::binary() idempotent

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8778 from ukernel/wip-open-symlink

mds: fix open symlink and truncate bugs

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #9187 from liewegas/wip-cppool

rados: refuse to cppool if there are snaps; warn about user_version

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9222 from xiexingguo/xxg-wip-scrubq-assert

mon: fix assert triggerred by the deprecated "scrubq" state

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9233 from vumrao/wip-vumrao-15809

mon : Display full flag in ceph status if full flag is set

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9234 from stiopaa1/osd_osdcap_preventCopyingVectorInConstructor

osd/OSDCap: prevent copying vector/strings

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9262 from songbaisen/song27

mon: remove the redundant judge on dispatch_op function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9298 from liewegas/wip-16012

osd: reject rados ops with bad PARALLELEXEC flag

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9379 from ceph/wip-cmake

cmake: add ObjectMapIterate.cc to librbd

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix build when WITH_RADOSGW is set OFF

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9382 from xiexingguo/xxg-wip-fixrbdut

test: fix ut failure caused by rbd usage update

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: fix ut failure caused by rbd usage update

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: librbd missing source file

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9347 from tchaikov/wip-cmake

cmake: some fixes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #8562 from ceph/wip-djf-object-map-check

rbd: add rbd object-map check

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9280 from zhuangzeqiang/wip15522

rbd: check value of config override when setting image-meta

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: check value of config override when setting image-meta

Fixes: http://tracker.ceph.com/issues/15522
Signed-off-by: zhuangzeqiang <zhuang.zeqiang@h3c.com>

Merge pull request #9326 from marcan/rbd-aio-completion

librbd: only mark async completion as done after callback

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind: misc fixesReviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind: misc fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9322 from trociny/wip-16011

test: failure in journal.sh workunit test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: remove cls_lock_client from libcephfs

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated src in osd_srcs

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add libcommon to libmds

MDSDaemon is using cpu_profiler_handle_command() offered by libcommon

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/ENABLE_SHARED/BUILD_SHARED_LIBS/

* BUILD_SHARED_LIB is a cmake variable respected by add_library().
  so we can simply expose it as an option instead of reinventing our own.
* and mark internal libraries as STATIC explicitly
* and add keyutil to librbd as secret.cc is using it. and secret.cc is
  linked into librbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add dl to libcommon

PluginRegistry.cc uses dlopen() functions, so added it to libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add missing obj to libosd

* add common_util_obj as OSD::_collect_metadata() is calling collect_sys_info(), which lives
  in util.c
* add global_common_objs as OSD::_committed_osd_maps() is calling
  queue_async_signal(), which lives in signal_handler.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use GNUInstallDirs for defining install dest directories

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install include/memory.h also

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install librbd headers

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install libradosstriper headers

also respect BUILD_SHARED_LIBS instead of always building this lib as a
static library. and we install .so in rpm/deb packaging scripts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile rbd-fuse

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install debug tools into usr/bin

otherwise dh_install won't find them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install mount.ceph into /sbin

should use absolute path otherwise it is installed into /usr/sbin, and
dh_install won't find it this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install bash-completion scripts also

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add missing installs

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: make shared library of lttng tracing instead

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: revamp on lttng support

* Findlttng-ust.cmake: detect lttng-gen-tp, and make it a required var
* Findlttng-ust.cmake: fix the detection of lttng
* src/tracing/CMakeLists.txt:
  - do not put the generated header files into ${CMAKE_SOURCE_DIR}/src/tracing,
    instead we should put the generated files into ${PROJECT_BINARY_DIR}/include.
  - do not compile the tracing library using the .c files in the repo,
    instead, we should generate them at compile time using lttng-gen-tp,
    and compile the genererated .c files.
* CMakeLists.txt: make the lttng-use package REQUIRED if WITH_LTTNG=ON
* src/CMakeLists.txt: let librados depend on librados-tp if LTTNG is
  enabled: librados includes tracing/librados.h. the same applies to
  librbd, libos and libosd. cmake could deduce the dependencies if we
  add the lttng header files under include/tracing into the
  corresponding target, as the lttng header files are already listed
  as OUTPUT and GENERATED files, but this requires us to put all the
  involved targets in the same CMakeLists.txt. in this case, three
  CMakeLists.txts are impacted. so we create a new target for each
  lttng header, and let the impacted target depend on it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9344 from renhwztetecs/renhw-wip-mdsdaemon

mds: mds-daemon init fixup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9112 from renhwztetecs/renhw-wip-mdbalancer

mds/mdbalancer: fixup assignment type in the hit_dir

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9335 from ceph/wip-16032-radosgw-flask

packaging: drop python-flask dependency from radosgw

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #9309 from dillaman/wip-15949

journal: support asynchronous shutdown

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: replayer should only handle a single pool

The pool watcher now only needs to scan a single pool for
its associated replayer since a peer is inherently tied to
a single pool.

Fixes: http://tracker.ceph.com/issues/16045
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: group peers by local pools

The peer structure is automatically unique per pool due to its
UUID, so grouping local pools by a single peer doesn't work.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: stop stale replayers before starting new replayers

If the connection details are tweaked for a remote peer, stop
the existing replayer before potentially starting a new replayer
against the same remote.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: normalize debug log message prefix

Signed-off-by: Jason Dillaman <dillaman@redhat.com>