test/run-cli-tests: enable RBD testing

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test/librbd/Clang: Clang complains about reinterpret_cast where static_cast is sufficient

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

tools/rbd/action/Kernel.cc: create stubs for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

include/rbd_types.h: include/types.h for __le*

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

include/compat.h: define EREMOTEIO for FreeBSD

 - and combined some FreeBSD defines with Apple's

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common/event_socket.h: include errno.h to use errno

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

./do_freebsd.sh: Do not use KRBD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

CMakeLists.txt: Introduce WITH_KBRD for kernel RBD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12843 from trociny/wip-18465

librbd: metadata_set API operation should not change global config setting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12880 from wjwithagen/wip-wjw-static_cast-NULL

tests: fix NULL references to be acceptable by Clang

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12683 from SUSE/wip-run-make-check-deps

build/ops: add sanity checks to run-make-check.sh

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12864 from gaurav36/bug-18422-rbd-bench-write-crash-4G-io-size

rbd: bench-write should return error if io-size >= 4G

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12863 from dillaman/wip-rbd-coverity

librbd: resolve static analyser warnings

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12820 from trociny/wip-18447

rbd-mirror: check remote image mirroring state when bootstrapping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12465 from jcsp/wip-18159

mount: do not print "unknown" option to kclient

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #12794 from ukernel/wip-multimds-misc2

mds: misc multimds fixes part2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12842 from trociny/wip-break_lock_test

test/librbd: add break_lock test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tests: fix NULL references to be acceptable by Clang

 - On some platforms NULL evaluates to nullptr, which will require a
   cast to the right type to be able to compile.
      error: reinterpret_cast from 'nullptr_t' to 'ContextWQ *'

 - It is a delicate change since otherwise GCC will start complaining
   about the reverse:
      error: invalid static_cast from type 'const long int' to type 'ContextWQ*'

 By casting right at the instance of NULL both compilers are happy.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12839 from dillaman/wip-17993

librbd: delay mirror registration when creating clones

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12838 from dillaman/wip-18419

librbd: possible deadlock with flush if refresh in-progress

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12837 from dillaman/wip-18441

rbd-mirror: avoid processing new events after stop requested

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12846 from dillaman/wip-rbd-mirror-ha-merge

librbd: refactor exclusive lock support into generic managed lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #12832 from wjin/clean_async

msg/async: cleanups

Reviewed-by: Haomai Wang <haomai@xsky.com>

mds: fix MDCache::trim() with max == 0

MDCache::shutdown_pass call MDCache::trim with max == 0. If the last
dentry is non-null, MDCache::trim does nothing

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12840 from tchaikov/wip-libceph-common

cmake: turn libcommon into a shared library

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #12809 from wjin/optimize_atomic

msg/async: avoid atomic variable overhead

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12826 from LiumxNL/fix-op-tracker-initiate

mon: optracker's initiated_at timestamp should not be NULL

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12871 from xiexingguo/xxg-wip-bluestore-20170111

os/bluestore: kill BufferSpace.empty()

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: kill BufferSpace.empty()

Because:
1. All other public methods are called under the protection of cache->lock,
which makes the usage of this method is weird.

2. The only caller is _dump_extent_map(), and we can do it without this
in a simpler way.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12860 from liupan1111/wip-fix-dpdk-prefix

async: fixed the error "Cause: Cannot create lock on '/var/run/.rte_c…

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12867 from ceph/wip-ceph-deploy-workaround

qa/tasks/ceph-deploy: create-keys explicitly

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #12626 from adamemerson/wip-denconstruction

include/denc: improvements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12862 from optimistyzy/110_1

os/bluestore/NVMEdevice: fix the unrelease segs issue

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12091 from wjwithagen/wip-wjw-nosetests

nosetests: use /usr/bin/env to find nosetests

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

denc: Support boost::optional

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Support tuples

We could support zero-length tuples, but contiguous appender doesn't
like zero-length writes, so I've ruled them out for now.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Add support for std::array

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Support boost::container::flat_set

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Scrap the container boilerplate!

Using template templates, kill off the reduplication in
container encoding.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

nosetests: start using /usr/bin/env to find noestests

 - Option nocapture is not really required so no problem
   with the fact that FreeBSD env does not work for params
   for nosetests

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd: bench-write should return error if io-size >= 4G

Currently if user perform bench-write with io-size > 4G
then its crashing because currently during memory allocation
bufferptr taking size of buffer as a unsigned and io-size > 4G
will overflow with unsigned. so during memset operation it will
try to set io_size size of memory area pointed by bufferptr,
(bufferptr area is:  (4G - io_size)), so it will cause
segmentation fault.

Fix is to return error if io-size >= 4G

Fixes: http://tracker.ceph.com/issues/18422
Reported-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

Merge pull request #12821 from SUSE/wip-drop-rbd_cli_tests

tests: drop rbd_cli_tests.pl and RbdLib.pm

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: resolve static analyser warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12850 from xiexingguo/xxg-wip-bluestore-20170110

os/bluestore: shrink buffer_map key into uint32_t

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12815 from rjfd/wip-journal-unittest-build-error

tests: fix template specialization of PromoteRequest class

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12853 from xiexingguo/xxg-wip-kill-orphan

os/bluestore: kill orphan declaration of do_write_check_depth()

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12856 from xiexingguo/xxg-wip-clean-kd

os/bluestore/KernelDevice: kill zeros

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12858 from yangdongsheng/rbd_cleanup

rbd: cleanup: fix the typo in namespace comment

NVMEdevice: fix the unrelease segs issue

When the read or flush command fails, we need to
release segs. Or we can abort it like what we do
for write.

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

cmake: s/snappy/${SNAPPY_LIBRARIES}/

also s/z/${ZLIB_LIBARIES}/

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: turn libcommon into a shared library

prior to this change, libcommon is a convenient library which gets
linked into librados, librbd and libcephfs and all ceph executables.
this incurs some problems:
 - double dose of libcommon in memory space and HDD: waste of memory
   and disk space.
 - if an application links to two libraries including libcommon at the
   same time. take librados and libcephfs as an example, they could
   interfere with each other by changing the other guy's status.
after this change, libcommon is tuned into a shared library and
renamed to libceph-common. it will be installed into $prefix/lib/ceph,
and packaged in librados2.

ceph.spec.in,debian/librados2.install: package libceph-common in
  librados2.
CMakeLists.txt:
  - do not link against libboost-* if not necessary.
  - s/common/ceph-common/g
  - install libceph-common into $prefix/lib/ceph
  - set rpath to $prefix/lib/ceph
  - link against ceph-common if an executable needs access to non public
    symbols in ceph.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: shrink buffer_map key into uint32_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

async: fixed the error "Cause: Cannot create lock on '/var/run/.rte_config'.",
when invoke more than one osds in same server and enable dpdk.

Signed-off-by: Pan Liu <pan.liu@istuary.com>

rbd: cleanup: fix the typo in namespace comment

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #12855 from badone/wip-md_config_t-_apply_changes-unchangeable

config: Improve warning for unobserved value

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/KernelDevice: reuse r

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/KernelDevice: drop redundant assignment statements

The memset() call has already done it.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/NVMeDevice: kill zeros

See https://github.com/ceph/ceph/commit/2cd63d2209aaa0221a3f0a69b3f49ab1e01d9e20
The aio_zero() interface is deprecated.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/KernelDevice: kill zeros

This is not needed anymore and can save us 1MB memory.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: don't abort request that has already started slave operation

Access check and fragment size check can fail after start slave
operation. This causes mds to crash.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: use locked dentry trace to compose slave rmdir/rename request

{rmdir/rename}_prepare_witness() use full path to compose slave
requests. But they do not lock all dentries in the path. So someone
else changes the unlocked dentry and causes path travsese of slave
request to fail.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: add replicas of destdn to rename witness list

it's possible that null dentry is replicated

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly put unexpired dentry back to lru

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: cleanup Server::rdlock_path_xlock_dentry

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: add -ESTALE recovery code for unlink/rmdir

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: limit number of pending exports when thash exports is enabled

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: make MDCache::show_subtree less verbose when thrash exports is enabled

Subtree map can be very large when thrash exports is enabled.
Printing subtree map causes size log to increase quickly.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't journal inode when merging subtrees

When doing subtree migration or cross subtree rename, we rdlock
fragtreelock of subtree boundaries' inodes. rdlock fragtreelock
has already journaled dirfragtree updates. I can't see why we
need to journal it again when merging subtrees.

The resaon I want to remove this code is that it can journal
unconnected dirfrag in the Migrator::export_reverse() case.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix MDSMap::is_rejoining()

the function checks if all recovering mds reach rejoin state. It
need to consider damaged mds set.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove stale warning in log replay code

we don't re-fill prealloc_inos each time a prealloc ino get used

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: don't modify session->info before ESession get journaled

Server::journal_close_session() modifies session->info before
corresponding ESession get journaled. It's possible the updated
session gets saved to sessionmap before Server::_session_logged()
increases sessionmap's version. This confuses the journal replay
code.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

config: Improve warning for unobserved value

When a variable is not being observed we currently mark it
"unchangable". This can be misleading so try something hopefully a
little more informative.

Fixes: http://tracker.ceph.com/issues/18424
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12854 from liupan1111/wip-fix-async-dpdk-issue

async: fixed coredump when enable dpdk.

Reviewed-by: Haomai Wang <haomai@xsky.com>

async: fixed coredump when enable dpdk.

Signed-off-by: Pan Liu <pan.liu@istuary.com>

os/bluestore: kill orphan declaration of do_write_check_depth()

The gc logic is gone and this is not relevant anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12849 from xiexingguo/xxg-wip-fix-compile-crash

librados: fix compile errors from simplified aio completions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon: we may track a self-initiated message, recv_stamp remain unset

so using ceph_clock_now() instead to make optracker happy

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

Merge pull request #9032 from tchaikov/wip-15786

doc: document repair/scrub features

Reviewed-by: David Zafman <dzafman@redhat.com>

librados: fix building crash

See https://github.com/ceph/ceph/pull/12607

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Remove debug overrides

the high level of debug for mon/osd is causing remoto to hang during get key

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add ceph-create-keys to explicitly create admin/bootstrap keys

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

librbd: potential nullptr dereference in ManagedLock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12726 from liewegas/wip-18372

osd: _exit() intead of exit() for failure injection

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12607 from liewegas/wip-librados-killack

librados,osdc: kill ack vs commit distinction

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12695 from hjwsm1989/mark-unfound-lost

osd/PrimaryLogPG: optimal pick_newest_available

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12724 from kylinstorage/wip-fix-coverity-warning-for-uninitialized-members

osd: fix coverity warning for uninitialized members

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12725 from liewegas/wip-bluestore-bluefs-reclaim

os/bluestore/BlueFS: fix reclaim_blocks

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #12727 from liewegas/wip-18369

osd/PG: publish PG stats when backfill-related states change

Reviewed-by: Samuel Just <sjust@redhat.com>

librbd: rewatch complete callback should be invoked w/o lock

It's currently being incorrectly invoked twice -- but the direct
callback is holding the Watcher lock. This results in a recursive
lock when ManagedLock attempts to retrieve the watch id from the
Watcher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: metadata_set API operation should not change global config setting

Fixes: http://tracker.ceph.com/issues/18465
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/librbd: add break_lock test

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12731 from oritwas/wip-rgw-realm-set

rgw realm set fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: fix template specialization of PromoteRequest class

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: delay mirror registration when creating clones

Fixes: http://tracker.ceph.com/issues/17993
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: check image mirroring state when bootstrapping

Fixes: http://tracker.ceph.com/issues/18447
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: async request to test if image is primary

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: possible deadlock with flush if refresh in-progress

Fixes: http://tracker.ceph.com/issues/18419
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: avoid processing new events after stop requested

Fixes: http://tracker.ceph.com/issues/18441
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: hold owner lock when testing if lock owner

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12274 from ukernel/wip-multimds-misc

mds: misc multimds fixes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>