os/bluestore: remove build warning in a better way

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11915 from ifed01/wip-bluestore-fix-test

os/test/store_test: fix legacy bluestore cache settings application

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11905 from xiexingguo/xxg-wip-kill-boost-warn

os/bluestore: fix compiler warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11896 from ifed01/wip-bluestore-cleanup

os/bluestore: cleanup around Blob::ref_map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11710 from ceph/wip-jlayton-fsync

Fix long stalls when calling ceph_fsync()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11293 from liewegas/wip-mon-markout

mon: preserve osd weight when marking osd out, then in

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/test/store_test: fix legacy bluestore cache settings application

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: make Blob::ref_map private

Signed-off-by: Igor Fedotov <ifedotov@minratis.com>

os/bluestore: start using Blob::get_ref over the code

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

client: rename flush_caps() with no arguments to flush_caps_sync()

Per Greg's recommendation, change the name of this function to better
indicate what it does now that we always request a journal flush on
the last cap flush.

Also, add a comment above the function to better explain why we do this.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: request journal flush after flushing caps in syncfs

Ensure that we ask the MDS to flush the journal on the last cap flush
from sync_fs and umount codepaths.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mds: only update change_attr and btime when client sets appropriate feature flags

The kernel client lags the userland code a bit, and feature support for
addr2 is not quite ready. Still, we want to allow the client to set the
new flags field in a cap request before then so it can get better fsync
performance.

When we go to update the cap fields, grab the features from the peer,
and verify that the appropriate flags are set before we apply updates
to the btime and change_attr.

Also, just have the function return early if dirty is 0, since it's
a no-op in that case, and turn the comment above the function into
an assertion.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: wire up the CHECK_CAPS_SYNCHRONOUS flag

Ensure that the client will request an immediate journal flush from the
MDS when we'll end up waiting on the flush response. This patch should
fix the fsync codepath, but we may need something similar for syncfs.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: change no_delay flag to a flags field

In a later patch, we'll want to have the client set the sync flag in
the cap flush, to hint to the MDS that it should process it immediately.

We could add a second bool, but let's instead do what the kernel client
does which is to have a flags field. With that, the existing no_delay
bool becomes CHECK_CAPS_NODELAY.

We'll add other flags in subsequent patches.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mds: do mds log flush if CLIENT_CAPS_SYNC is set

If the client has set the sync flag in a cap update, then it
is indicating that it's waiting on the reply. Ensure that we flush
the journal in that case.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #11790 from jcsp/wip-mclientcaps

client: remove unneeded layout on MClientCaps

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #11789 from jcsp/wip-17798

mds: require MAY_SET_POOL to set pool_ns

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11783 from jcsp/wip-17796

mds: enable rmxattr on pool_namespace attrs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11739 from SUSE/wip-17747

mds: remove "--journal-check" help text

Merge pull request #11692 from stiopaa1/mds_flock_addConstToMemFun

mds/flock: add const to member functions

Merge pull request #11545 from majianpeng/osdc-objectcache-fix-bug

osdc: After write try merge bh.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10853 from wido/docs-mon-pool-settings

doc: Add MON docs about pool flags and pool removal config settings

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Add MON docs about pool flags and pool removal config settings

Signed-off-by: Wido den Hollander <wido@42on.com>

os/bluestore: fix compiler warnings

As follows:

/home/jenkins-build/build/workspace/ceph-pull-requests/build/boost/include/boost/intrusive/pointer_plus_bits.hpp: In member function ‘bool BlueStore::ExtentMap::encode_some(uint32_t, uint32_t, ceph::bufferlist&, unsigned int*)’:
/home/jenkins-build/build/workspace/ceph-pull-requests/build/boost/include/boost/intrusive/pointer_plus_bits.hpp:76:7: warning: ‘dummy’ is used uninitialized in this function [-Wuninitialized]
       n = pointer(uintptr_t(p) | (uintptr_t(n) & Mask));
       ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueStore.cc:1779:10: note: ‘dummy’ was declared here
   Extent dummy(offset);

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11902 from ceph/wip-krbd-mons-fix

krbd: kernel client expects ip[:port], not an entity_addr_t

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11903 from dmick/master

README: hint at where to look to diagnose test failures

README: hint at where to look to diagnose test failures

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #11895 from xiexingguo/xxg-wip-bluestore-blob-range

os/bluestore: readability improvements and doxygen fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11900 from linuxbox2/fix-civetweb-link

cmake: replace civetweb symlink w/file copy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

krbd: kernel client expects ip[:port], not an entity_addr_t

Commit 2ee1b9a4084f ("krbd.cc: don't rely on MonMap internal members")
inadvertently dropped .get_sockaddr() call, breaking rbd map.  Fix it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11897 from liewegas/wip-pool-delete

tests: fix tests vs pool deletion

The default changed to disallow pool delete as of #11665; the tests assume it's allowed.
Reviewed-by: Dan Mick <dmick@redhat.com>

cmake: replace civetweb symlink w/file copy

Intended to remove an apparent race.  The two effects are
1. replace top-level command callouts w/file builtins
2. do them in the src/rgw sub-cmake

This is cleaner, and ideally avoids the race.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

vstart.sh: allow pool deletion

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/ceph-helpers.sh: allow pool deletes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11894 from xiexingguo/xxg-wip-bluestore-1110

os/bluestore: speed up omap-key generation for same onode(the read paths)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11893 from xiexingguo/xxg-wip-bluestore-fix-clear

os/bluestore: fix potential memory leak

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11892 from tchaikov/wip-move-context

Context: std::move the callback param in FunctionContext's ctor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: doxygen fixes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11889 from trociny/wip-vstart-bashism

vstart.sh: fix bashism in the script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11601 from ceph/wip-ms-async-keepalive2

msg/async/AsyncConnection: dispatch write handler on keepalive2

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11633 from ddiss/tcc17684_rados_segv_without_pool

rados: fix segfaults when run without --pool

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11665 from wido/mon-prevent-pool-delete

mon: Do not allow pools to be deleted by default

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11795 from runsisi/wip-fix-osd-noin

mon/OSDMonitor: newly created osd should not be wrongly marked in

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10049 from sponce/wip-striper-async-rm-master

librados: improvements async IO in librados and libradosstriper

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Context: std::move the callback param in FunctionContext's ctor

for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11883 from jcsp/wip-mgr-functioncontext

mgr: remove unnecessary C_StdFunction

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: improve the readability of Extent::blob_escapes_range()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix potential memory leak

Call to extent_map.clear_and_dispose() will be noop after extent_map.clear()
hence causing memory leak.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

vstart.sh: fix bashism in the script

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: speed up omap-key generation for same onode(the read paths)

In https://github.com/ceph/ceph/pull/11807 we only modify the write paths,
this patch fixes the read parts too, which are more time-sensitive.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11886 from stiopaa1/mds_mantle_includeCorrectHeaders

mds/Mantle.h: include correct header files

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11683 from wjwithagen/wip-wjw-freebsd-vstart

src/vstart.sh: Only execute btrfs if it is available

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11702 from wjwithagen/wip-wjw-freebsd-cephtool-test

qa/workunits/cephtool/test.sh: FreeBSD has no distro.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11885 from cbodley/wip-python-rgw-deb

deb: update python-rgw dependencies to librgw2

Reviewed-by: Sage Weil <sage@redhat.com>

mds/Mantle.h: include correct header files

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

deb: update python-rgw dependencies to librgw2

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11881 from linuxbox2/rgw-file-fsid

rgw_file: fix spurious mount entries w/Linux NFS client
verified that failed check is in osd-scrub-repair.sh

Merge pull request #11882 from yehudasa/wip-yehuda-spdk

spdk: a few fixes

Reviewed-by: Sage Weil <sage@redhat.com>

mgr: remove unnecessary C_StdFunction

FunctionContext works fine and already existed.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11811 from tanghaodong25/reap-ioc-wip

os/bluestore: reap ioc when stopping aio_thread.

Reviewed-by: Sage Weil <sage@redhat.com>

rgw_file: fix spurious mount entries w/Linux NFS client

This change fixes the initialization of root file handles to have
their correct fs instance.  Previously, the root file handle had
fs_inst-1.  The apparent change in fsid in a subordinate directory
triggers the client's behavior for new filesystem mounts, which
we don't want.

Fixes: http://tracker.ceph.com/issues/17850
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #11832 from ceph/wip-python-rgw

deb: add python-rgw packages

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

debian: add python-rgw package

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11845 from xiexingguo/xxg-wip-bs-1109

os/bluestore: remove duplicated namespace of tx state

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11859 from ceph/wip-py-rgw

doc: add python-rgw doc

Reviewed-by: Haomai Wang <haomai@xsky.com>

client/mds: add flags field to MClientCaps

...and encode/decode it appropriately.

The idea of this field is to be advisory, to allow the MDS to handle
things differently if it so chooses. We deliberately do _not_ offer
any firm policy here.

We start with a flag that tells the MDS when an application may end up
blocking on the result of this cap request.

A new "sync" arg is added to send_cap, and we set the new flag in the
cap request based on its value. For now, the callers all set the sync
boolean to false everywhere to preserve the existing behavior.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: change "is_delay" parameter in check_caps to "no_delay"

Currently, when "is_delay" is true, we send the caps immediately. When
it's false, we delay them. This seems backward to me and is highly
confusing.

Sage says:

"It's definitely confusing.  the call site that motivated the naming is
 the one from tick(), that passes true ... for the delayed flush that
 forces them to be sent but lots of other callers set it to true to avoid
 delaying. It could use a rename/cleanup."

Let's change the name to "no_delay" to indicate that the caller wants
to send caps immediately instead of delaying them.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: sync metadata even when just write caps are dirty

We still need to sync out metadata on size and mtime changes, and we
definitely do not want to delay syncing just because we only have
write caps that are dirty. Remove the mask and check_caps whenever
any caps are dirty.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #11823 from xiexingguo/xxg-wip-bluestore-1108

os/bluestore: refactor _do_write(); move initializaiton of csum out of loop

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10748 from yuyuyu101/wip-dpdk8

msg/async: DPDKStack as AsyncMessenger backend

doc: add doc for pybind of rgw

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: rename rgw_file.pyx to rgw.pyx

so the init function offered by rgw python extension is named "initrgw"
instead of "initrgw_file". and "initrgw" is the expected name by python
intepreter when loading an extension.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11847 from yuyuyu101/wip-fix-rgw-spec

ceph.spec: add pybind rgwfile

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec: add pybind rgwfile

Signed-off-by: Haomai Wang <haomai@xsky.com>

os/bluestore: remove duplicated namespace of tx state

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/CMakeLists: only enable ceph_test_userspace when dpdk is true

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/dpdk: change error to warning

Signed-off-by: Haomai Wang <haomai@xsky.com>

update dpdk submodule to ceph/dpdk

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Stack: allow to setup dpdkstack

Signed-off-by: Haomai Wang <haomai@xsky.com>

test_async_networkstack: fix unmatched create/delete file event

Signed-off-by: Haomai Wang <haomai@xsky.com>

test/test_async_networkstack: add dpdk backend test

Signed-off-by: Haomai Wang <haomai@xsky.com>

test: add userspace_event

Signed-off-by: Haomai Wang <haomai@xsky.com>

Event: don't wakeup dpdk driver since it's spinning on cpu

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: add DPDKDriver type to allow userspace event

Like epoll, kqueue, select, dpdk backend also need to allow notify EventCenter
when a connection is ready to read/write. So we need to let DPDKStack can kick
EventCenter. DPDKDriver which introduced UserspaceEvent to simulate epoll
event achieve the goal.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: add dpdk stack plugin

Signed-off-by: Haomai Wang <haomai@xsky.com>

buffer: add new ptr type which destruct with passed deleter

deleter is a wrapper for lambda which can used to free resource associated
with object

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: add Poller to allow event thread spin on Poller::poll

Signed-off-by: Haomai Wang <haomai@xsky.com>

Event: Add get_driver to return driver backend

Signed-off-by: Haomai Wang <haomai@xsky.com>

Throttler: Add reset support

Signed-off-by: Haomai Wang <haomai@xsky.com>

os/bluestore: move selectiong of csum/crr out of for loop

We don't have to trace the csum/compression_required_ration option changing
at such a small granularity, which costs too much.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: refactor _do_write()

By moving garbage collection logic into a separate method.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11681 from stiopaa1/mds_mdstableclient_addConsttoMemFun

mds/MDSTableClient.h: add const to member funct

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11818 from liewegas/wip-bluestore-aio-stalls3

os/bluestore: optional debug mode to identify aio stalls

Merge pull request #11826 from ovh/wip-doc-update-xattr-config-documentation

doc: Update filestore xattr config documentation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11831 from ceph/wip-dang-rgw-pybind

RGW Python bindings - use explicit array

RGW Python bindings - use explicit array

Older Cython doesn't to implicit simple arrays, so use an explicit array
for stat output.

Revert change removing stat

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #11816 from cbodley/wip-17812

rgw: delete entries_index in RGWFetchAllMetaCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: reap ioc when stopping aio_thread.

there is possibillity that reap_ioc don't excute when stopping
aio_thread. so add reap_ioc after aio_thread is stopped.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #11817 from linuxbox2/wip-boost-3

boost: embedded

Merge pull request #11825 from ceph/wip-11820

pybind: fix build failure of rgwfile binding

Reviewed-by: Haomai Wang <haomai@xsky.com>