BlueStore: speedup the performance of multi-replication flow by
switching the callback order in bluestore.

In Bluestore ack callback and commit callback are queued one by one in
the function "BlueStore;:_txc_finish_kv". Therefore, just only one callback
is needed to call, in order to improve performance. We do this by switching
the callback order in bluestore, and the callback work can be done in
sub_op_modify_commit and doesn't need to do it again in sub_op_modify_applied.

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #11681 from stiopaa1/mds_mdstableclient_addConsttoMemFun

mds/MDSTableClient.h: add const to member funct

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11818 from liewegas/wip-bluestore-aio-stalls3

os/bluestore: optional debug mode to identify aio stalls

Merge pull request #11826 from ovh/wip-doc-update-xattr-config-documentation

doc: Update filestore xattr config documentation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11831 from ceph/wip-dang-rgw-pybind

RGW Python bindings - use explicit array

RGW Python bindings - use explicit array

Older Cython doesn't to implicit simple arrays, so use an explicit array
for stat output.

Revert change removing stat

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #11816 from cbodley/wip-17812

rgw: delete entries_index in RGWFetchAllMetaCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11817 from linuxbox2/wip-boost-3

boost: embedded

Merge pull request #11825 from ceph/wip-11820

pybind: fix build failure of rgwfile binding

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: Update filestore xattr config documentation.

Signed-off-by: Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

pybind: fix build failure of rgwfile binding

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9624 from gaurav36/coverity-1355603-dead-code

coverity fix: fixing few coverity issue

Merge pull request #11798 from stiopaa1/mds_fsmap_preventcopyinginloop

mds/FSMap.cc: prevent unneeded copy of map entry

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11802 from stiopaa1/mds_fsmap_constReference

mds/FSMap.h: add const and reference

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11819 from stiopaa1/mds_migrator_addConstTomemFun

mds/Migrator.h: add const to member functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11807 from xiexingguo/xxg-wip-bs-1107

os/bluestore: speed up omap-key generation for same onode

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: speed up omap-key generation for same onode

This is useful if we have to batch-process omap kvs.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11754 from athanatos/wip-17789

osd/ReplicatedPG: do_update_log_missing: take the pg lock in the callback

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/KernelDevice: debug support to detect stalled aios

This is additional bookkeeping to track the oldest aio in flight, so we
make it optional, and disabled by default.  It is useful for identifying
buggy kernel aio or bad devices.

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: add aio_list_t type

Signed-off-by: Sage Weil <sage@redhat.com>

os/fs: uninline aio methods

Signed-off-by: Sage Weil <sage@redhat.com>

mds/Migrator.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

boost:  pull up boost to enable boost_python build

Pullup to just the merges on Sep 10 fixes an obvious compile
problem, but there are other relevant changes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Adapt for ceph-manager

The ceph manager change added a separate Boost dependency
check, add it to the combined section.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

do_cmake.sh: set BOOST_J to $(nproc)

This lets us build boost with all processors.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

boost:  update boost to 8/29

Pull-up Boost to the commit which merged the upstream fix for
upstream Boost/iostreams bug #11860 (template arity warnings),
per discussion in core.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cmake:  find and build bundled boost

Build Boost using the src/boost submodule, unless overridden
by -DWITH_SYSTEM_BOOST.

If -DBOOST_J=<n> is provided, builds Boost with <n> jobs.

Boost builds in the configured Build directory.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

boost: add submodule at tag boost-1.61.0

This implements option #4 for external boost, based on upstream
discussion.  In option #4:

1. boost is added as a submodule
2. builds default to using the attached boost module
3. building against a system-provided boost is supported, but
   must be configured explicitly

Because all of the boost components are attached as nested
submodules in the upstream boost repository, neither the nested
submodules nor the root boost submodule have been cloned into
modules in github.com/ceph (acked by Sage).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #11647 from ceph/wip-jlayton-cephfs

libcephfs client API overhaul and update

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11425 from weiqiaomiao/wqm-rgw-oplog

rgw: fix the field 'total_time'  of log entry in log show opt

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8712 from diwakargoel/master

doc: Correcting the sample python tempurl generation script.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11648 from cbodley/wip-17698

rgw: fix for bucket delete racing with mdlog sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11598 from cbodley/wip-rgw-bucket-sync

rgw multisite: move lease up to RunBucketSync instead of child crs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: delete entries_index in RGWFetchAllMetaCR

use a unique_ptr to ensure the entries_index is freed on destruction

Fixes: http://tracker.ceph.com/issues/17812
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11715 from dzafman/wip-test-ec-eio

Fix racey test by setting noout flag (tracker 17757)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11624 from yuyuyu101/wip-rgwfile-pybind

Wip rgwfile pybind

pybind/rgwfile: change description

Signed-off-by: Haomai Wang <haomai@xsky.com>

pybind/rgwfile: fixes hardcode way to find include dir

Signed-off-by: Haomai Wang <haomai@xsky.com>

pybind/rgwfile: raise Exception when failing to umount

Signed-off-by: Haomai Wang <haomai@xsky.com>

pybind/rgwfile: change FileHandler to FileHandle

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11794 from rzarzynski/wip-rgw-17803

rgw: fix uncompressed object size deduction in RGWRados::copy_obj_data.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11797 from yuyuyu101/wip-bluestore-pending

os/bluestore: reap collection after all pending ios done

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11713 from stiopaa1/mds_mantle_passByConstRef

mds/Mantle: pass parameters by const ref

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11727 from stiopaa1/mds_mdsrank_addOverideToThreadClass

mds/MDSRank.h: add override to virtual function

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11781 from trociny/wip-qa-rbd-nbd-sudo

qa/workunits/rbd: simplify running nbd test under build env

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11806 from david-z/wip-remove-dup-log

mds: remove duplicated log in handle_client_readdir

mds: remove duplicated log in handle_client_readdir

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

mds/FSMap.h: add const and reference

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11343 from ceph/wip-civetweb-1.8

Update version of civetweb to 1.8

mds/FSMap.cc: prevent unneeded copy of map entry

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: reap collection after all pending ios done

Otherwise when umount finish reap and wait for pending io done, the last
pending io may aim to remove collection and make removed_collection not
empty. So the leaked collection ref will result in BlueStore deconstruction
segment failt

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11251 from stiopaa1/osd_replicatedpg_removeCount2

osd/ReplicatedPG: remove unneeded use of count

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11510 from vshankar/wip-17356

librbd: batch object map updates during trim

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11766 from trociny/wip-rbd-info-fix-upgrade-client

librbd: snap_get_limit compatibility check

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10907 from jecluis/wip-mon-features

mon: have mon-specific features & rework internal monmap structures

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11772 from liewegas/wip-sortbitwise

osd,mon: require sortbitwise flag to upgrade beyond jewel

Merge pull request #11791 from liewegas/wip-remove-move-ranges

os: remove move_ranges_destroy_src

rgw: fix uncompressed object size deduction in RGWRados::copy_obj_data.

Fixes: http://tracker.ceph.com/issues/17803
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #10447 from stiopaa1/msg_pipe_moveDelayedDelivery2

msg/Pipe: move DelayedDelivery class to cc file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11780 from xiexingguo/xxg-wip-bluestore-1104

os/bluestore: a few more cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11498 from liewegas/wip-stop-at-jewel

mon: prevent post-jewel OSDs from booting if require_jewel_osds is not set

os/bluestore: fix output message when passing the end of list range

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix typo in comments

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check against collection existence

To keep pace with other methods.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: narrow boundary check for fiemap

By including off-by-one.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test-erasure-eio: Avoid racing recovery by setting noout

Fixes: http://tracker.ceph.com/issues/17757
Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "tests: disable failing test-erasure-eio.sh"

This reverts commit f018c9ec08242c6d32e041b868be14e0a4d3d34c.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11527 from wjwithagen/wip-wjw-readable.sh

test/encoding/readable.sh: fix shell script warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11775 from jdurgin/wip-jewel-osds-warning

mon/OSDMonitor: remove duplicate jewel/kraken flag warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9345 from SUSE/wip_remove_mon_startup_magic

mon: remove ceph-create-keys from mon startup

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

src/init-ceph.in:Remove ceph-create-keys for sysv

ceph-create-keys should not be started on boot of mons with sysv init.

Signed-off-by: Owen Synge <osynge@suse.com>

systemd/CMakeLists.txt:Remove ceph-create-keys cmake

ceph-create-keys should not be started on boot of mons with systemd so should
not exist as 'After' or 'Wants' for the ceph-mon.service

Signed-off-by: Owen Synge <osynge@suse.com>

systemd/ceph-mon@.service:Remove ceph-create-keys for mon in systemd

ceph-create-keys should not be started on boot of mons with systemd so should
not exist as 'After' or 'Wants' for the ceph-mon.service

Signed-off-by: Owen Synge <osynge@suse.com>

systemd/ceph-create-keys@.service:Remove ceph-create-keys for systemd

ceph-create-keys should not be started on boot of mons with systemd so should
not exist in the systemd files

Signed-off-by: Owen Synge <osynge@suse.com>

src/upstart/ceph-create-keys.conf:Remove ceph-create-keys for upstart

ceph-create-keys should not be started on boot of mons with upstart so should
not exist in the upstart files

Signed-off-by: Owen Synge <osynge@suse.com>

debian/rules:Remove ceph-create-keys for deb packaging

ceph-create-keys should not be started on boot of mons with upstart so should
not exist in the deb.

Signed-off-by: Owen Synge <osynge@suse.com>

ceph.spec.in:Remove ceph-create-keys for systemd

ceph-create-keys should not be started on boot of mons with systemd so should
not exist in the rpm.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #11749 from ceph/wip-17753

ceph-create-keys should not try forever to do things

Merge pull request #11788 from liewegas/wip-bluestore-blackhole

os: make filestore_blackhole -> objectstore_blackhole

os: remove move_ranges_destroy_src

We are going to take a different path (rollback support in ObjectStore
instead of roll-forward via splicing data from a temp object).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11747 from jecluis/wip-17725

mon: MonmapMonitor: return success when monitor will be removed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9768 from cxwshawn/wip-tc1-fix

log: optimize header file dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9288 from yuyuyu101/wip-add-shutdown-log

osd: print log when osd want to kill self

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11770 from liewegas/wip-kill-tbl

os/ObjectStore: remove legacy tbl support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11785 from Liuchang0812/patch-1

doc: Update crush-map.rst, fix a typo mistake

os/bluestore: implement blackhole

Signed-off-by: Sage Weil <sage@redhat.com>

os: add objectstore_blackhole option; deprecate filestore_blackhole

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11531 from yuyuyu101/wip-rdma

msg/async: ibverbs/rdma support

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mellanox <adirl@mellanox.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11637 from liupan1111/master

common: make l_finisher_complete_lat more accurate

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

qa/workunits/rbd: simplify running nbd test under build env

Previously running the script as unprivileged user was not very useful
due to difficulty to change path sudo was looking for a command to
execute.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Update crush-map.rst

fix typo

Merge pull request #11771 from liewegas/wip-bluestore-collection-end

os/bluestore: fix collection_list end bound off-by-one

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11784 from jcsp/wip-jcsp-docs

Documentation syntax cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephfs: separate front page section for dev stuff

The "additional details" section was getting longer
and longer, and I don't want developer-only stuff
in there anyway.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: fix rst syntax in plugin guide

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: fix rst syntax in mantle.rst

Signed-off-by: John Spray <john.spray@redhat.com>

cephfs/capabilities: fix rst syntax

Although we don't link this into built docs,
it generates warnings during build.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: s/CephFS Quotas/Quotas/

This looks odd in the TOC (we're already
in the cephfs section)

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: fix header depths

Some files had multiple top-level headers resulting in
subsections being listed in the TOC alongside the
main page header.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: link in mantle doc

...and add a warning at the very top of the document
that it is not for ordinary user use.

Considered just not linking this in at all, but it
seems a shame to have it here but not be able to
e.g. google it and link to it.

Signed-off-by: John Spray <john.spray@redhat.com>