doc/bluestore: add SPDK usage for bluestore

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17018 from alex-mikheev/wip_rdma_rxpool_2

msg/async/rdma: refactor rx buffer pool allocator

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17504 from dillaman/wip-21251

qa/suites/rbd: miscellaneous test fixes

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17532 from dillaman/wip-21179

rbd: support iterating over metadata items when listing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17509 from dillaman/wip-21217

test/cls_rbd: mask newer feature bits to support upgrade tests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17499 from dillaman/wip-21247

cls/rbd: metadata_list not honoring max_return parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge PR #17340 into master

* refs/remotes/upstream/pull/17340/head:
mds: void sending cap import message when inode is frozen
client: fix message order check in handle_cap_export()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16778 into master

* refs/remotes/upstream/pull/16778/head:
mds: fix return value of MDCache::dump_cache
mds: new cap message flags indicate if there is pending capsnap
mds: properly do null snapflush part2
mds: track snap inodes through sorted map
mds: properly drop wrlock when finishing snapflush

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17291 into master

* refs/remotes/upstream/pull/17291/head:
mds: fix 'dirfrag end' check in Server::handle_client_readdir

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge PR #17289 into master

* refs/remotes/upstream/pull/17289/head:
osd, mds, tools: drop the invalid comment and some unused variables

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17219 into master

* refs/remotes/upstream/pull/17219/head:
mds: fix StrayManager::truncate()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17505 from liewegas/wip-20910

qa/objectstore/bluestore*: less debug output

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17459 from xiexingguo/wip-bs-tracked-key

os/bluestore: add bluestore_prefer_deferred_size_hdd/ssd to tracked keys

Reviewed-by: Pan Liu <liupan1111@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17463 from tchaikov/wip-ceph-tell-mds-star

ceph: fixes for "tell <service>.*" command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #17503 from liewegas/wip-21250

os/bluestore/BlueFS: prevent _compact_log_async reentry

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

cls/rbd: metadata_list not honoring max_return parameter

Fixes: http://tracker.ceph.com/issues/21247
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: support iterating over metadata items when listing

Fixes: http://tracker.ceph.com/issues/21179
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17510 from liewegas/wip-crush-fix-rule-lookup

crush: fix fast rule lookup when uniform

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17356 from shashalu/bucket_link/unlink_olh

rgw: don't write bucket_header when it is not changed in bucket_link/unlink

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17434 from iliul/remove-useless-output

rgw: Remove the useless output when list zones

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17518 from wjwithagen/wjw-githubmap

.githubmap: Add wjwithagen as a known Ceph reviewer

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: fix return value of MDCache::dump_cache

previous commit "mds: track snap inodes through sorted map" makes
MDCache::dump_cache return 1 on success.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

.githubmap: Add myself

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #16145 from yehudasa/wip-20234

rgw: add tail tag to track tail instance
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17502 from dillaman/wip-21248

librbd: rename of non-existent image results in seg fault

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #17375 from liupan1111/wip-final-fix-nbd

rbd-nbd: fix generic option issue

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17507 from batrick/githubmap-update

githubmap: add some known Ceph reviewers

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #17319 into master

* refs/remotes/upstream/pull/17319/head:
qa: whitelist expected rstat warning
qa: sync whitelist with fs/basic_functional
qa: whitelist expected MDS_CACHE_OVERSIZED

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17301 into master

* refs/remotes/upstream/pull/17301/head:
mds: fix "1 filesystem is have a..." message

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17263 into master

* refs/remotes/upstream/pull/17263/head:
mds: remove unused method
mds: move EMetaBlob cons to header

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17178 into master

* refs/remotes/upstream/pull/17178/head:
ceph-dencoder: simplify decoding/encoding cephfs inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17095 into master

* refs/remotes/upstream/pull/17095/head:
client: reset unmounting flag to false when starting a new mount
client: add mountedness check inside client_lock
client: rework Client::get_local_osd() return codes
client: remove misleading comment in get_cap_ref

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge PR #16562 into master

* refs/remotes/upstream/pull/16562/head:
cephfs/fuse: set big_writes default is false

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16305 into master

* refs/remotes/upstream/pull/16305/head:
qa/cephfs: test CephFS recovery pools
qa/cephfs: support CephFS recovery pools
qa/ceph_test_case: support CephFS recovery pools
qa/cephfs: Allow deferred fs creation
qa/cephfs: Refactor alternate pool test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

githubmap: add some known GitHub reviewers

Selection from [1] where the GitHub username is available.

[1] http://pad.ceph.com/p/reviewers

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17447 from tchaikov/wip-freebsd-coredump

test/coredumpctl: support freebsd

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

crush: fix fast rule lookup when uniform

Older clients will search for the first rule with a matching ruleset,
type, and size.  The has_uniform_rules bool is only set if we have rule
ids and rulesets that line up, but we must also verify that the rest of the
mask matches or else we can get a different CRUSH mapping result because
the mask might not match and old clients will fail to find a rule and we
will find one.  We also can't just check the ruleset as the legacy clients
find the *first* (of potentially many) matching rules; hence we only do
the fast check if all rulesets == rule id.

Signed-off-by: Sage Weil <sage@redhat.com>

test/cls_rbd: mask newer feature bits to support upgrade tests

Fixes: http://tracker.ceph.com/issues/21217
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/objectstore/bluestore*: less debug output

Let's see if this makes the spurious MON_DOWN failures go away?  (See
http://tracker.ceph.com/issues/20910)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rbd: reduce monitor requirements for mirroring tests

Fixes: http://tracker.ceph.com/issues/21251
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: fixed cache override

Fixes: http://tracker.ceph.com/issues/21251
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: reduce OSD debug log levels

The teuthology machines are periodically running out of space
due to the aggressive log settings.

Fixes: http://tracker.ceph.com/issues/21251
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #17373 into master

* refs/remotes/upstream/pull/17373/head:
doc/cephfs: add info on using EC pools with CephFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore/BlueFS: prevent _compact_log_async reentry

_should_compact_log uses new_log != nullptr to tell whether compaction is
already in progress, but we don't set it until we are midway through the
process.  Set it at the top of the method to prevent reentry.

See 455cc6cea2e5e8bc1126b5047b6003fa583af1b5, which failed to implement
this properly.

Fixes: http://tracker.ceph.com/issues/21250
Signed-off-by: Sage Weil <sage@redhat.com>

ceph: collect all mds in mdsids()

otherwise, only the active mds are returned.

Fixes: http://tracker.ceph.com/issues/21230
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph: always populate `targets` with ids_by_service()

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph: extract ids_by_service() so it can be reused

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: rename of non-existent image results in seg fault

Fixes: http://tracker.ceph.com/issues/21248
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17141 from theanalyst/doc-civetweb-ports

doc: rgw: mention the civetweb support for binding to multiple ports

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17436 from ashishkumsingh/wip-doc-fix-snapshot-flatten-example

doc: Fixes rbd snapshot flatten example

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17457 from batrick/githubmap

githubmap: add map for GitHub contributor lookup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17489 from mychoxin/yuanxin_deep_copy

include/buffer.h: fix typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

include/buffer.h: fix comment 'no not allow' -> 'do not allow'

Signed-off-by: mychoxin <mychoxin@gmail.com>

Merge pull request #17430 from Songweibin/wip-osd-misc

osd: misc cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16476 from renhwztetecs/renhw-wip-mgr-standby-ip

mgr: add the ip addr of standbys

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17431 from tchaikov/wip-da-SCA-20170510

Coverity and SCA fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: add bluestore_prefer_deferred_size_hdd/ssd to tracked keys

To make these two keys can be changed online:

./bin/ceph tell osd.* injectargs "--bluestore_prefer_deferred_size_hdd 0"
osd.0: bluestore_prefer_deferred_size_hdd = '0' (not observed, change may require restart)
osd.1: bluestore_prefer_deferred_size_hdd = '0' (not observed, change may require restart)
osd.2: bluestore_prefer_deferred_size_hdd = '0' (not observed, change may require restart)

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/coredumpctl: support freebsd

setrlimit() on platforms without prctl()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17437 from tchaikov/wip-refactor-build_initial_pg_history

osd: update info only if new_interval

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17393 from tchaikov/wip-cmake-disable-vta

cmake: disable VTA on options.cc

Reviewed-by: Sage Weil <sage@redhat.com>

githubmap: add map for GitHub contributor lookup

Idea of this is to allow scripts to lookup the contributor name/email by GitHub
username. This is useful in particular for adding appropriate "Reviewed-by"s
for each GitHub style "review".

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rbd-nbd: fix generic option issue

Fixes: http://tracker.ceph.com/issues/20426
Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>

Merge pull request #15705 from xiexingguo/wip-fix-blob-tracker-check

os/bluestore: pass strict flag to bluestore_blob_use_tracker_t::equal()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

doc: Fixes rbd snapshot flatten example

Snapshot flatten example has incorrect image name, fixing the same.

Fixes: #17723
Signed-off-by: Ashish Singh <assingh@redhat.com>

Merge pull request #15363 from ShiqiCooperation/master

test/unittest_bluefs: check whether rmdir success

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: remove the useless output when list zones

Signed-off-by: iliul <liul124@chinaunicom.cn>

osd: update info only if new_interval

refactor OSD::build_initial_pg_history() so we update the info only if a
new interval is identified. also, this is consistent with
OSD::build_past_intervals_parallel().

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17435 from liupan1111/wip-fix-spdk-perf

bluestore/NVMEDevice: accurate the latency perf counter of queue latency

Reviewed-by: Haomai Wang <haomai@xsky.com>

bluestore/NVMEDevice: accurate the latency of queue latency.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

bluestore/NVMEDevice: remove mm_pause according to the example givn by
spdk.

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

mgr: add the ip addr of standbys

we need to manage the ip addr of the "standbys" state,
because the hostname/gid is insufficient to locate the
Standby node. we add ip of the mgr standby to metadata.

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

bluestore/NVMEDevice: rename the perf counter *aio_write* to *write*,
because it will also statistic the latency of sync write

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

bluestore/NVMEDevice.cc: rename the thread from ceph-osd to
nvme-device-manager, to avoid name duplication with main thread
"ceph-osd"

Signed-off-by: Pan Liu <wanjun.lp@alibaba-inc.com>
Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

Merge pull request #17409 from liewegas/wip-21207

os/bluestore: separate finisher for deferred_try_submit

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #17395 from theanalyst/cli-rados-stat2

cli: rados: support for high precision time using stat2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17100 from mychoxin/yuanxin_vstart_default_bluestore

vstart.sh: use bluestore as default osd objectstore backend

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #16855 from amitkumar50/rgw_module

rgw: Initializes uninitialized members

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17129 from C2python/clear_para

rgw: Remove unused Parameter in Function RGWConf::init()

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15341 from chardan/jfw-wip-rand_makes_puppies_cry

common: implement random number generator (following N3551)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #15565 from gmayyyha/ceph-osd-help-output

ceph-osd: some flags are not documented in the help output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16942 from jcsp/wip-20888

mon: rate limit on health check update logging

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17404 from rzarzynski/wip-bs-ldout-in-stupid

os/bluestore: replace dout with ldout in StupidAllocator.

Merge pull request #17406 from liewegas/wip-21206

qa/tasks/ceph_manager: revive osds before doing final rerr reset

Merge pull request #17416 from xiexingguo/wip-handle-small-block

os/bluestore: handle small main device properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17423 from liewegas/wip-21203

osd: fix build_initial_pg_history

Reviewed-by: Sage Weil <sage@redhat.com>

test/librados/aio.cc: silence coverity with scope_guard

Fix for:

CID 1313405 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable my_completion2 going out of scope
 leaks the storage it points to.

CID 1219522 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable my_completion going out of scope
 leaks the storage it points to.

CID 1219501 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable my_completion going out of scope
 leaks the storage it points to.

CID 1313406 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable my_completion2 going out of scope
 leaks the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

c_read_operations.cc: silence coverity with scope_guard

CID 1219593 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable completion going out of scope leaks
 the storage it points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

LibradosTestStub.cc: memset struct sockaddr_in

Fix for:

CID 1402624 (#1 of 1): Uninitialized scalar variable (UNINIT)
 uninit_use: Using uninitialized value sin. Field sin.sin_zero is
 uninitialized.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

common/Timer.h: ~SafeTimer needs to be virtual

Fix for:

CID 1396232 (#1 of 1): Non-virtual destructor (VIRTUAL_DTOR)
 nonvirtual_dtor: Class librbd::<unnamed>::SafeTimerSingleton has a
 destructor and a pointer to it is upcast to class SafeTimer which
 doesn't have a virtual destructor.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test_ipaddr.cc: memset with 0 and not '0'

Fix for:

CID 1405070 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405071 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405073 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405074 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405075 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405077 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405083 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405086 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
CID 1405087 (#1 of 1): Memset fill value of '0' (NO_EFFECT)
 bad_memset: "memset" with fill value "'0'" (the zero character).
 memset(&net, 48, 28UL). (CWE-665)

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/MetaRequest.h: fix UNINIT_CTOR

Fix for:

CID 717207 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
 uninit_member: Non-static class member dirp is not initialized
 in this constructor nor in any functions that it calls.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Client.cc: fix UNINIT_CTOR

Fix for:

CID 1406088 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
 uninit_member: Non-static class member root_ancestor is not
 initialized in this constructor nor in any functions that it calls.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

crush/CrushCompiler.cc: fix error handling

calloc() does not take negative values, check return value from
get_max_buckets() and handle it correctly.

Fix for:

CID 1405301 (#1 of 1): Argument cannot be negative (NEGATIVE_RETURNS)
 negative_returns: arg_map.size is passed to a parameter that cannot
 be negative.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

client/Client.cc: fix USE_AFTER_FREE

Don't call put_request() twice, it's already called by
unregister_request()

Fix for:

CID 1405360 (#1 of 1): Use after free (USE_AFTER_FREE)
 deref_arg: Calling put_request dereferences freed pointer request

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tools/rbd/Utils.cc: yank features_set_specified and related logic

since it isn't used

Fix for:

CID 1394854 (#1 of 1): 'Constant' variable guards dead code (DEADCODE)
 dead_error_line: Execution cannot reach this statement:
 opts->set(RBD_IMAGE_OPTION_....
 Local variable features_set_specified is assigned only once, to a
 constant value, making it effectively constant throughout its scope.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #17304 from kungf/calc_replicated_acting

osd/PG: remove unused parameter in calc_ec_acting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17088 from zmedico/interval_set_intersection_of_optimization

interval_set: optimize intersection_of

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15816 from chardan/jfw-wip-consolidate-spinlocks

common: consolidate spinlocks

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17265 from zmedico/interval-set-intersection-insert-optimization

interval_set: optimize intersect_of insert operations

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #17391 from myoungwon/wip-proxywrite-cleanup

osd: remove unused variable in do_proxy_write

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>