os/bluestore/BlueFS: dump allocator freelist on failure in reclaim_blocks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: handle failure to reclaim blocks without crashing

We shouldn't fail to reclaim space in general, but if we do, do not treat
it as a fatal error.  Log loudly.

Signed-off-by: Sage Weil <sage@redhat.com>

unitest_bit_alloc, unittest_alloc: fixes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prealloc/reserve extent vector for common alloc path

No need to worry about the gift/reclaim path--those are very rare.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/Allocator: drop unused and goofy release_extents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop useless count arg to allocate

The vector<> has a size.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: return blocks allocated from allocate()

Instead of having a separate output argument with the number of
blocks allocated, just return it via the return value.  Simplifies
the calling convention.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: manage vector from ExtentList

ExtentList was previous relying the caller to preallocate/size the
vector to be large enough for the worst case allocation of extents,
and keeping it's own manual count of the extent list size.  Instead,
manage that from ExtentList, and remove the preallocation from the
callers.

Fixes: http://tracker.ceph.com/issues/18573
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12681 from xiexingguo/xxg-wip-bluestore-1227

os/bluestore: generate same onode extent-shard keys in a more efficient way

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12696 from xiexingguo/xxg-wip-improve-loops

os/bluestore: miscellaneous fixes to BitAllocator

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #12719 from xiexingguo/xxg-wip-bluestore-1229

os/bluestore/BitmapFreelistManager: readability improvements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12942 from Liuchang0812/cleanup-bluestore

os/bluestore: avoid unnecessary memory copy, use variable reference in BlockDevice::Open

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12946 from ifed01/wip-bluestore-pextent-vector-mempooled

os/bluestore: Add bluestore pextent vector to mempool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12964 from jcsp/wip-18574

qa: update remaining ceph.com to download.ceph.com

Merge pull request #12962 from dillaman/wip-rbd-boost-crash

rbd: remove direct linking to static boost libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: update remaining ceph.com to download.ceph.com

Fixes: http://tracker.ceph.com/issues/18574
Signed-off-by: John Spray <john.spray@redhat.com>

rbd: remove direct linking to static boost libraries

The boost libraries are included in libceph-common. If linked
directly, boost::regex will abort during shutdown since it uses
a global static variable and it will be destructed twice.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12885 from rjfd/wip-18325

librbd: allow to open an image without opening the parent image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12922 from trociny/wip-managed_lock_refactoring

librbd: managed lock refactoring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12953 from dillaman/wip-18542

qa/tasks/qemu: update default image url after ceph.com redesign

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12954 from dillaman/wip-rbd-coverity

rbd: correct coverity warnings

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12467 from bassam/pr-external-rocksdb

cmake: support for external rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: correct coverity warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9028 from tchaikov/wip-bt-dump

common/BackTrace: add operator<<

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/qemu: update default image url after ceph.com redesign

Fixes: http://tracker.ceph.com/issues/18542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12807 from runsisi/wip-fix-pg-ls

mon/PGMonitor: fix description for ceph pg ls

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/PGMonitor: ceph pg ls-by-pool only supports string type of pool arg

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #12928 from tmuthamizhan/wip-ansible-fs-suite

qa/suites/ceph-ansible: removing fs workunit

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

librbd: helper methods to query and break lock

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: managed_lock: make AcquireRequest use GetLockRequest and BreakRequest

Initially this was implemented for exclusive_lock (03533b9,
23f60fe) but was lost when merging managed_lock refactoring.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12915 from jcsp/wip-18361

client: populate metadata during mount

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12107 from mslovy/wip-monmsg-minwait

mon: no delay for single message MSG_ALIVE and MSG_PGTEMP

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11955 from batrick/log-2-to-1-write

log: use one write system call per message

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9924 from Yan-waller/yj-wiph-commonentitynamecc

common/config: fix return type of string::find and use string::npos

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12925 from liewegas/wip-crypto-shutdown-fixes

common: misc fixes detected by crypto shutdown assert

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12944 from liewegas/wip-centos-latest

qa/distros: make centos_latest 7.3

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #12841 from cbodley/wip-18407

rgw multisite: fix ref counting of completions

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12605 from cbodley/wip-18300

rgw: RGWMetaSyncShardCR drops stack refs on destruction

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: allow to open an image without opening parent image

Fixes: http://tracker.ceph.com/issues/18325
Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: support for external rocksdb

add support for building with an external rocksdb.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #12798 from wjwithagen/wip-wjw-krbd-optional

rbd: enable rbd on FreeBSD (without KRBD)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: add pextent vector to mempool

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/objectstore: add bufferptr & blouestore_blob_t::extents to "sizeof" UT

Signed-off-by: Igor Fedotvo <ifedotov@mirantis.com>

Merge pull request #12930 from chendave/wip_cephx_refactor

msg: Revert the change from assert(0)-> ceph_abort() where is not applicable

Reviewed-by: Sage Weil <sage@redhat.com>

qa/distros: make centos_latest 7.3

This will fix smithi runs and break vps runs until we get a 7.3 cloud-init
image ready.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid unnecessary memory copy, use variable reference in BlockDevice::Open

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12939 from johnlinp/fix-nspr-include

cmake: search for nspr include files for both suffixes: nspr4 and nspr

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: search for nspr include files for both suffixes: nspr4 and nspr

Fixes: http://tracker.ceph.com/issues/18535
Signed-off-by: John Lin <johnlinp@gmail.com>

Merge pull request #12936 from badone/wip-pg-states-doc-fix

doc: min_size advice is not helpful

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: generate same onode extent-shard keys in a more efficient way

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: min_size advice is not helpful

When we are in the incomplete state adjusting min_size will not help with recovery.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12247 from YankunLi/fix_bf_append_parameter

bufferlist: cleanup semantical wrong for bufferlist::append

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12345 from rook/wip-embedded-add-rgw-mds

cmake: add RGW and MDS to libcephd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/BackTrace: add operator<<

replace BackTrace::print() with the operator<< where the former is used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12784 from LiumxNL/fix-wrongly-delete-routed-op

mon: fix wrongly delete routed pgstats op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12651 from liewegas/wip-mempool-string

os/bluestore: put strings in mempool

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12661 from xiexingguo/xxg-wip-cleanup-bitalloc

os/bluestore: cleanup BitAllocator

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #12700 from ifed01/wip-bluestore-collection-everywhere

os/bluestore: various onode changes to reduce its in-memory footprint.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12876 from xiexingguo/xxg-wip-be-for-cache

os/bluestore: use Best-Effort policy when evicting onode from cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12877 from xiexingguo/xxg-wip-compress-opt

os/bluestore: conditionally load crr option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12900 from XinzeChi/wip-bluestore-valid

os/bluestore: nullptr in OmapIteratorImpl::valid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12844 from athanatos/wip-17831

osd/ReplicatedBackend: take read locks for clone sources during recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert the change from assert(0)-> ceph_abort() where is not applicable

Signed-off-by: Dave Chen <wei.d.chen@intel.com>

removing fs workunit

removing fs workunit for now until tracker issue 18528 is resolved.

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

Merge pull request #12924 from dillaman/wip-librbd-test-header

test: correct language mode in file headers

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12892 from ceph/wip-cd-fs-fix

qa/tasks/ceph-deploy: use the new create option during instantiation

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: RGWCloneMetaLogCoroutine uses RGWMetadataLogInfoCompletion

Fixes: http://tracker.ceph.com/issues/18412
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: expose completion for RGWMetadataLog::get_info_async()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWMetaSyncShardCR drops stack refs on destruction

if the coroutine is canceled before collect_children() can clean up
all of its child stacks, those stack refs will leak. store these
stacks as boost::intrusive_ptr so the ref is dropped automatically on
destruction

Fixes: http://tracker.ceph.com/issues/18300
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12884 from cbodley/wip-18488

rgw: fix off-by-one in RGWDataChangesLog::get_info

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: put Onode::key in mempool

This is a pain in the butt because std::string and
std::basic_string<...,custom allocator> are incompatible.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid substr

This creates an intermediate string, which is a waste.

Signed-off-by: Sage Weil <sage@redhat.com>

kv: add char* versions of get, set, rmkey

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore,kstore: combine kv helpers in header

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use mempool string for onode_t attr map

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: parameterize string by allocator

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: Add string to mempool data types.

Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>

global/signal_handler: dump backtrace even from within logging code

We can dump the backtrace regardless of where we hit the assertion.

Signed-off-by: Sage Weil <sage@redhat.com>

global/signal_handler: behave if g_ceph_context or _log are null

This can happen if we trigger an assertion while g_ceph_context is being
torn down.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_crypto: fix leaks

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: use std::unique_ptr<>; fix leak

We were leaking several heap items from
LibRadosMiscConnectFailure.ConnectFailure when connect() was called
multiple times (after failure).  Use std::unique_ptr to avoid maintaining
fragile cleanup paths.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12233 from kylinstorage/optimize_request_auth_log

osd/pg: bound the portion of the log we request in GetLog::GetLog()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12919 from tchaikov/wip-qa-cleanup

tests: remove temporary file

Reviewed-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #12615 from linuxbox2/wip-sanitizers

build/cmake:  provide asan, tsan, ubsan builds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #12920 from Liuchang0812/update-document-for-logging-level

doc: explain about logging levels

Reviewed-by: Sage Weil <sage@redhat.com>

test: correct language mode in file headers

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12887 from jronak/documentation

Doc:Fixes Python Swift client commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11879 from trociny/wip-15371

rbd-mirror: delayed replication support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: explain about logging levels

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12870 from xiaoxichen/mds_readdir

mds/server: skip unwanted dn in handle_client_readdir

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12831 from jcsp/wip-libcephfs-refcount

libcephfs: fix cct refcount constructing from rados

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Doc: Fixes Python Swift client commands

Fixes: http://tracker.ceph.com/issues/17746
Signed-off-by: Ronak Jain <ronakjain@outlook.in>

Merge pull request #12859 from ukernel/wip-18460

client: fix Client::handle_cap_flushsnap_ack() crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12852 from ukernel/wip-18461

mds: finish clientreplay requests before requesting active state

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12833 from ukernel/wip-18396

tasks/cephfs: fix kernel force umount

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12808 from ukernel/wip-18306

mds: fix null pointer dereference in Locker::handle_client_caps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12749 from ukernel/wip-18179

mds: propagate error encountered during opening inode by number

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12501 from ganeshmaharaj/fix-osd-tracing

tracing: Fix error in including all files in osd_tp

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: remove temporary file

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BitAllocator: kill find_any_free_bits()

Which has no consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>