Merge pull request #12247 from YankunLi/fix_bf_append_parameter

bufferlist: cleanup semantical wrong for bufferlist::append

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12345 from rook/wip-embedded-add-rgw-mds

cmake: add RGW and MDS to libcephd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12784 from LiumxNL/fix-wrongly-delete-routed-op

mon: fix wrongly delete routed pgstats op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12651 from liewegas/wip-mempool-string

os/bluestore: put strings in mempool

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12661 from xiexingguo/xxg-wip-cleanup-bitalloc

os/bluestore: cleanup BitAllocator

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #12700 from ifed01/wip-bluestore-collection-everywhere

os/bluestore: various onode changes to reduce its in-memory footprint.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12876 from xiexingguo/xxg-wip-be-for-cache

os/bluestore: use Best-Effort policy when evicting onode from cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12877 from xiexingguo/xxg-wip-compress-opt

os/bluestore: conditionally load crr option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12900 from XinzeChi/wip-bluestore-valid

os/bluestore: nullptr in OmapIteratorImpl::valid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12844 from athanatos/wip-17831

osd/ReplicatedBackend: take read locks for clone sources during recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #12924 from dillaman/wip-librbd-test-header

test: correct language mode in file headers

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12892 from ceph/wip-cd-fs-fix

qa/tasks/ceph-deploy: use the new create option during instantiation

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #12884 from cbodley/wip-18488

rgw: fix off-by-one in RGWDataChangesLog::get_info

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: put Onode::key in mempool

This is a pain in the butt because std::string and
std::basic_string<...,custom allocator> are incompatible.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid substr

This creates an intermediate string, which is a waste.

Signed-off-by: Sage Weil <sage@redhat.com>

kv: add char* versions of get, set, rmkey

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore,kstore: combine kv helpers in header

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use mempool string for onode_t attr map

Signed-off-by: Sage Weil <sage@redhat.com>

include/denc: parameterize string by allocator

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: Add string to mempool data types.

Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #12233 from kylinstorage/optimize_request_auth_log

osd/pg: bound the portion of the log we request in GetLog::GetLog()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12919 from tchaikov/wip-qa-cleanup

tests: remove temporary file

Reviewed-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #12615 from linuxbox2/wip-sanitizers

build/cmake:  provide asan, tsan, ubsan builds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #12920 from Liuchang0812/update-document-for-logging-level

doc: explain about logging levels

Reviewed-by: Sage Weil <sage@redhat.com>

test: correct language mode in file headers

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12887 from jronak/documentation

Doc:Fixes Python Swift client commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11879 from trociny/wip-15371

rbd-mirror: delayed replication support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: explain about logging levels

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12870 from xiaoxichen/mds_readdir

mds/server: skip unwanted dn in handle_client_readdir

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12831 from jcsp/wip-libcephfs-refcount

libcephfs: fix cct refcount constructing from rados

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Doc: Fixes Python Swift client commands

Fixes: http://tracker.ceph.com/issues/17746
Signed-off-by: Ronak Jain <ronakjain@outlook.in>

Merge pull request #12859 from ukernel/wip-18460

client: fix Client::handle_cap_flushsnap_ack() crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12852 from ukernel/wip-18461

mds: finish clientreplay requests before requesting active state

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12833 from ukernel/wip-18396

tasks/cephfs: fix kernel force umount

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12808 from ukernel/wip-18306

mds: fix null pointer dereference in Locker::handle_client_caps

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12749 from ukernel/wip-18179

mds: propagate error encountered during opening inode by number

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12501 from ganeshmaharaj/fix-osd-tracing

tracing: Fix error in including all files in osd_tp

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: remove temporary file

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12848 from dzafman/wip-18469

ceph-objectstore-tool: Handle object names that are also valid json

Reviewed-by: Dan MIck <dmick@redhat.com>

Merge pull request #12873 from Liuchang0812/wip-static-check

test: clean up unused variable

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12913 from zmc/wip-smoke-openstack

Add openstack requirements to smoke suite

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #12914 from tmuthamizhan/wip-ansible-fs-suite

qa: specify client for fs workunit

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa: specify client for fs workunit
s/all/client.0

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>

Add openstack requirements to smoke suite

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #12869 from athanatos/wip-18358

osdc/Objecter: resend pg commands on interval change

Reviewed-by: Sage Weil <sage@redhat.com>

Objecter: resend pg commands on interval change

mark_lost_unfound* are now async since the rework, so we need
the Objecter to be able to resend on interval change.  This
is preferable to somehow requeueing the Command because they
don't use the normal op queue.

Fixes: http://tracker.ceph.com/issues/18358
Signed-off-by: Samuel Just <sjust@redhat.com>

Objecter: fold last_force_resend into op_target_t

It's always passed along with op_target_t, this seems simpler.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: take read locks for clone sources during recovery

Otherwise, we run the risk of a clone source which hasn't actually
come into existence yet being used if we grab a clone which *just*
got added the the ssc, but has not yet actually had time to be
created (can't rely on message ordering here since recovery messages
don't necessarily order with client IO!).

Fixes: http://tracker.ceph.com/issues/17831
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12888 from athanatos/wip-18165

osd/PrimaryLogPG::failed_push: update missing as well

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12894 from dmick/wip-doc

doc/cephfs/troubleshooting: fix broken bullet list

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12905 from rjfd/wip-15824

librbd: added rbd_flatten_with_progress to API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: delayed replication support

Fixes: http://tracker.ceph.com/issues/15371
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test/rbd_mirror: ImageReplayer mock tests

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: tweaks to support creating mock test cases

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: add "resync requested" unit test for rbd-mirror bootstrap request

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: allow to tweak rbd-mirror test setup

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12843 from trociny/wip-18465

librbd: metadata_set API operation should not change global config setting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12880 from wjwithagen/wip-wjw-static_cast-NULL

tests: fix NULL references to be acceptable by Clang

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: added rbd_flatten_with_progress to librbd header

Fixes: http://tracker.ceph.com/issues/15824
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #12683 from SUSE/wip-run-make-check-deps

build/ops: add sanity checks to run-make-check.sh

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: nullptr in OmapIteratorImpl::valid

call stack:
SnapMapper::get_next_object_to_trim -> MapCacher::get_next ->
OSDriver::get_next-> BlueStore::OmapIteratorImpl::valid()

In OSDriver::get_next, it call iter->upper_bound(key); // this may reset it to nullptr

Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #12864 from gaurav36/bug-18422-rbd-bench-write-crash-4G-io-size

rbd: bench-write should return error if io-size >= 4G

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks: add test_open_ino_errors

Validate that errors encountered during opening inos are properly
propagated

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12863 from dillaman/wip-rbd-coverity

librbd: resolve static analyser warnings

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

doc/cephfs/troubleshooting: fix broken bullet list

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #12820 from trociny/wip-18447

rbd-mirror: check remote image mirroring state when bootstrapping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: use Best-Effort policy when evicting onode from cache

We want a precise control of cache usage, and BE can achieve this goal
in a better way.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12465 from jcsp/wip-18159

mount: do not print "unknown" option to kclient

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #12794 from ukernel/wip-multimds-misc2

mds: misc multimds fixes part2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12842 from trociny/wip-break_lock_test

test/librbd: add break_lock test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: make fcgi optional during build

fcgi can now be compiled out of the binary. this reduces the system
dependencies when fcgi is not needed or used.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: add RGW to libcephd

added RGW and rgw-admin to libcephd.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: add MDS to libcephd

Added MDS to libcephd

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

tests: fix NULL references to be acceptable by Clang

 - On some platforms NULL evaluates to nullptr, which will require a
   cast to the right type to be able to compile.
      error: reinterpret_cast from 'nullptr_t' to 'ContextWQ *'

 - It is a delicate change since otherwise GCC will start complaining
   about the reverse:
      error: invalid static_cast from type 'const long int' to type 'ContextWQ*'

 By casting right at the instance of NULL both compilers are happy.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12839 from dillaman/wip-17993

librbd: delay mirror registration when creating clones

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12838 from dillaman/wip-18419

librbd: possible deadlock with flush if refresh in-progress

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12837 from dillaman/wip-18441

rbd-mirror: avoid processing new events after stop requested

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12846 from dillaman/wip-rbd-mirror-ha-merge

librbd: refactor exclusive lock support into generic managed lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

rgw: fix off-by-one in RGWDataChangesLog::get_info

Fixes: http://tracker.ceph.com/issues/18488
Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: pass Collection* to access BlueStore instance and other needed stuff rather than have additional referencing members in BlueStore child classes.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: refactor to avoid both SharedBlobSet and Cache pointers tracking in SharedBlob & BufferSpace respectively. Use Collection* in SharedBlob instead.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: make BufferSpace offsets 32-bit wide.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12832 from wjin/clean_async

msg/async: cleanups

Reviewed-by: Haomai Wang <haomai@xsky.com>

mds: fix MDCache::trim() with max == 0

MDCache::shutdown_pass call MDCache::trim with max == 0. If the last
dentry is non-null, MDCache::trim does nothing

Signed-off-by: Yan, Zheng <zyan@redhat.com>

os/bluestore: conditionally load crr option

And moving COMP_NONE checking ahead, since this is the preferred option.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: propagate error encountered during opening inode by number

Fixes: http://tracker.ceph.com/issues/18179
Signed-off-by: Yan, Zheng <zyan@redhat.com>

test: clean up unused variable

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12840 from tchaikov/wip-libceph-common

cmake: turn libcommon into a shared library

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #12809 from wjin/optimize_atomic

msg/async: avoid atomic variable overhead

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12826 from LiumxNL/fix-op-tracker-initiate

mon: optracker's initiated_at timestamp should not be NULL

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12871 from xiexingguo/xxg-wip-bluestore-20170111

os/bluestore: kill BufferSpace.empty()

Reviewed-by: Sage Weil <sage@redhat.com>

mds/server: skip unwanted dn in handle_client_readdir

We can skip unwanted dn which  < (offset_key, snap) via map.lower_bound, rather than
iterate across them.

Previously we iterate and skip dn which < (offset_key, dn->last), as dn->last >= snap
 means (offset_key, dn->last) >= (offset_key, snap), and such iterate_and_skip logic
still keep, so this commit doesnt change code logic but an optimization.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

os/bluestore: kill BufferSpace.empty()

Because:
1. All other public methods are called under the protection of cache->lock,
which makes the usage of this method is weird.

2. The only caller is _dump_extent_map(), and we can do it without this
in a simpler way.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12860 from liupan1111/wip-fix-dpdk-prefix

async: fixed the error "Cause: Cannot create lock on '/var/run/.rte_c…

Reviewed-by: Haomai Wang <haomai@xsky.com>

use the create option during instantiation

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #12867 from ceph/wip-ceph-deploy-workaround

qa/tasks/ceph-deploy: create-keys explicitly

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #12626 from adamemerson/wip-denconstruction

include/denc: improvements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12862 from optimistyzy/110_1

os/bluestore/NVMEdevice: fix the unrelease segs issue

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12091 from wjwithagen/wip-wjw-nosetests

nosetests: use /usr/bin/env to find nosetests

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

denc: Support boost::optional

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>