os/bluestore: implement blackhole

Signed-off-by: Sage Weil <sage@redhat.com>

os: add objectstore_blackhole option; deprecate filestore_blackhole

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11531 from yuyuyu101/wip-rdma

msg/async: ibverbs/rdma support

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mellanox <adirl@mellanox.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11637 from liupan1111/master

common: make l_finisher_complete_lat more accurate

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11771 from liewegas/wip-bluestore-collection-end

os/bluestore: fix collection_list end bound off-by-one

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11784 from jcsp/wip-jcsp-docs

Documentation syntax cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephfs: separate front page section for dev stuff

The "additional details" section was getting longer
and longer, and I don't want developer-only stuff
in there anyway.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: fix rst syntax in plugin guide

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: fix rst syntax in mantle.rst

Signed-off-by: John Spray <john.spray@redhat.com>

cephfs/capabilities: fix rst syntax

Although we don't link this into built docs,
it generates warnings during build.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: s/CephFS Quotas/Quotas/

This looks odd in the TOC (we're already
in the cephfs section)

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: fix header depths

Some files had multiple top-level headers resulting in
subsections being listed in the TOC alongside the
main page header.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: link in mantle doc

...and add a warning at the very top of the document
that it is not for ordinary user use.

Considered just not linking this in at all, but it
seems a shame to have it here but not be able to
e.g. google it and link to it.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11741 from trociny/wip-17219

rbd-nbd: disallow mapping images >2TB in size

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11744 from trociny/wip-16962

rbd-mirror: snap protect of non-layered image results in split-brain

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11755 from trociny/wip-17791

librbd: proper check for get_data_pool compatibility

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10852 from wido/doc-mon-dns

doc: Add docs about looking up Monitors through DNS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11774 from wjwithagen/wip-wjw-test_bloom_filter

test_bloom_filter.cc: Fix a mismatch for the random_seed parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11731 from kylinstorage/wip-avoid-unnecessary-verify

FileStore: Only verify split when it has been really done and done correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/common/test_bloom_filter.cc: Fix a mismatch for the random_seed parameter

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11554 from mslovy/wip-fix-fiemap

os/FileStore: fix fiemap issue in xfs when #extents > 1364

Reviewed-by: Sage Weil <sage@redhat.com>

test: add test for fiemap xfs issue when #extents > 1364

Signed-off-by: Ning Yao <yaoning@unitedstack.com>

FileStore:: fix fiemap issue in xfs when #extents > 1364

Signed-off-by: Ning Yao <yaoning@unitedstack.com>

os/kstore: fix collection_list end bound off-by-one

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix collection_list end bound off-by-one

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test collection_list end bound

Ensure that it is not included in the result!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11617 from umullan/brokenlinks

doc: Fixing the broken hyperlinks by pointing to correct documentation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10707 from SUSE/wip-fix-suse-usermod

rpm: Remove trailing whitespace in usermod command (SUSE)

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11494 from aclamk/wip-rgw-compression

[RGW] Wip rgw compression
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: fix build error

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11768 from linuxbox2/xio-build-2

xio: fix build

Merge pull request #11769 from liewegas/wip-option-name

kv/RocksDBStore: rename option

Reviewed-by: Mark Nelson <mnelson@redhat.com>

kv/RocksDBStore: rename option

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11767 from ceph/wip-cmake

cmake: pass CMAKE_BUILD_TYPE down to rocksdb

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>

xio: build fixes (some unittest targets need XIO_LIBRARY)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #11748 from liewegas/wip-bluestore-split

os/bluestore: fix _split_collections race with osr_reap

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

xio: build fixes (XioMessenger lacks features argument)

Applies the constructor signature change to the uses in
xio_client and xio_server test programs.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

xio:  compile fix removing undeclared 'features'

This fix was proposed on ceph-devel and acked by Sage Weil.
The corresponding change in in the later print is obvious
from context.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: fix _split_collections race with osr_reap

The SharedBlobSet may not yet be empty at split time because previous
transactions may not have been reaped, leaving some Blobs still alive
even after the cache refs are cleared.  Drop them explicitly here (they
will go away shortly).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9444 from Adirl/dout_prefix

xio: provide dout_prefix for XioConnection

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: pass CMAKE_BUILD_TYPE down to rocksdb

so rocksdb share the same ${CMAKE_BUILD_TYPE} with ceph

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11704 from dillaman/wip-17752

librbd: do not create empty object map object on image creation

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11696 from tanghaodong25/rocksdb-breakdown-wip

kv/rocksdb: enable rocksdb write path breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11752 from stiopaa1/mds_mdsrank_addConstToMemFunc

mds/MDSRank: add const to member functions

Merge pull request #11761 from xiexingguo/xxg-wip-fix-envmirror-cleanup

os/bluestore: fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11762 from atheism/doc-rgw-s3-javaapi

doc: rgw: fix a typo in S3 java api example

librbd: proper check for get_data_pool compatibility

Fixes: http://tracker.ceph.com/issues/17791
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-nbd: disallow mapping images >2TB in size

Fixes: http://tracker.ceph.com/issues/17219
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: rgw: fix a typo in S3 java api example

    ObjectSummary.getSize() should be objectSummary.getSize()

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #11753 from yehudasa/wip-17790

rgw: support for x-robots-tag header
Reviewed-by: Orit wasserman <owasserm@redhat.com>

doc: Fixing the broken hyperlinks by pointing to correct documentation.
Signed-off-by: umullangi@walmartlabs.com

os/bluestore: fix improper call of shutdown() to mempool_thread

Because below here we haven't called mempool_thread.init() yet,
and call to mempool_thread.shutdown() will therefore raise an assert instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop safe check against g_conf

We are using g_conf unconditionly everywhere...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop redundant boundary check

As we have done the sanity check ahead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix "rm -rf" cmd

The original cmd won't do the tidy-up work completedly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: support for x-robots-tag header

Fixes: http://tracker.ceph.com/issues/17790
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11331 from liewegas/wip-mempool

mempool: mempool infrastructure, bluestore changes to use it

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

mempool: use function static for pool table

The compiler/linker guarantee this is initialized before any invocation
of this function... even if it is by a ctor in another compilation unit
that is initialized by the mempool.cc compilation unit.

Suggested by Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>

mempool: do not dump unused pools

Notably, unittest_[12]. :)

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: put pool namespaces within mempool::

e.g., mempool::bluestore_meta_other::list<...> ...

This avoids colliding with other names and types when the pool
name is something common (like "osd").

Signed-off-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/include/mempool.h

mempool: simplify debug settings

There's no need for a per-pool debug flag; just use the global
debug_mode bool!

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: force per-type factories to register with the pool

We want these to get their debug flag adjusted when it is turned
on and off.

(In contrast, we accept that STL containers will only be debugged
if mempool debug was on when they are constructed.)

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: parameterize stl containers encoders with allocator

This will let us encode/decode containers with custom allocators
(i.e., in mempools).

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: simplify initialization

Assume that the array will be zeroed when the process starts, and
avoid the possibility that it will be zeroed *again* when the
ctors run for this link module (potentially clobbering values
that have been filled in by other ctors who ran earlier than
ours).

Signed-off-by: Sage Weil <sage@redhat.com>

include/types: parameterize operator<< helpers on comparator, allocator

This allows these templates to apply to containers with
non-default comparators or allocators.

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: fix comment

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: dynamic index by type in pool_t

This avoids having to use statics for the pool_allocators and
guessing what intenral types the containers are going to need.
It'll be a bit slower in debug on on construction and destruction,
but who cares!

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: define additional pair factory for unordered_map

This fixes build on debian jessie, libstdc++-4.9-dev:amd64 4.9.2-10.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: pick shard on every allocation; move types to pool_t

If we have a static choice of shard for every *type*
(pool_allocator_base_t) then we will hammer that shard from all
threads and play cacheline ping-pong.  Instead, move the types
list to pool_t (there aren't that many anyway) and pick a shard
on every allocate/deallocate call.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: make num_shards a power of 2

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: rename containers -> types

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: note about runtime complexity

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: disable ref_map internal _check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: restructure cache trimming in terms of mempool

Trim cache based on overall memory utilization by cache objects,
as tracked by the bluestore_meta_* mempools.  This lets you
configure the bluestore cache size in terms of bytes of memory.

Note that we do not account for other memory utilization by the
OSD.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move most cache types into mempools

Keep onodes separate so we can use onodes as a sentinal for
overall memory usage (this is what we trim in terms of anyway).

Signed-off-by: Sage Weil <sage@redhat.com>

global: introduce mempool_debug config option, asok command

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: introduce mempool infrastructure

This is heavily based on Allen Samuels' prototype.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2 from cbodley/wip-rgw-compression-doc

doc/rgw: document rgw_compression_type

rgw: remove unused rgw_compression_enabled

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: document rgw_compression_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11726 from liewegas/wip-bluestore-fsck-omap

os/bluestore: fsck: fix omap_head check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fsck: fix omap_head check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11740 from xiexingguo/xxg-wip-bluestore-1102

os/bluestore: less code redundancy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11725 from liewegas/wip-bluestore-shard-span

os/bluestore: fix extent shard span check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Fixed warnings.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

rgw: maintain uncompressed size in copy_obj_data

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Fixed bad rebase on read_op.prepare()

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Fixed problem with isal-regular gzip compatibility.
Added tests.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Fixed missing param to processor->complete().

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

radosgw-admin: add 'size_utilized' to bucket stats

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use RGWStorageStats::dump() to format 'bucket stats'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Bucket::List::list_objects no longer reads compression attribute for size

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove compression attribute from bucket instance

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Fixed possible problems when compression expands data.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Fixed problem with snappy compressor when data size <=3.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

rgw: pass accounted_size directly into RGWPutObjProcessor::complete

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size to RGWUploadPartInfo

and use accounted_size for ListMultipart operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size argument to Write::write_meta

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size argument to UpdateIndex::complete

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size to RGWObjEnt

and use accounted_size for BucketList operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>