rgw: fix uncompressed object size deduction in RGWRados::copy_obj_data.

Fixes: http://tracker.ceph.com/issues/17803
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #10852 from wido/doc-mon-dns

doc: Add docs about looking up Monitors through DNS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11774 from wjwithagen/wip-wjw-test_bloom_filter

test_bloom_filter.cc: Fix a mismatch for the random_seed parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11731 from kylinstorage/wip-avoid-unnecessary-verify

FileStore: Only verify split when it has been really done and done correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/common/test_bloom_filter.cc: Fix a mismatch for the random_seed parameter

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11554 from mslovy/wip-fix-fiemap

os/FileStore: fix fiemap issue in xfs when #extents > 1364

Reviewed-by: Sage Weil <sage@redhat.com>

test: add test for fiemap xfs issue when #extents > 1364

Signed-off-by: Ning Yao <yaoning@unitedstack.com>

FileStore:: fix fiemap issue in xfs when #extents > 1364

Signed-off-by: Ning Yao <yaoning@unitedstack.com>

Merge pull request #11617 from umullan/brokenlinks

doc: Fixing the broken hyperlinks by pointing to correct documentation.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10707 from SUSE/wip-fix-suse-usermod

rpm: Remove trailing whitespace in usermod command (SUSE)

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #11494 from aclamk/wip-rgw-compression

[RGW] Wip rgw compression
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: fix build error

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11768 from linuxbox2/xio-build-2

xio: fix build

Merge pull request #11769 from liewegas/wip-option-name

kv/RocksDBStore: rename option

Reviewed-by: Mark Nelson <mnelson@redhat.com>

kv/RocksDBStore: rename option

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11767 from ceph/wip-cmake

cmake: pass CMAKE_BUILD_TYPE down to rocksdb

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>

xio: build fixes (some unittest targets need XIO_LIBRARY)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #11748 from liewegas/wip-bluestore-split

os/bluestore: fix _split_collections race with osr_reap

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

xio: build fixes (XioMessenger lacks features argument)

Applies the constructor signature change to the uses in
xio_client and xio_server test programs.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

xio:  compile fix removing undeclared 'features'

This fix was proposed on ceph-devel and acked by Sage Weil.
The corresponding change in in the later print is obvious
from context.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: fix _split_collections race with osr_reap

The SharedBlobSet may not yet be empty at split time because previous
transactions may not have been reaped, leaving some Blobs still alive
even after the cache refs are cleared.  Drop them explicitly here (they
will go away shortly).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9444 from Adirl/dout_prefix

xio: provide dout_prefix for XioConnection

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: pass CMAKE_BUILD_TYPE down to rocksdb

so rocksdb share the same ${CMAKE_BUILD_TYPE} with ceph

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11704 from dillaman/wip-17752

librbd: do not create empty object map object on image creation

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11696 from tanghaodong25/rocksdb-breakdown-wip

kv/rocksdb: enable rocksdb write path breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11752 from stiopaa1/mds_mdsrank_addConstToMemFunc

mds/MDSRank: add const to member functions

Merge pull request #11761 from xiexingguo/xxg-wip-fix-envmirror-cleanup

os/bluestore: fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11762 from atheism/doc-rgw-s3-javaapi

doc: rgw: fix a typo in S3 java api example

doc: rgw: fix a typo in S3 java api example

    ObjectSummary.getSize() should be objectSummary.getSize()

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #11753 from yehudasa/wip-17790

rgw: support for x-robots-tag header
Reviewed-by: Orit wasserman <owasserm@redhat.com>

doc: Fixing the broken hyperlinks by pointing to correct documentation.
Signed-off-by: umullangi@walmartlabs.com

os/bluestore: fix improper call of shutdown() to mempool_thread

Because below here we haven't called mempool_thread.init() yet,
and call to mempool_thread.shutdown() will therefore raise an assert instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop safe check against g_conf

We are using g_conf unconditionly everywhere...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop redundant boundary check

As we have done the sanity check ahead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix "rm -rf" cmd

The original cmd won't do the tidy-up work completedly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: support for x-robots-tag header

Fixes: http://tracker.ceph.com/issues/17790
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11331 from liewegas/wip-mempool

mempool: mempool infrastructure, bluestore changes to use it

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

mempool: use function static for pool table

The compiler/linker guarantee this is initialized before any invocation
of this function... even if it is by a ctor in another compilation unit
that is initialized by the mempool.cc compilation unit.

Suggested by Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>

mempool: do not dump unused pools

Notably, unittest_[12]. :)

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: put pool namespaces within mempool::

e.g., mempool::bluestore_meta_other::list<...> ...

This avoids colliding with other names and types when the pool
name is something common (like "osd").

Signed-off-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/include/mempool.h

mempool: simplify debug settings

There's no need for a per-pool debug flag; just use the global
debug_mode bool!

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: force per-type factories to register with the pool

We want these to get their debug flag adjusted when it is turned
on and off.

(In contrast, we accept that STL containers will only be debugged
if mempool debug was on when they are constructed.)

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: parameterize stl containers encoders with allocator

This will let us encode/decode containers with custom allocators
(i.e., in mempools).

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: simplify initialization

Assume that the array will be zeroed when the process starts, and
avoid the possibility that it will be zeroed *again* when the
ctors run for this link module (potentially clobbering values
that have been filled in by other ctors who ran earlier than
ours).

Signed-off-by: Sage Weil <sage@redhat.com>

include/types: parameterize operator<< helpers on comparator, allocator

This allows these templates to apply to containers with
non-default comparators or allocators.

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: fix comment

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: dynamic index by type in pool_t

This avoids having to use statics for the pool_allocators and
guessing what intenral types the containers are going to need.
It'll be a bit slower in debug on on construction and destruction,
but who cares!

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: define additional pair factory for unordered_map

This fixes build on debian jessie, libstdc++-4.9-dev:amd64 4.9.2-10.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: pick shard on every allocation; move types to pool_t

If we have a static choice of shard for every *type*
(pool_allocator_base_t) then we will hammer that shard from all
threads and play cacheline ping-pong.  Instead, move the types
list to pool_t (there aren't that many anyway) and pick a shard
on every allocate/deallocate call.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: make num_shards a power of 2

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: rename containers -> types

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: note about runtime complexity

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: disable ref_map internal _check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: restructure cache trimming in terms of mempool

Trim cache based on overall memory utilization by cache objects,
as tracked by the bluestore_meta_* mempools.  This lets you
configure the bluestore cache size in terms of bytes of memory.

Note that we do not account for other memory utilization by the
OSD.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move most cache types into mempools

Keep onodes separate so we can use onodes as a sentinal for
overall memory usage (this is what we trim in terms of anyway).

Signed-off-by: Sage Weil <sage@redhat.com>

global: introduce mempool_debug config option, asok command

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: introduce mempool infrastructure

This is heavily based on Allen Samuels' prototype.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2 from cbodley/wip-rgw-compression-doc

doc/rgw: document rgw_compression_type

rgw: remove unused rgw_compression_enabled

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: document rgw_compression_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11726 from liewegas/wip-bluestore-fsck-omap

os/bluestore: fsck: fix omap_head check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fsck: fix omap_head check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11740 from xiexingguo/xxg-wip-bluestore-1102

os/bluestore: less code redundancy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11725 from liewegas/wip-bluestore-shard-span

os/bluestore: fix extent shard span check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Fixed warnings.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

rgw: maintain uncompressed size in copy_obj_data

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Fixed bad rebase on read_op.prepare()

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Fixed problem with isal-regular gzip compatibility.
Added tests.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Fixed missing param to processor->complete().

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

radosgw-admin: add 'size_utilized' to bucket stats

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use RGWStorageStats::dump() to format 'bucket stats'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Bucket::List::list_objects no longer reads compression attribute for size

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove compression attribute from bucket instance

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Fixed possible problems when compression expands data.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Fixed problem with snappy compressor when data size <=3.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

rgw: pass accounted_size directly into RGWPutObjProcessor::complete

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size to RGWUploadPartInfo

and use accounted_size for ListMultipart operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size argument to Write::write_meta

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size argument to UpdateIndex::complete

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size to RGWObjEnt

and use accounted_size for BucketList operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add accounted_size to RGWObjState

RGWRados::get_obj_state() now initializes accounted_size based on
the compression attribute, if present

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: use rgw_bucket_dir_entry_meta::size to accumulate actual_size

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: add actual_size to rgw_bucket_caregory_stats

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: fix function names of log output

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Added missing decompression when reading SLO.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Merge pull request #11738 from ceph/wip-cmake

cmake: do not link erasure tests again libosd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11384 from wjwithagen/patch-14

test_subman.sh: Don't use --tmpdir

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Added compression handling when syncing.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Moved handling extra data from RGWPutObjProcessor to RGWRadosPutObj

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Cleanup of compression-related code in RGWGetObj::execute.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

RGWRados::Object::Read::prepare no longer calculates offsets.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Fixed problem with using uninitialized 'compressed' field.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

test/rgw: add unit test for rgw compression

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move compression code in filters system
Signed-off-by: Alyona Kiseleva <akiselyova@mirantis.com>

Renamed and moved RGWPutObjEncryptFilter -> RGWPutObj_Filter.

Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com)

Introduced filter concepts to RGWGetObj. Plugged in decryption.
Allows to plug in decompression in future.

Signed-off-by: Adam Kupczyk (a.kupczyk@mirantis.com)
Conflicts:
src/rgw/rgw_op.cc

Change contract of rgw put operation. Now last, 0 - sized chunk is also processed.
Allows to implement flush-like operations on data.

Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com)
Conflicts:
src/rgw/rgw_op.cc

Added RGWPutObjEncryptFilter. Will be used for encryption and compression filters.

Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com)

Changed hash calculation logic.
Changed RGWPutObjProcessor interface with regard to hash calculation.

Signed-off-by: Adam Kupczyk (akupczyk@mirantis.com)
Conflicts:
src/rgw/rgw_op.cc
src/rgw/rgw_rados.cc

rgw: fix output for stat functions to return decompressed data size
Signed-off-by: Alyona Kiseleva <akiselyova@mirantis.com>