xio: build fixes (some unittest targets need XIO_LIBRARY)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

xio: build fixes (XioMessenger lacks features argument)

Applies the constructor signature change to the uses in
xio_client and xio_server test programs.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

xio:  compile fix removing undeclared 'features'

This fix was proposed on ceph-devel and acked by Sage Weil.
The corresponding change in in the later print is obvious
from context.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9444 from Adirl/dout_prefix

xio: provide dout_prefix for XioConnection

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11704 from dillaman/wip-17752

librbd: do not create empty object map object on image creation

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11696 from tanghaodong25/rocksdb-breakdown-wip

kv/rocksdb: enable rocksdb write path breakdown

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11752 from stiopaa1/mds_mdsrank_addConstToMemFunc

mds/MDSRank: add const to member functions

Merge pull request #11761 from xiexingguo/xxg-wip-fix-envmirror-cleanup

os/bluestore: fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11762 from atheism/doc-rgw-s3-javaapi

doc: rgw: fix a typo in S3 java api example

doc: rgw: fix a typo in S3 java api example

    ObjectSummary.getSize() should be objectSummary.getSize()

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #11753 from yehudasa/wip-17790

rgw: support for x-robots-tag header
Reviewed-by: Orit wasserman <owasserm@redhat.com>

os/bluestore: fix improper call of shutdown() to mempool_thread

Because below here we haven't called mempool_thread.init() yet,
and call to mempool_thread.shutdown() will therefore raise an assert instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop safe check against g_conf

We are using g_conf unconditionly everywhere...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop redundant boundary check

As we have done the sanity check ahead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix "rm -rf" cmd

The original cmd won't do the tidy-up work completedly.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: support for x-robots-tag header

Fixes: http://tracker.ceph.com/issues/17790
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11331 from liewegas/wip-mempool

mempool: mempool infrastructure, bluestore changes to use it

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

mempool: use function static for pool table

The compiler/linker guarantee this is initialized before any invocation
of this function... even if it is by a ctor in another compilation unit
that is initialized by the mempool.cc compilation unit.

Suggested by Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

Signed-off-by: Allen Samuels <allen.samuels@sandisk.com>

mempool: do not dump unused pools

Notably, unittest_[12]. :)

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: put pool namespaces within mempool::

e.g., mempool::bluestore_meta_other::list<...> ...

This avoids colliding with other names and types when the pool
name is something common (like "osd").

Signed-off-by: Sage Weil <sage@redhat.com>
# Conflicts:
# src/include/mempool.h

mempool: simplify debug settings

There's no need for a per-pool debug flag; just use the global
debug_mode bool!

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: force per-type factories to register with the pool

We want these to get their debug flag adjusted when it is turned
on and off.

(In contrast, we accept that STL containers will only be debugged
if mempool debug was on when they are constructed.)

Signed-off-by: Sage Weil <sage@redhat.com>

include/encoding: parameterize stl containers encoders with allocator

This will let us encode/decode containers with custom allocators
(i.e., in mempools).

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: simplify initialization

Assume that the array will be zeroed when the process starts, and
avoid the possibility that it will be zeroed *again* when the
ctors run for this link module (potentially clobbering values
that have been filled in by other ctors who ran earlier than
ours).

Signed-off-by: Sage Weil <sage@redhat.com>

include/types: parameterize operator<< helpers on comparator, allocator

This allows these templates to apply to containers with
non-default comparators or allocators.

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: fix comment

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: dynamic index by type in pool_t

This avoids having to use statics for the pool_allocators and
guessing what intenral types the containers are going to need.
It'll be a bit slower in debug on on construction and destruction,
but who cares!

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: define additional pair factory for unordered_map

This fixes build on debian jessie, libstdc++-4.9-dev:amd64 4.9.2-10.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: pick shard on every allocation; move types to pool_t

If we have a static choice of shard for every *type*
(pool_allocator_base_t) then we will hammer that shard from all
threads and play cacheline ping-pong.  Instead, move the types
list to pool_t (there aren't that many anyway) and pick a shard
on every allocate/deallocate call.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: make num_shards a power of 2

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: rename containers -> types

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: note about runtime complexity

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: disable ref_map internal _check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: restructure cache trimming in terms of mempool

Trim cache based on overall memory utilization by cache objects,
as tracked by the bluestore_meta_* mempools.  This lets you
configure the bluestore cache size in terms of bytes of memory.

Note that we do not account for other memory utilization by the
OSD.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move most cache types into mempools

Keep onodes separate so we can use onodes as a sentinal for
overall memory usage (this is what we trim in terms of anyway).

Signed-off-by: Sage Weil <sage@redhat.com>

global: introduce mempool_debug config option, asok command

Signed-off-by: Sage Weil <sage@redhat.com>

include/mempool: introduce mempool infrastructure

This is heavily based on Allen Samuels' prototype.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11726 from liewegas/wip-bluestore-fsck-omap

os/bluestore: fsck: fix omap_head check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fsck: fix omap_head check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11740 from xiexingguo/xxg-wip-bluestore-1102

os/bluestore: less code redundancy

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11725 from liewegas/wip-bluestore-shard-span

os/bluestore: fix extent shard span check

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11738 from ceph/wip-cmake

cmake: do not link erasure tests again libosd

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11384 from wjwithagen/patch-14

test_subman.sh: Don't use --tmpdir

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test_subman.sh: Don't use --tmpdir

test_subman.sh: Don't use --tmpdir since on most Linux-is it is consideren default
when creating a directory:
`If TEMPLATE is not specified, use tmp.XXXXXXXXXX, and --tmpdir is implied.`

And on FreeBSD --tmpdir is not available anyways.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: use string for most cases in _open_db()

For a better readability.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: do not link erasure tests again libosd

they are not referencing any symbols from it. and also the tests testing
plugins are not linked against the ec plugins, they *depend* on them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

kv/rocksdb: enable rocksdb write path breakdown

bluestore has unstable write performance partly because of
rocksdb internal overhead. So we add more breakdown in
rocksdb writh path.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #11730 from ceph/wip-17654

librados-dev: install inline_memory.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: drop unused parameter "deep" from _fsck_check_extents()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/ceph_objectstore_tool: handle fsck and fsck-deep op in unified path

https://github.com/ceph/ceph/pull/11724/commits/181e28ba999c4134957f04183bad5c4dfc0e7062
introduces too much code redundancy, this patch simplifies it a little.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds/MDSRank: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librados-dev: install inline_memory.h

This is needed to compile against the c++ api since
c38869232c3c852d98fa1826632db360c5a6afd4

Fixes: http://tracker.ceph.com/issues/17654
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11697 from weiqiaomiao/wqm-wip-rgw-bilist

rgw: fix osd crashes when execute "radosgw-admin bi list --max-entries=1" command

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11695 from tanghaodong25/cache-logger-wip

os/bluestore: update cache logger after 'trim_cache' operation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11506 from cbodley/wip-17569

rgw: clear data_sync_cr if RGWDataSyncControlCR fails

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11707 from yehudasa/wip-17755

rgw: json encode/decode index_type, allow modification

os/bluestore: rename variable in set_lextent

Defensive naming

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix shard span check in set_lextent

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: check for extents spanning shards

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: udpate cache logger after trim_cache

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #11708 from nitinkamble/master

install-deps.sh: allow building on SLES systems

Merge pull request #10767 from rzarzynski/wip-rgw-frontend-rework

rgw: frontend subsystem rework

all requested changes have been made by the author
tested by-hand w/sigv2 and sigv4, and already passed teuthology rgw suite

Merge pull request #11724 from liewegas/wip-bluestore-fsck

os/bluestore: deep fsck

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: verify we can read objects on deep fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os: add fsck deep/shallow option

deep==false checks just metadata, and deep==true will (eventually) check
all data (and checksums) too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix synthetic fsck

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11537 from liewegas/wip-bluestore-parallel

os/bluestore: refactor bluestore_sync_submit_transaction

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #11661 from ceph/wip-git-ver

cmake: fix git version string, cleanup

os/bluestore: default to bluestore_sync_submit_transaction=true

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: randomly inject serialize transaction submission

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: move main() to bottom

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix {nid,blobid}_max

- single case to cover increases ahead of schedule or just in time
- update global max only after txn commits, eliminating race
- drop unneeded id_lock
- improve t naming

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prevent sync transaction submit with extent freelist

It doesn't support unordered transactions.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: submit txn via kv_sync_thread if ids exceed max

This ensures the txn will not commit before an update to the
global max.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: refactor bluestore_sync_submit_transaction

And drop bluestore_sync_transaction, at least for now.

The key change here is to make a per-txc flag indicating
whether the txn was already submitted.  This allows us
to make a choice between sync and not-sync on a per-txn
basis.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11721 from dachary/wip-17561-log

tests: TestErasureCodePluginJerasure must stop the log thread

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11718 from xiexingguo/xxg-wip-bluestore-1101

os/bluestore: add counter to trace blob splitting

Reviewed-by: Sage Weil <sage@redhat.com>

tests: TestErasureCodePluginJerasure must stop the log thread

When a log entry is created in dout.h, it holds a pointer to the

   static size_t _log_exp_length

variable. When OnExitManager::~OnExitManager from Log.cc is called
during global destruction, it may happen after the static variables
referenced by some entries have been deallocated. The flush() function
will try to access these deallocated variables via the hint_size()
method and core dump.

To prevent this race, the Log::stop() method must be called explicitly
to ensure no entries are left when the global destructors are called.

Fixes: http://tracker.ceph.com/issues/17561
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10248 from ktdreyer/wip-doc-no-rpm-md

doc: rm "type=rpm-md" from yum repositories

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #11690 from yonghengdexin735/wip-zzz-fixword

src/common/perf_counters.h: fix wrong word

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check when compress_extent_map()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting changes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add counter to trace blob splitting

As we want to know if it really happens and how often does it happen.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11568 from dillaman/wip-17588

librbd: cannot access non-primary image when mirroring force disabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11280 from ceph/wip-17440

rgw: fixes for virtual hosting of buckets
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11559 from dillaman/wip-17618

librbd: exclusive lock incorrectly initialized when switching to head revision

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11623 from dillaman/wip-cls-rbd-group

librbd: ensure consistency groups will gracefully fail on older OSDs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11703 from dillaman/wip-17750

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11714 from dachary/wip-17757-eio

tests: disable failing tests

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

tests: disable failing unittest_erasure_code_plugin_jerasure

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17561

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: disable failing test-erasure-eio.sh

While figuring out what's wrong with it.

Refs: http://tracker.ceph.com/issues/17757

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11382 from yuyuyu101/wip-fix-shadow

osd/PGBackend: fix collection_list shadow return value

Reviewed-by: Kefu Chai <kchai@redhat.com>
tchaikov

Merge pull request #11547 from javeme/check-pgid

osd/command tell: check pgid at the right time

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11203 from stiopaa1/osd_pg_replaceCountPart1

osd/PG.cc: prevent repeated searching of map/set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11242 from stiopaa1/osd_replicatedpg_removeCount1

osd/ReplicatedPG: remove unneeded use of count

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11365 from badone/wip-lsb_release-remove

common: Remove the runtime dependency on lsb_release

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer (Red Hat) <kdreyer@redhat.com>

test: skip TestLibRBD.DiscardAfterWrite if skip partial discard enabled

Fixes: http://tracker.ceph.com/issues/17750
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: allow building on SLES systems

Avoids this error on SLES systems:

> ./install-deps.sh
sles is unknown, dependencies will have to be installed manually.

Signed-off-by: Nitin A Kamble <Nitin.Kamble@Teradata.com>

rgw: json encode/decode index_type, allow modification

Fixes: http://tracker.ceph.com/issues/17755
We didn't dump the index_type field of RGWBucketInfo, also didn't
allow modifying it via metadata api.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: rm "type=rpm-md" from yum repositories

As far as I can tell, this is only relevant to SUSE's Zypper, and yum
does not need it.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>