msg: make loopback Connection feature accurate all the time

In 626360aab05545ddacb0ac28e54a70e31fd5695d we made the
OSD cluster loopback connection CEPH_FEATURES_ALL, but
all other loopback connections got features == 0.  I
can't come up with any reason we wouldn't want those
connections to have accurate feature bits, so let's just
use CEPH_FEATURES_ALL for all of them.

While we're here, make the cflags argument required.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11281 from jcsp/wip-17466

mds: handle bad standby_for_fscids in fsmap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10722 from stiopaa1/mon_moncap_addMoveToStrings

mon/MonCap.h: add std::move for std::string

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10916 from jordan41177/fix-17169

mon: OSDMonitor: fix the check error of pg creating

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10720 from wjwithagen/wip-wjw-freebsd-socket-shutdown

msg/simple/Accepter.cc: replace shutdown() with selfpipe event in poll() (FreeBSD)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10601 from stiopaa1/mon_monclient_subwantRemoveCount

mon/MonClient.h: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11145 from atheism/msg-async-fix-specifier

msg: fix format specifier for unsigned value id

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11379 from zealoussnow/fix-dead-link

doc: fix dead link "Hardware Recommendations"

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix dead link "Hardware Recommendations"

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

Merge pull request #10057 from Yan-waller/yj-wip-rgwradosccremover-0701

rgw: minor cleanup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10674 from ceph/wip-rgw-no-empty-amzrqid

rgw/rest: don't print empty x-amz-request-id

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10849 from zhangsw/master

rgw: remove a redundant judgement when listng objects.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11353 from dillaman/wip-17423

librbd: API changes to support separate data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11361 from batrick/fs-rm-formatting

mon: add missing space in warning message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11364 from badone/wip-_lsb_release_-dead-declarations

osd: Remove unused '_lsb_release_' declarations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11349 from liewegas/wip-compressor

compressor: fixes and tests; disable zlib isal (it's broken)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11366 from xiexingguo/xxg-wip-bluestore-1007

os/bluestore: simplify blob status checking for small writes

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_compression: naming nit

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression: restore zlib_isal setting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11236 from renhwztetecs/renhw-wip-cephfs-journal-tool

tool/cephfs: displaying "list" in journal event mode

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10021 from rzarzynski/wip-rgw-swift-auth-errors

rgw: handle Swift auth errors in a way compatible with new Tempests.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: make randomizing value process of kvpair faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fsck() on idempotent mkfs() request

E.g., if the fsck() found something during the first mkfs() try,
it can still be caught by the following retry.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix comment

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: Remove unused '_lsb_release_' declarations

_lsb_release_set and _lsb_release_parse declarations still exist despite their
definitions being removed some time ago.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: sanity check iterator of BufferSpace::_rm_buffer()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify blob status checking for small writes

If a blob is marked as compressed and still mutable, something
is wrong.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11302 from asheplyakov/rados-doc-fixup

doc: fix librados example programs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon: add missing space in warning message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

unittest_compression: subsume plugin test

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression: subsume _zlib and _snappy test cases

This includes zlib and isal interop.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compressor: smaller big test

Just so it runs a bit faster!

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/zlib: disable ISA-L

This fails the unit tests.  Do not reenable until it is fixed!

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/zlib: add compressor_zlib_isal config option to disable ISA-L

We dynamically enable this if the necessary processor features are present.
Allow this probing to be disabled explicitly.

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/snappy: fix decompress buffer sizing for large buffers

4 bytes is enough for small buffers, but fails for larger buffers because
snappy encodes the length as a varint.  8 is enough.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compressor: unified compression tests, parameterized by plugin

This obsoletes the per-plugin plugin tests, which are tedious anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: utilize new API methods for image id and block name prefix

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new API methods to retrieve image id and block name prefix

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: relocate image option processing to create state machine

Replace the individual image option parameters with the ImageOptions
wrapper and extract the options within the state machine itself.
This will simplify adding support for the data pool and other options
in the future.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add data pool option to API and config

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: fix compiler warning due to feature bit macro type change

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11326 from dillaman/wip-17416

rbd-mirror: improve resiliency of stress test case

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11327 from dillaman/wip-17422

librbd: cls_rbd updates for separate data pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11329 from atheism/rgw-b64-remove-ostr

rgw: remove unused variable "ostr" in rgw_b64.h and fix the comment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11351 from ifed01/wip-bluestore-misc2

os/bluestore: Fix invalid compression statfs caused by clone op

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11350 from liewegas/wip-bluestore-compression

os/bluestore: fix up compression tests and debug output

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fix invalid compression statfs caused by clone op

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11317 from liewegas/wip-bluestore-aios

os/bluestore: fix has_aios

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

ceph_test_objectstore: adjust SyntheticMatrixCsumVsCompression parameters

- larger writes
- test both codecs
- use smaller min_alloc_size so that it's easier to write compressed data
- don't bother with compression off; other matrices cover that.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clean up compression debug output

The previous output was impossible to interpret properly.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: limit entropy in 'random' data so it can compress

Both zlib and snappy weren't able to do anything with the current data.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix build error

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11346 from ifed01/wip-bluestore-misc

os/bluestore: misc cleanup and test fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11328 from liewegas/wip-bluestore-more-types

os/bluestore: pack a few more in-memory types

test/store_test: BluestoreStatFSTest fixes for issues introduced with bluestore_compression_min_blob_size adjustment

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: some cleanup around garbage collector

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11344 from xiexingguo/xxg-wip-bluestore-1006

os/bluestore: fix write_big counter and some more cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: fix comment

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify bluestore_gc_merge_data checking

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix counter of l_bluestore_write_big_bytes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: more cleanups on _do_write_small()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add _dump_transaction to reduce code redundancy

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: upgrade wal_seq to std::atomic

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11336 from liewegas/wip-bluestore-spanning-map

os/bluestore: switch spanning_blob_map to std::map

Merge pull request #11340 from yehudasa/wip-13286

rgw-admin: allow unsetting user's email

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11334 from ceph/wip-make-dist-tag

make-dist: set rpm_release correctly for release builds

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: remove user email index using helper function

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw:user email can modify to empty

Fixes: http://tracker.ceph.com/issues/13286
Signed-off-by: Weijun Duan <duanweijun@h3c.com>

make-dist: set rpm_release correctly for release builds

Similar to autobuild-ceph 17591db, only try to set rpm_release if there
is a release part to the git describe output (i.e. if it's not at a
version tag)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

os/bluestore: switch spanning_blob_map to std::map

The intrusive map overhead on Blob is expensive when a tiny fraction of
Blobs actually end up in the spanning_blob_map.  Let's pay the extra
allocation overhead (for the map node holding the BlobRef) for those rare
cases.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11127 from SirishaGuduru/multitenancy-doc-change

Changed config parameter "rgw keystone make new tenants" in radosgw multitenancy

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10159 from mhackett/wip-mhackett-16604

doc: add rgw_enable_usage_log option in Rados Gateway admin guide

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10641 from weiqiaomiao/wqm-wip-rgw-s3api

doc/radosgw: fix description of response elements 'Part'

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11286 from yehudasa/wip-16930

rgw: don't fail if lost race when setting acls

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused variable "ostr" in rgw_b64.h and fix the comment

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #11319 from markhpc/wip-bluestore-optimize-size

os/bluestore: optimize intrusive sets for size.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11313 from ceph/wip-krbd-lock-on-read-option

rbd: recognize lock_on_read option

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: simplify BufferSpace writing_map / list

We were tracking buffers by seq.  In reality, though, we
won't have very many buffers that have in flight IO to
the same blob at the same time, so we can get by with
just a list here.  We can rely on the fact that the
list will be sorted (seq's increase, and we put new
buffers at the end).

This saves us a memory allocation.  No change in size of
BufferSpace and Blob (std::list -> intrusive::list).

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: more type sizeofs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11264 from liewegas/wip-bluestore-blob-split

os/bluestore: try to split blobs instead of spanning them

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: make Buffer smaller

160 -> 152 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

cls_rbd: add initial support for separate data pools

Fixes: http://tracker.ceph.com/issues/17422
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new RBD_FEATURE_DATA_POOL feature bit

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11323 from liewegas/wip-bluestore-more-types

unittest_bluestore_types: a few more types for sizeof

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

qa/workunits: reduce amount of client IO for rbd-mirror stress

Journal IO is faster now, resulting in OSDs quickly filling up
under the stress test case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: potential race allocating tag during shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ignore attempts to request lock if already lock owner

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: clean up object recorder closed/overflow callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: delay moving overflowed buffers until quiesced

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: use reverse iterator search to find flush record

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: avoid holding lock while sending journal append

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10223 from mikulely/doc-s3-static-website-support

doc: update s3 static webiste feature support status

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11325 from yehudasa/wip-17499

rgw: need to 'open_object_section' before dump stats in 'RGWGetUsage_…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: need to 'open_object_section' before dump stats in 'RGWGetUsage_ObjStore_S3::send_response()' function

Fixes: http://tracker.ceph.com/issues/17499
Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

Merge pull request #10820 from yanghonggang/master

rgw: fix get object instance  returned NoSuchKey error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10726 from weiqiaomiao/wqm-wip-remove-ret

rgw: remove the field ret from class RGWPutLC

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: make BufferSpace smaller

Saves 24 bytes!

BufferSpace  104 -> 80 bytes
SharedBlob   216 -> 192 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pack SharedBlob

224 -> 216 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pack Blob members

272 -> 264 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: a few more types for sizeof

Signed-off-by: Sage Weil <sage@redhat.com>