os/bluestore: allow default to buffered write

Add config option to buffered writes (in the absense of
a client WILLNEED hint).  Set the config option to false,
though, so that the default behavior is unchanged.

2Q should do a half-decent job of preventing this from
pushing actually-hot data from the cache when it's
enabled, but we'll need to do more testing first.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11329 from atheism/rgw-b64-remove-ostr

rgw: remove unused variable "ostr" in rgw_b64.h and fix the comment

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11351 from ifed01/wip-bluestore-misc2

os/bluestore: Fix invalid compression statfs caused by clone op

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11350 from liewegas/wip-bluestore-compression

os/bluestore: fix up compression tests and debug output

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fix invalid compression statfs caused by clone op

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11317 from liewegas/wip-bluestore-aios

os/bluestore: fix has_aios

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

ceph_test_objectstore: adjust SyntheticMatrixCsumVsCompression parameters

- larger writes
- test both codecs
- use smaller min_alloc_size so that it's easier to write compressed data
- don't bother with compression off; other matrices cover that.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clean up compression debug output

The previous output was impossible to interpret properly.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: limit entropy in 'random' data so it can compress

Both zlib and snappy weren't able to do anything with the current data.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix build error

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11346 from ifed01/wip-bluestore-misc

os/bluestore: misc cleanup and test fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11328 from liewegas/wip-bluestore-more-types

os/bluestore: pack a few more in-memory types

test/store_test: BluestoreStatFSTest fixes for issues introduced with bluestore_compression_min_blob_size adjustment

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: some cleanup around garbage collector

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11344 from xiexingguo/xxg-wip-bluestore-1006

os/bluestore: fix write_big counter and some more cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: fix comment

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify bluestore_gc_merge_data checking

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix counter of l_bluestore_write_big_bytes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: more cleanups on _do_write_small()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add _dump_transaction to reduce code redundancy

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: upgrade wal_seq to std::atomic

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11336 from liewegas/wip-bluestore-spanning-map

os/bluestore: switch spanning_blob_map to std::map

Merge pull request #11340 from yehudasa/wip-13286

rgw-admin: allow unsetting user's email

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11334 from ceph/wip-make-dist-tag

make-dist: set rpm_release correctly for release builds

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: remove user email index using helper function

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw:user email can modify to empty

Fixes: http://tracker.ceph.com/issues/13286
Signed-off-by: Weijun Duan <duanweijun@h3c.com>

make-dist: set rpm_release correctly for release builds

Similar to autobuild-ceph 17591db, only try to set rpm_release if there
is a release part to the git describe output (i.e. if it's not at a
version tag)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

os/bluestore: switch spanning_blob_map to std::map

The intrusive map overhead on Blob is expensive when a tiny fraction of
Blobs actually end up in the spanning_blob_map.  Let's pay the extra
allocation overhead (for the map node holding the BlobRef) for those rare
cases.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11127 from SirishaGuduru/multitenancy-doc-change

Changed config parameter "rgw keystone make new tenants" in radosgw multitenancy

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10159 from mhackett/wip-mhackett-16604

doc: add rgw_enable_usage_log option in Rados Gateway admin guide

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10641 from weiqiaomiao/wqm-wip-rgw-s3api

doc/radosgw: fix description of response elements 'Part'

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11286 from yehudasa/wip-16930

rgw: don't fail if lost race when setting acls

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused variable "ostr" in rgw_b64.h and fix the comment

Signed-off-by: Weibing Zhang <zhangweibing@unitedstack.com>

Merge pull request #11319 from markhpc/wip-bluestore-optimize-size

os/bluestore: optimize intrusive sets for size.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11313 from ceph/wip-krbd-lock-on-read-option

rbd: recognize lock_on_read option

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: simplify BufferSpace writing_map / list

We were tracking buffers by seq.  In reality, though, we
won't have very many buffers that have in flight IO to
the same blob at the same time, so we can get by with
just a list here.  We can rely on the fact that the
list will be sorted (seq's increase, and we put new
buffers at the end).

This saves us a memory allocation.  No change in size of
BufferSpace and Blob (std::list -> intrusive::list).

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: more type sizeofs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11264 from liewegas/wip-bluestore-blob-split

os/bluestore: try to split blobs instead of spanning them

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: make Buffer smaller

160 -> 152 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11323 from liewegas/wip-bluestore-more-types

unittest_bluestore_types: a few more types for sizeof

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10223 from mikulely/doc-s3-static-website-support

doc: update s3 static webiste feature support status

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11325 from yehudasa/wip-17499

rgw: need to 'open_object_section' before dump stats in 'RGWGetUsage_…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: need to 'open_object_section' before dump stats in 'RGWGetUsage_ObjStore_S3::send_response()' function

Fixes: http://tracker.ceph.com/issues/17499
Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

Merge pull request #10820 from yanghonggang/master

rgw: fix get object instance  returned NoSuchKey error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10726 from weiqiaomiao/wqm-wip-remove-ret

rgw: remove the field ret from class RGWPutLC

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: make BufferSpace smaller

Saves 24 bytes!

BufferSpace  104 -> 80 bytes
SharedBlob   216 -> 192 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pack SharedBlob

224 -> 216 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: pack Blob members

272 -> 264 bytes

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: a few more types for sizeof

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6501 from dwj192/check-index-multipart

rgw:bucket check remove _multipart_ prefix

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10558 from cbodley/wip-rgw-mdlog-history

rgw: store oldest mdlog period in rados

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: optimize intrusive sets for size.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

os/bluestore: fix has_aios -> has_pending_aios

This is only called from safe callers (with exclusive access),
so we don't need a lock.

It also should only return true for *pending* aios--not aios
we've already submitted!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: fix aio_submit race

Behave if pending is empty.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11312 from liewegas/wip-bluestore-assign-blob

os/bluestore: fix crash in decode_some()

rbd: recognize lock_on_read option

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #10372 from cbodley/wip-rgw-data-log-trim

rgw multisite: trim data logs as peer zones catch up

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: fix crash in decode_some()

shared_blob must be non-null when we call Extent::assign_blob().

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11295 from liewegas/wip-bluestore-cache

os/bluestore: make cache settings process-wide

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11291 from trociny/wip-qa-mirror-asok

qa/workunits/rbd: check status also in pool dir after asok commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11278 from gaurav36/wip-obsolete-rgw-admin-period-prepare

rgw: obsolete 'radosgw-admin period prepare' command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

unittest_bluestore_types: show sizeof for key types

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11307 from jcsp/wip-mgr-compile

mgr: squash compiler warnings

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: use Extent::assign_blob

More explicit, less error-prone.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11299 from liewegas/wip-mgr-fix

mgr: implement con reset handling

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11308 from liewegas/wip-mgr-valgrind

mgr/MgrMap: initialize all fields

Reviewed-by: John Spray <john.spray@redhat.com>

os/bluestore: only split blobs on min_alloc_size boundaries

Otherwise we can throw off the allocator.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw multisite: obsolete 'radosgw-admin period prepare' command

All that the 'radosgw-admin period prepare' command does is create an
empty period and write it to rados. When we added the staging period
(see 'period get --staging'), uncommitted changes were written to that
instead - so the 'period prepare' command became obsolete. It should be
removed!

Fixes: http://tracker.ceph.com/issues/17387
Reported-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

os/bluestore: prune deallocated blob tails

This saves us space: no invalid extent, and no unused csum_data.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: try to split blobs instead of spanning them

If a blob crosses a shard boundary, try to split it into pieces
instead of making it spanning.  This is only possible under
certain conditions (uncompressed, no in-flight writes, no unused
space, shard boundary aligned with chunk boundary).

Generally speaking this will reduce the number of blobs that end
up spanning and reduce the onode key size.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrMap: initialize all fields

Fixes: http://tracker.ceph.com/issues/17492
Signed-off-by: Sage Weil <sage@redhat.com>

mgr: squash compiler warnings

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11254 from xiexingguo/xxg-wip-fs-0929

mds/MDCache: kill a comipler warning

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11306 from trociny/fixup-notify_image_updated

librbd: memory leak in MirroringWatcher::notify_image_updated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11303 from ceph/wip-rbd-clarify-size

doc: clarify rbd size units

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: memory leak in MirroringWatcher::notify_image_updated

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11257 from trociny/wip-17355-fixup

journal: ensure in-flight ops are complete destroying journaler

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11259 from b-ranto/wip-selinux-tmp-files

selinux: Allow ceph to manage tmp files

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: clarify rbd size units

It wasn't clear from the man page that --size defaults to M.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11218 from ceph/wip-getuid

Client: pass "UserPerm" struct everywhere for security checks

Reviewed-by: Jeff Layton <jlayton@redhat.com>

rgw: fix versioned object IO error

    When accessing a copied destination object, its source object's instance ID
    information is needed, however it's missing now in the destination object's
    manifest.

    In order to fix this problem, we can record source object's version_id/instance
    into dest object's manifest(a new filed 'tail_instance' is added). When creating
    a new object(not copy), 'tail_instance' should be equal to its instance value.
    When copy/get a object, 'tail_instance' should always be used to get the right
    tail objects.

Fixes: http://tracker.ceph.com/issues/17111
Signed-off-by: Yang Honggang <joseph.yang@xtaotech.com>

os/bluestore: turn bluestore_onode_cache_size 16k -> 4k

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unnecessary assert

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument cache blob and extent counts

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: attach blobs to cache

Signed-off-by: Sage Weil <sage@redhat.com>

do_cmake.sh: set .ceph_port to random port

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add perf counters for cache size

Onodes, buffers, buffer bytes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix typo in write_penalty_read_ops perf counter

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix cache tunables to be process-wide

Divide by the shard count so that the user isn't totally
confused.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11300 from liewegas/wip-bluestore-onode-add

os/bluestore: fix onode cache addition race

os/bluestore: fix onode cache addition race

Two threads may try to add the same onode to the cache.
This is rare, but allowed (in the case of the meta
collection).  If that happens, one of them will just
back off and use the winning onode ref.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: implement con reset handling

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11298 from liewegas/wip-mgr-fix

mgr/MgrClient: fix ms_handle_reset
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/MgrClient: fix ms_handle_reset

Return false because we don't handle the reset.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11292 from rzarzynski/yj-wip-swift-build_token-cleanup

rgw/swift: remove redundant assignment operation

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #11019 from liewegas/wip-autotools-must-die

autotools: remove a few other remaining traces

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11294 from trociny/fixup-qa-rbd_mirror-metadata_set

qa/workunits/rbd: set image-meta on primary image and wait it is replicated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11288 from stiopaa1/osd_replicatedpg_moveCOSDOnAppliedClass

osd/ReplicatedPG:move C_OSD_OnApplied class to cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11289 from stiopaa1/mds_sessionmap_moveStatementUnderAssertPushPv

mds/SessionMap.h: change statement to assertion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits/rbd: set image-meta on primary image and wait it is replicated

After recently added image metadata replication it is not possible any
more to update it on non-primary image.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>