qa/workunits: reduce amount of client IO for rbd-mirror stress

Journal IO is faster now, resulting in OSDs quickly filling up
under the stress test case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: potential race allocating tag during shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ignore attempts to request lock if already lock owner

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: clean up object recorder closed/overflow callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: delay moving overflowed buffers until quiesced

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: use reverse iterator search to find flush record

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: avoid holding lock while sending journal append

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10223 from mikulely/doc-s3-static-website-support

doc: update s3 static webiste feature support status

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #11325 from yehudasa/wip-17499

rgw: need to 'open_object_section' before dump stats in 'RGWGetUsage_…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: need to 'open_object_section' before dump stats in 'RGWGetUsage_ObjStore_S3::send_response()' function

Fixes: http://tracker.ceph.com/issues/17499
Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

Merge pull request #10820 from yanghonggang/master

rgw: fix get object instance  returned NoSuchKey error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10726 from weiqiaomiao/wqm-wip-remove-ret

rgw: remove the field ret from class RGWPutLC

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6501 from dwj192/check-index-multipart

rgw:bucket check remove _multipart_ prefix

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10558 from cbodley/wip-rgw-mdlog-history

rgw: store oldest mdlog period in rados

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11312 from liewegas/wip-bluestore-assign-blob

os/bluestore: fix crash in decode_some()

Merge pull request #10372 from cbodley/wip-rgw-data-log-trim

rgw multisite: trim data logs as peer zones catch up

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: fix crash in decode_some()

shared_blob must be non-null when we call Extent::assign_blob().

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11295 from liewegas/wip-bluestore-cache

os/bluestore: make cache settings process-wide

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11291 from trociny/wip-qa-mirror-asok

qa/workunits/rbd: check status also in pool dir after asok commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11278 from gaurav36/wip-obsolete-rgw-admin-period-prepare

rgw: obsolete 'radosgw-admin period prepare' command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

unittest_bluestore_types: show sizeof for key types

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11307 from jcsp/wip-mgr-compile

mgr: squash compiler warnings

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: use Extent::assign_blob

More explicit, less error-prone.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11299 from liewegas/wip-mgr-fix

mgr: implement con reset handling

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11308 from liewegas/wip-mgr-valgrind

mgr/MgrMap: initialize all fields

Reviewed-by: John Spray <john.spray@redhat.com>

rgw multisite: obsolete 'radosgw-admin period prepare' command

All that the 'radosgw-admin period prepare' command does is create an
empty period and write it to rados. When we added the staging period
(see 'period get --staging'), uncommitted changes were written to that
instead - so the 'period prepare' command became obsolete. It should be
removed!

Fixes: http://tracker.ceph.com/issues/17387
Reported-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

mgr/MgrMap: initialize all fields

Fixes: http://tracker.ceph.com/issues/17492
Signed-off-by: Sage Weil <sage@redhat.com>

mgr: squash compiler warnings

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11254 from xiexingguo/xxg-wip-fs-0929

mds/MDCache: kill a comipler warning

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11306 from trociny/fixup-notify_image_updated

librbd: memory leak in MirroringWatcher::notify_image_updated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11303 from ceph/wip-rbd-clarify-size

doc: clarify rbd size units

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: memory leak in MirroringWatcher::notify_image_updated

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11257 from trociny/wip-17355-fixup

journal: ensure in-flight ops are complete destroying journaler

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11259 from b-ranto/wip-selinux-tmp-files

selinux: Allow ceph to manage tmp files

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: clarify rbd size units

It wasn't clear from the man page that --size defaults to M.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11218 from ceph/wip-getuid

Client: pass "UserPerm" struct everywhere for security checks

Reviewed-by: Jeff Layton <jlayton@redhat.com>

rgw: fix versioned object IO error

    When accessing a copied destination object, its source object's instance ID
    information is needed, however it's missing now in the destination object's
    manifest.

    In order to fix this problem, we can record source object's version_id/instance
    into dest object's manifest(a new filed 'tail_instance' is added). When creating
    a new object(not copy), 'tail_instance' should be equal to its instance value.
    When copy/get a object, 'tail_instance' should always be used to get the right
    tail objects.

Fixes: http://tracker.ceph.com/issues/17111
Signed-off-by: Yang Honggang <joseph.yang@xtaotech.com>

os/bluestore: turn bluestore_onode_cache_size 16k -> 4k

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unnecessary assert

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument cache blob and extent counts

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: attach blobs to cache

Signed-off-by: Sage Weil <sage@redhat.com>

do_cmake.sh: set .ceph_port to random port

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add perf counters for cache size

Onodes, buffers, buffer bytes.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix typo in write_penalty_read_ops perf counter

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix cache tunables to be process-wide

Divide by the shard count so that the user isn't totally
confused.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11300 from liewegas/wip-bluestore-onode-add

os/bluestore: fix onode cache addition race

os/bluestore: fix onode cache addition race

Two threads may try to add the same onode to the cache.
This is rare, but allowed (in the case of the meta
collection).  If that happens, one of them will just
back off and use the winning onode ref.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: implement con reset handling

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11298 from liewegas/wip-mgr-fix

mgr/MgrClient: fix ms_handle_reset
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/MgrClient: fix ms_handle_reset

Return false because we don't handle the reset.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11292 from rzarzynski/yj-wip-swift-build_token-cleanup

rgw/swift: remove redundant assignment operation

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #11019 from liewegas/wip-autotools-must-die

autotools: remove a few other remaining traces

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11294 from trociny/fixup-qa-rbd_mirror-metadata_set

qa/workunits/rbd: set image-meta on primary image and wait it is replicated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11288 from stiopaa1/osd_replicatedpg_moveCOSDOnAppliedClass

osd/ReplicatedPG:move C_OSD_OnApplied class to cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11289 from stiopaa1/mds_sessionmap_moveStatementUnderAssertPushPv

mds/SessionMap.h: change statement to assertion

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits/rbd: set image-meta on primary image and wait it is replicated

After recently added image metadata replication it is not possible any
more to update it on non-primary image.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

run-make-check.sh: use cmake exclusively

..and drop the separate cmake script

Signed-off-by: Sage Weil <sage@redhat.com>

make-debs.sh: use make-dist

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/swift: remove redundant assignment operation

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #11086 from bassamtabbara/wip-ec-simd-runtime-detection

erasure-code: Runtime detection of SIMD for jerasure and shec

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: check status also in pool dir after asok commands

wait_for_image_replay_stopped returns not when the state is stopped,
but when the state is not replaying. So a race was possible when an
asok command was running when the previos stop command was still in
progress, leading to unexpected results.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

mds/SessionMap.h: change statement to assertion

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

osd/ReplicatedPG:move C_OSD_OnApplied class to cc

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

journal: ensure in-flight ops are complete destroying journaler

Fixes: http://tracker.ceph.com/issues/17446
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: complete action only after notification completed

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11285 from liewegas/wip-mgr-rpm

ceph.spec: fix ceph-mgr version requirement

Reviewed-by: Dan Mick <dmick@redhat.com>

client: in UserPerm, correctly use delete[] and don't leak gids on copy

gids is an array. We also need to clean it up when deep_copy()ing, which
requires initializing our member fields in the copy-constructor.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

ceph.spec: fix ceph-mgr version requirement

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11282 from liewegas/wip-bluestore-gc-fix

os/bluestore: fix gc when blob extends past eof

os/bluestore: fix gc when blob extends past eof

This can happen if we aren't referencing the bob.  Don't try to
read what isn't logically referenced.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11279 from liewegas/wip-bluestore-gc

os/bluestore: compaction fixes

os/bluestore: fix compaction left-side length

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: more concise SharedBlob print

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cosmetic gc cleanups

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11271 from stiopaa1/osd_repicatedpg_addOverride

osd/ReplicatedPG.h: add override to virtual funct

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11239 from liewegas/wip-bluestore-max-blob

os/bluestore: add max blob size; fix compressed min blob size logic

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

os/bluestore: set min compressed blob size floor at 2x min_alloc_size

Otherwise there's not point in compressing--we won't be able to
allocation a smaller space to store it.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: introduce bluestore_max_blob_size

This is enforced for compressed and non-compressed blobs alike.
We keep the separate compressoin min and max tunables since these
may be adjusted on a per-pool basis as well.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: adjust compression blob sizes down

These were way too big.  64KB is enough to get good compression,
and 4MB is way too big.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix blob depth encoding

1) Fix the flag constnat
2) Redefine the meaning to be more intuitive.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11232 from Roushan81/wip-bluestore-blob-gc

os/bluestore: garbage collect partially overlapped blobs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11263 from liewegas/wip-bluestore-reshard-slop

os/bluestore: sloppy reshard boundaries to avoid spanning blobs

Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #11275 from xiexingguo/xxg-wip-make-audit-buffer-option

os/bluestore: drop redundant call of get_blob

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: drop redundant call of get_blob

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11268 from liewegas/wip-bluefs-aio-leak

os/bluestore/BlueFS: release completed aios

Merge pull request #11005 from jazeltq/add-sparse-read

rbd-mirror: Add sparse read for sync image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11168 from trociny/wip-16212

rbd-mirror: replicate image metadata settings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11274 from xiexingguo/xxg-wip-bluestore-0930

os/bluestore: still more cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11136 from jcsp/wip-16016

mds: populate DamageTable from scrub and log more quietly

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #11270 from ceph/wip-17443

rgw: set correct instance on the object
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10328 from jcsp/wip-ceph-mgr

Create ceph-mgr

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: avoid comparing object key and name twice

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use id of shard_id_t for encoding

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill struct DeferredCsum

Which is dead, and kill some other dead lines too.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/ReplicatedPG.h: add override to virtual funct

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: add garbage collection

Signed-off-by: Roushan Ali <roushan.ali@sandisk.com>

Merge pull request #11267 from liewegas/wip-bluestore-ref-map

os/bluestore/bluestore_types: uint64_t for ref_map

rgw: set correct instance on the object

Fixes: http://tracker.ceph.com/issues/17443
This was broken by commit bc840afafdfe5e528e5c1b711e71420ac3cb5a67

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore/BlueFS: be less complainy about fallback allocations

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objecstore: use 1TB block device

Big enough to get into 32-bit LBAs.

Signed-off-by: Sage Weil <sage@redhat.com>