qa/objectstore/bluestore: lower full thresholds

On NVMe we can write really fast... fast enough that we may fill up before
we realize we are nearly full.  It's a problem here only because our
devices are so small that the OSDs fill up proportionally quickly.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/powercycle/osd/tasks/radosbench: consume less space

Fixes: http://tracker.ceph.com/issues/20302
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15783 from liewegas/wip-20358

os/bluestore: move sharedblob to new collection in same shard

Merge pull request #15785 from liewegas/wip-bluestore-fsck

qa/objectstore/bluestore*: fsck on mount

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15734 from dillaman/wip-librbd-image-remove

librbd: fix issues with image removal state machine

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15727 from ovh/bp-runtime-cfg-of-hb-inflator

osd, messages/MOSDPing: bunch of fixes related to ping inflation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15723 from badone/wip-ceph-mon-startup-error-logging

mon: Log errors at startup

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15784 from liewegas/wip-thrash-cleanup

qa/tasks/ceph_manager: get osds all in after thrashing

Merge pull request #15787 from liewegas/wip-19943-fix

ceph-objectstore-tool: do not populate snapmapper with missing clones

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15813 from tchaikov/wip-cmake

cmake: do not add dependencies to INTERFACE library on cmake < 3.3

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #15060 from shashalu/fix-slo-dlo-range

rgw: fix SLO/DLO range requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15791 from liewegas/wip-20360

Revert "qa: do not restrict valgrind runs to centos"

cmake: do not add dependencies to INTERFACE library on cmake < 3.3

otherwise we will have

add_dependencies Cannot add target-level dependencies to INTERFACE
library

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13683 from liewegas/wip-crush-rulesets

crush: detect and (usually) fix ruleset != rule id

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15789 from linuxbox2/wip-nfs-doc

RGW NFS: add nfs.rst to doc/radosgw
(override because make check fail [missing selinux policy help file] unrelated)

RGW NFS: add nfs.rst to doc/radosgw

This document contains sections by John Wilkins <jowilkin@redhat.com>.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15255 from david-z/wip-mgr-fsstatus

pybind/mgr/fsstatus: use mds_mem.dn as dentry counter

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15621 from mingshuaiwang/master

mds: mds perf item 'l_mdl_expos' always behind journaler

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15662 from renhwztetecs/renhw-wip-client-trim_cache

client: call the lru_remove() twice,when trim cache

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15715 from jcsp/wip-jt-rmapply

tools/cephfs: remove `apply` mode of cephfs-journal-tool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15779 from renhwztetecs/renhw-wip-client-dentry

client: fix Dentry::dump

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: fix SLO/DLO range requests

Sending any SLO/DLO range requests will retrun InvalidRange error. SLO/DLO should call range_to_ofs after got object size instead of read_op.prepare() to deal with negative offsets.

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #15728 from majianpeng/bluefs-remove-memcopy-for-continue-memory

os/bluestore/BlueFS: Rebuild memcopy for bufferlist::page_aligned_app…

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: test page_aligned_appender

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: replace claim_append with claim_append_piecewise to reduce memory copy.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/buffer: add function  bufferlist::claim_append_piecewise(list&  bl).

This only useful for bl is bufferlist::page_aligned_appender. Using
this function can remove memcopy for continue ptrs.
Because page_aligned_appender::flush will split a ptr into two or
more ptrs. For this case, rebuild_aligned_size_and_memory can't handle,
it  will rebuild.

For example
a=bl.get_page_aligned_appender(1);
a.append(3K)
a.flush();
t.claim_append(bl);
a.append(1K);
a.flush();
t.claim_append(bl);

dst.claim_append(t);
//3K and 1K ptr are continue in memory. But they are two ptrs..
dst.is_aligned_size_and_memory(4096,4096) is false.

We add new function claim_append_piecewise() to specially
handle this case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: replicated_ruleset -> replicated_rule

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa: do not restrict valgrind runs to centos"

This reverts commit 5923961465e1efd3cc6bc072fef293dda3d975d0.

See http://tracker.ceph.com/issues/20360

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15260 from shashalu/fix-secondary-zonegroup-cors

rgw: update bucket cors in secondary zonegroup should forward to master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15709 from jcsp/wip-dashboard-loglast

mgr/dashboard: load log lines on startup, split out audit log

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15750 from xiexingguo/wip-bluestore-20170619

os/bluestore: refactor BlueStore::_do_write;  kill dead ExtentMap::find_lextent() method

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15766 from liewegas/wip-20227-fix

os/bluestore: fix unsharing blob dirty_range args

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15014 from zhangsw/fix-rgw-versionIdMarker-bug

rgw: VersionIdMarker and NextVersionIdMarker should be returned when listing object versions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14500 from zhangsw/fix-rgw-multipart-index-bug

rgw: fix the bug that part's index can't be removed after completing multipart upload

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

os/bluestore: debug OmapIteratorImpl

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: debug keys inserted, removed, and iterated

Hunting bug http://tracker.ceph.com/issues/19943

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: do not populate snapmapper with missing clones

Cache pools may have clones represented in SnapSet that are not stored
locally.  This is fine as long as ceph-objectstore-tool doesn't go and
add them when it shouldn't.

Fixes: http://tracker.ceph.com/issues/19943
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15771 from fangyuxiangGL/cleanup_rgw_admin

rgw: cleanup unused codes in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15768 from ukernel/wip-objecter-inflight-ops

osdc/Objecter: fix inflight_ops update

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15737 from wjwithagen/wip-wjw-test_rgw_string

rgw/rgw_string.h: FreeBSD would like errno.h included

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15782 from cbodley/wip-rgw-vstart-s3tests

vstart: add rgw configuration needed to pass all s3tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

vstart: add rgw configuration needed to pass all s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/objectstore/bluestore*: fsck on mount

This will catch more bugs!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15751 from xiexingguo/wip-add-state-name

src/common/ceph_string: stringify new osd states

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: get osds all in after thrashing

Otherwise we might end up with some PGs remapped, which means they won't
get scrubbed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move sharedblob to new collection in same shard

We have to move the SharedBlob to the new collection even if
the new collection is part of the same cache shard.  It's only the buffers
that need to be conditioned on the shard.

Fixes: http://tracker.ceph.com/issues/20358
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14703 from zhangsw/feature-lifecycle-deletemarker

rgw: add support for delete marker expiration in s3 lifecycle.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15540 from joscollin/wip-cleanup-test-commented-code-2

test/osd: Removed Commented Code - 2

Merge pull request #15770 from tchaikov/wip-test-test_mon_crush

qa/workunits/cephtool/test.sh: only include last line for epoch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15733 from wjwithagen/wip-wjw-freebsd-ceph-disk-service

ceph-disk: command invocation needs all fields separate

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/osd: Removed Commented Code - 2

Removed commented code found in test/osd/.

Fixes: http://tracker.ceph.com/issues/20207 (Partially)
Signed-off-by: Jos Collin <jcollin@redhat.com>

ceph-disk: command invocation needs all fields separate

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: removed synchronous watcher check from remove image state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: clean up remove request state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove state machine should ignore group not supported errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15592 from david-z/wip-fix-dump-cache

mds: explicitly output error msg for dump cache asok command

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/cephtool/test.sh: only include last line for epoch

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1606: test_mon_crush:  epoch='2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1607: test_mon_crush:  '[' '2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12' -gt 1 ']'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh: line 1607: [: 2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12: integer expression expected

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: cleanup unused codes in rgw_admin.cc

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #15742 from wjwithagen/wip-wjw-freebsd-host-errno-3

common/freebsd_errno.cc: fixed again a stupid typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: fix inflight_ops update

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15651 from zealoussnow/wip-0613

config_opt: use bool instead of int for the default value of filestore_debug_omap_check

Merge pull request #15676 from Linbing/wip-mon-delete-unused-config

mon: delete unused config opts of mon_sync_fs_threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15697 from gregsfortytwo/wip-mgr-eaccss-20296

mgr: print a more helpful error message for when users lack mgr ceph caps

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15744 from badone/wip-CID-1412579

ceph-dencoder: Silence coverity CID 1412579

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15637 from liewegas/wip-point-to-point

qa/upgrade/jewel-x/point-to-point: add a mgr during final upgrade

os/bluestore: fix unsharing blob dirty_range args

Fixes: http://tracker.ceph.com/issues/20227
Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: add_simple_ruleset[_at] -> add_simple_rule[_at]

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: some cosmetic cleanup (ruleset -> rule)

Signed-off-by: Sage Weil <sage@redhat.com>

crush: simplify osd_pool_default_crush_rule config

Make an incompat change here with a release note since
this only affects pool creation, a rare event, and folks
who have customized their configs (also rare).

Keep it simple: a config sets the default rule, or else we pick
the first TYPE_REPLICATED pool in the crush map.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: rename functions and variables, ruleset -> rule

Signed-off-by: Sage Weil <sage@redhat.com>

mon: scrub 'ruleset' from user-visible output

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove 'crush_ruleset' property

The 'crush_rule' name has been present since before hammer; users should
use that instead

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pg_pool_t: rename crush_ruleset -> crush_rule

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: rename 'osd pool create' argument field name

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: optimize find_rule when rules are uniform

On finalize, set a flag indicating whether our rules are uniform.  If so,
find_rule() is trivial--it just verifies the rule exists.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: issue health warning if there are multirule rulesets

The user will need to manually fix the CRUSH map if this happens.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: automatically convert crushmap if ruleset != rule id

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: prevent injecting CRUSH maps with ruleset != rule id

Signed-off-by: Sage Weil <sage@redhat.com>

crush: add rule/set checks, conversion helpers

Add chekcs to see whether the CRUSH map has legacy ruleset != rule id
or (worse yet) multiple rules sharing the same ruleset.

Add a method to convert the former case, and error out on the latter.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15365 from ShiqiCooperation/newshiqi

test/unittest_bluefs: When fsync ret is less than 0, fsync can not be…

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15533 from liewegas/wip-crush-version

mon,osd: add crush_version to OSDMap, and allow crush map updates to gate on crush_version

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15685 from wjwithagen/wip-wjw-freebsd-ENODATA-2

core/common: Fix ENODATA for FreeBSD with compat.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15710 from liewegas/wip-20273

osd: bail from _committed_osd_maps inside osd_lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw_string.h: FreeBSD would like errno.h included

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15333 from mikulely/validate-bucket-location

rgw/multisite: validate bucket location during bucket creation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15633 from cloudbau/reduce-rgw-copy-loglevel

rgw: do not log debug output at level 0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15598 from aclamk/encryption_refactor

radosgw, crypto: simplified code in handle_data functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15599 from theanalyst/rgw/fix/bl-cstr

rgw: don't specify a length when converting bl -> string

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15756 from rzarzynski/wip-rgw-cleanup-swift-errmapping

rgw: clean-up error mapping in Swift's authentication strategy.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15677 from zhangsw/fix-rgw-lifecycle-bug

rgw: lifecycle thread shouldn't process the bucket which has been deleted

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15682 from mikulely/rgw-putobj-params

rgw: abort early when s->length empty during putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15747 from liewegas/wip-scrub-retry

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

qa/suites/rados/singleton-nomsgr/multi-backfill-reject: clean up

Set pool size back to 2 so we don't have to have backfill
complete (despite rejection probability) in order to get back to
healthy.  This way we scrub on cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15712 from liewegas/wip-19882

osd/ECTransaction: only read partial stripes when below *original* object size

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton-nomsgr/full-tiering: unset quota at end

If we leave the quota set, the proxied ops will block
indefinitely, which will block scrubbing on the cache tier pgs
indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/resolve_stuck_peering: start osd at end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15679 from guihecheng/rgw_file-fix-setattr

rgw_file: fix assert upon setattr on bucket

Merge pull request #15660 from fangyuxiangGL/meta_sync_crash

rgw: meta sync thread crash at RGWMetaSyncShardCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/cephtool/test.sh: silence dev warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

The scrub commands are not reliable: if the OSD doesn't happen to
be connected at the time the command is issued it may not get
delivered.  Re-request scrubs for each PG that has not yet been
scrubbed so that we don't wait forever when the original request
is dropped.

Signed-off-by: Sage Weil <sage@redhat.com>