mon/PGMap: slightly better debugging around pgmap updates

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15813 from tchaikov/wip-cmake

cmake: do not add dependencies to INTERFACE library on cmake < 3.3

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #15060 from shashalu/fix-slo-dlo-range

rgw: fix SLO/DLO range requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15791 from liewegas/wip-20360

Revert "qa: do not restrict valgrind runs to centos"

cmake: do not add dependencies to INTERFACE library on cmake < 3.3

otherwise we will have

add_dependencies Cannot add target-level dependencies to INTERFACE
library

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13683 from liewegas/wip-crush-rulesets

crush: detect and (usually) fix ruleset != rule id

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15789 from linuxbox2/wip-nfs-doc

RGW NFS: add nfs.rst to doc/radosgw
(override because make check fail [missing selinux policy help file] unrelated)

RGW NFS: add nfs.rst to doc/radosgw

This document contains sections by John Wilkins <jowilkin@redhat.com>.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15255 from david-z/wip-mgr-fsstatus

pybind/mgr/fsstatus: use mds_mem.dn as dentry counter

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15621 from mingshuaiwang/master

mds: mds perf item 'l_mdl_expos' always behind journaler

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15662 from renhwztetecs/renhw-wip-client-trim_cache

client: call the lru_remove() twice,when trim cache

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15715 from jcsp/wip-jt-rmapply

tools/cephfs: remove `apply` mode of cephfs-journal-tool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15779 from renhwztetecs/renhw-wip-client-dentry

client: fix Dentry::dump

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: fix SLO/DLO range requests

Sending any SLO/DLO range requests will retrun InvalidRange error. SLO/DLO should call range_to_ofs after got object size instead of read_op.prepare() to deal with negative offsets.

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #15728 from majianpeng/bluefs-remove-memcopy-for-continue-memory

os/bluestore/BlueFS: Rebuild memcopy for bufferlist::page_aligned_app…

Reviewed-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: test page_aligned_appender

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: replace claim_append with claim_append_piecewise to reduce memory copy.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/buffer: add function  bufferlist::claim_append_piecewise(list&  bl).

This only useful for bl is bufferlist::page_aligned_appender. Using
this function can remove memcopy for continue ptrs.
Because page_aligned_appender::flush will split a ptr into two or
more ptrs. For this case, rebuild_aligned_size_and_memory can't handle,
it  will rebuild.

For example
a=bl.get_page_aligned_appender(1);
a.append(3K)
a.flush();
t.claim_append(bl);
a.append(1K);
a.flush();
t.claim_append(bl);

dst.claim_append(t);
//3K and 1K ptr are continue in memory. But they are two ptrs..
dst.is_aligned_size_and_memory(4096,4096) is false.

We add new function claim_append_piecewise() to specially
handle this case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: replicated_ruleset -> replicated_rule

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa: do not restrict valgrind runs to centos"

This reverts commit 5923961465e1efd3cc6bc072fef293dda3d975d0.

See http://tracker.ceph.com/issues/20360

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15260 from shashalu/fix-secondary-zonegroup-cors

rgw: update bucket cors in secondary zonegroup should forward to master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15709 from jcsp/wip-dashboard-loglast

mgr/dashboard: load log lines on startup, split out audit log

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15750 from xiexingguo/wip-bluestore-20170619

os/bluestore: refactor BlueStore::_do_write;  kill dead ExtentMap::find_lextent() method

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15766 from liewegas/wip-20227-fix

os/bluestore: fix unsharing blob dirty_range args

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15014 from zhangsw/fix-rgw-versionIdMarker-bug

rgw: VersionIdMarker and NextVersionIdMarker should be returned when listing object versions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #14500 from zhangsw/fix-rgw-multipart-index-bug

rgw: fix the bug that part's index can't be removed after completing multipart upload

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15771 from fangyuxiangGL/cleanup_rgw_admin

rgw: cleanup unused codes in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15768 from ukernel/wip-objecter-inflight-ops

osdc/Objecter: fix inflight_ops update

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15737 from wjwithagen/wip-wjw-test_rgw_string

rgw/rgw_string.h: FreeBSD would like errno.h included

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15782 from cbodley/wip-rgw-vstart-s3tests

vstart: add rgw configuration needed to pass all s3tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

vstart: add rgw configuration needed to pass all s3tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15751 from xiexingguo/wip-add-state-name

src/common/ceph_string: stringify new osd states

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14703 from zhangsw/feature-lifecycle-deletemarker

rgw: add support for delete marker expiration in s3 lifecycle.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15540 from joscollin/wip-cleanup-test-commented-code-2

test/osd: Removed Commented Code - 2

Merge pull request #15770 from tchaikov/wip-test-test_mon_crush

qa/workunits/cephtool/test.sh: only include last line for epoch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15733 from wjwithagen/wip-wjw-freebsd-ceph-disk-service

ceph-disk: command invocation needs all fields separate

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/osd: Removed Commented Code - 2

Removed commented code found in test/osd/.

Fixes: http://tracker.ceph.com/issues/20207 (Partially)
Signed-off-by: Jos Collin <jcollin@redhat.com>

ceph-disk: command invocation needs all fields separate

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15592 from david-z/wip-fix-dump-cache

mds: explicitly output error msg for dump cache asok command

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/cephtool/test.sh: only include last line for epoch

/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1606: test_mon_crush:  epoch='2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh:1607: test_mon_crush:  '[' '2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12' -gt 1 ']'
/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/qa/workunits/cephtool/test.sh: line 1607: [: 2017-06-20 04:44:52.862459 ffffad4d0200 -1 asok(0xffffa8000b10) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: The UNIX domain socket path /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/src/test/td/t-7202/out/client.admin.48876.asok is too long! The maximum length on this system is 107
12: integer expression expected

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: cleanup unused codes in rgw_admin.cc

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #15742 from wjwithagen/wip-wjw-freebsd-host-errno-3

common/freebsd_errno.cc: fixed again a stupid typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: fix inflight_ops update

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #15651 from zealoussnow/wip-0613

config_opt: use bool instead of int for the default value of filestore_debug_omap_check

Merge pull request #15676 from Linbing/wip-mon-delete-unused-config

mon: delete unused config opts of mon_sync_fs_threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15697 from gregsfortytwo/wip-mgr-eaccss-20296

mgr: print a more helpful error message for when users lack mgr ceph caps

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15744 from badone/wip-CID-1412579

ceph-dencoder: Silence coverity CID 1412579

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15637 from liewegas/wip-point-to-point

qa/upgrade/jewel-x/point-to-point: add a mgr during final upgrade

os/bluestore: fix unsharing blob dirty_range args

Fixes: http://tracker.ceph.com/issues/20227
Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: add_simple_ruleset[_at] -> add_simple_rule[_at]

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: some cosmetic cleanup (ruleset -> rule)

Signed-off-by: Sage Weil <sage@redhat.com>

crush: simplify osd_pool_default_crush_rule config

Make an incompat change here with a release note since
this only affects pool creation, a rare event, and folks
who have customized their configs (also rare).

Keep it simple: a config sets the default rule, or else we pick
the first TYPE_REPLICATED pool in the crush map.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: rename functions and variables, ruleset -> rule

Signed-off-by: Sage Weil <sage@redhat.com>

mon: scrub 'ruleset' from user-visible output

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove 'crush_ruleset' property

The 'crush_rule' name has been present since before hammer; users should
use that instead

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pg_pool_t: rename crush_ruleset -> crush_rule

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: rename 'osd pool create' argument field name

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: optimize find_rule when rules are uniform

On finalize, set a flag indicating whether our rules are uniform.  If so,
find_rule() is trivial--it just verifies the rule exists.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: issue health warning if there are multirule rulesets

The user will need to manually fix the CRUSH map if this happens.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: automatically convert crushmap if ruleset != rule id

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: prevent injecting CRUSH maps with ruleset != rule id

Signed-off-by: Sage Weil <sage@redhat.com>

crush: add rule/set checks, conversion helpers

Add chekcs to see whether the CRUSH map has legacy ruleset != rule id
or (worse yet) multiple rules sharing the same ruleset.

Add a method to convert the former case, and error out on the latter.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15365 from ShiqiCooperation/newshiqi

test/unittest_bluefs: When fsync ret is less than 0, fsync can not be…

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15533 from liewegas/wip-crush-version

mon,osd: add crush_version to OSDMap, and allow crush map updates to gate on crush_version

Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15685 from wjwithagen/wip-wjw-freebsd-ENODATA-2

core/common: Fix ENODATA for FreeBSD with compat.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15710 from liewegas/wip-20273

osd: bail from _committed_osd_maps inside osd_lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw_string.h: FreeBSD would like errno.h included

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15333 from mikulely/validate-bucket-location

rgw/multisite: validate bucket location during bucket creation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15633 from cloudbau/reduce-rgw-copy-loglevel

rgw: do not log debug output at level 0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15598 from aclamk/encryption_refactor

radosgw, crypto: simplified code in handle_data functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15599 from theanalyst/rgw/fix/bl-cstr

rgw: don't specify a length when converting bl -> string

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15756 from rzarzynski/wip-rgw-cleanup-swift-errmapping

rgw: clean-up error mapping in Swift's authentication strategy.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15677 from zhangsw/fix-rgw-lifecycle-bug

rgw: lifecycle thread shouldn't process the bucket which has been deleted

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #15682 from mikulely/rgw-putobj-params

rgw: abort early when s->length empty during putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15747 from liewegas/wip-scrub-retry

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

qa/suites/rados/singleton-nomsgr/multi-backfill-reject: clean up

Set pool size back to 2 so we don't have to have backfill
complete (despite rejection probability) in order to get back to
healthy.  This way we scrub on cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15712 from liewegas/wip-19882

osd/ECTransaction: only read partial stripes when below *original* object size

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton-nomsgr/full-tiering: unset quota at end

If we leave the quota set, the proxied ops will block
indefinitely, which will block scrubbing on the cache tier pgs
indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/resolve_stuck_peering: start osd at end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15679 from guihecheng/rgw_file-fix-setattr

rgw_file: fix assert upon setattr on bucket

Merge pull request #15660 from fangyuxiangGL/meta_sync_crash

rgw: meta sync thread crash at RGWMetaSyncShardCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/cephtool/test.sh: silence dev warnings

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: reissue scrub requests in loop

The scrub commands are not reliable: if the OSD doesn't happen to
be connected at the time the command is issued it may not get
delivered.  Re-request scrubs for each PG that has not yet been
scrubbed so that we don't wait forever when the original request
is dropped.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: tolerate down osd at initial scrub time

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15738 from cbodley/wip-rgw-string-constexpr

rgw: fix constexpr for string_size in clang

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15544 from taodd/master

mds:  Client syncfs is slow (waits for next MDS tick)

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15746 from liewegas/wip-20227

os/bluestore: better debug output on unsharing blobs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

client: signal MDS to flush log when doing a syncfs

Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Client: signal MDS to flush log when doing a syncfs
Fixes: http://tracker.ceph.com/issues/20129
Signed-off-by: dongdong tao <tdd21151186@gmail.com>

Merge pull request #15745 from zealoussnow/wip-0617

.gitignore: exclude rpm files

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15481 from jcsp/wip-20178

client: avoid returning negative space available

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: kill dead ExtentMap::find_lextent() method

Which currently has no consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: refactor BlueStore::_do_write

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15614 from jcsp/wip-jt-help

tools/cephfs: fix cephfs-journal-tool --help

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

rgw: clean-up error mapping in Swift's authentication strategy.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #15749 from Adirl/wip

msg/async/rdma: handle buffers after close msg

Reviewed-by: Haomai Wang <haomai@xsky.com>

src/common/ceph_string: stringify new osd states

In https://github.com/ceph/ceph/pull/15725 I introduce
two more osd states.

This patch pulls them in.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: meta sync thread crash at RGWMetaSyncShardCR

Fixes: http://tracker.ceph.com/issues/20251
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com

rgw: add support for delete marker expiration in s3 lifecycle.

Fixes: http://tracker.ceph.com/issues/19730
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

rgw: update bucket cors in secondary zonegroup should forward to master

pr#14082 is incomplete, cors should be redirect to master

Fixes: http://tracker.ceph.com/issues/16888
Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>