os/bluestore: kill dead ExtentMap::find_lextent() method

Which currently has no consumers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: refactor BlueStore::_do_write

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15725 from xiexingguo/wip-noupin

mon/OSDMonitor: batch noup/noin osds support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15748 from Liuchang0812/wip-fix-release-note

doc/release-note: update release-note

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15736 from cbodley/wip-20297

rgw: forward RGWPutBucketPolicy to meta master
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15132 from cbodley/wip-doc-rgw-remove-agent

doc/rgw: remove Federated Configuration, clean up multisite

doc/release-note: update release-note

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mon/OSDMonitor: batch noup/noin osds support

This is a follow-up change of https://github.com/ceph/ceph/pull/15381.
This patch also simplifies the original code logic a bit.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15658 from xiexingguo/wip-fix-reshard

os/bluestore: fix possible out of order shard(offset == 0); add sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15716 from jcsp/wip-mgr-fsmap

mgr: fix MgrStandby eating messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15732 from liewegas/wip-osd-boot-message

osd: do not try to boot until we've seen the first osdmap

Merge pull request #15740 from liewegas/wip-pgmap-debug

mon,mgr: print pgmap reports to debug (not cluster) log

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15389 from gregsfortytwo/wip-18126

qa: do not restrict valgrind runs to centos

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15618 from liupan1111/wip-fix-comment

tools/rados: fixed typo in help information

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15720 from ShiqiCooperation/rbdshiqi

test/unittest_librbd: remove unused variables

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15741 from wjwithagen/wip-wjw-freebsd-host-errno-2

common/freebsd_errno.cc: fix missing ;

common/freebsd_errno.cc: fix missing ;

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/DaemonServer: print PGMapDigest summary on send

Useful for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15495 from wjwithagen/wip-wjw-freebsd-host-errno

core: introduce (and fix) code to pass errno to other OSes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15717 from liewegas/wip-20326

qa/tasks/ceph.py: tolerate active+clean+something

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15721 from majianpeng/bluestore-nodevice-nosymlink

os/bluestore/BlueStore: no device no symlink.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15559 from dzafman/wip-20186

osd: Check snapset for validity when selecting authoritative shard

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15552 from TsaiJin/wip-queue-scrub-with-higher-priority

osd/OSD.h: requeue the scrub job with higher priority to shorten the blocking time of related requests

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mon/MgrStatMonitor: print PGMapDigest summary on receipt

Useful for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15722 from dmick/master

test/osd/osd-dup.sh: use wait_for_clean

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15735 from theanalyst/rgw-uninit-atomic

rgw_rados: initialize cur_shard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15524 from cbodley/wip-rgw-sync-debug-logging

rgw: raise debug level of meta sync logging

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15711 from cbodley/wip-rgw-swift-default-auth-error

rgw: fix swift default auth error after auth strategy refactoring

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: add multisite test_set_bucket_policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: forward RGWPutBucketPolicy to meta master

Fixes: http://tracker.ceph.com/issues/20297
Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore/BlueStore: no device no symlink.

If bluestore_block_db_create or bluestore_block_db_create is false
and we use non-exist blockdevice as db/wal. ceph-osd still work and it
use block as db/wal. This bug introduced by 316810d7e74.

If stat(symlink) error and symlink still exist, this is bug.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rgw_rados: initialize cur_shard

valgrind reported an uninitialized value on use of cur_shard at a few
places, so explicitly declaring this to 0 as we are incrementing this
later

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc/rgw: update pool names, document namespaces

Fixes: http://tracker.ceph.com/issues/19504
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: remove Upgrade to Older Versions of Jewel page

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: remove Federated Configuration page

Fixes: http://tracker.ceph.com/issues/18082
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: clean up multisite config options

remove region stuff, radosgw-agent stuff, and zone/pool stuff that is
duplicated in the multisite page

moved sync log config options into multisite section, added description
for rgw_run_sync_thread

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15678 from cbodley/wip-rgw-string-join

rgw: add variadic string join for s3 signature generation

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: remove debug log statements in RGWMetaSyncShardCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: raise debug level of meta sync logging

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15036 from joke-lee/master

rgw-multisite: fix the problem of rgw website configure 'RedirectAllRequestsTo' failed to sync to slave zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15731 from cbodley/wip-rgw-rest-client-signature

rgw: fix rest client's order of args in get_v2_signature
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

osd: do not try to boot until we've seen the first osdmap

This is actually not a change in behavior *except* we see this message
instead of

2017-06-15 18:05:29.760938 24a7f700 -1 osd.1 0 osdmap SORTBITWISE OSDMap flag is NOT set; please set it

because the OSDMap() epoch 0 does not have SORTBITWISE.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: fix error mapping after auth strategy refactoring

the changes in bd81c216d0ef36a66c27180f13430c9702393fb6 inadvertently
changed swift's default auth error from EPERM to ACCESS. hacky fix to
change it back

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix rest client's order of args in get_v2_signature

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15672 from liewegas/wip-bluestore-cache-ratios

os/bluestore: handle rounding error in cache ratios

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #15718 from ztczll/zl

os/bluestore: remove unused variables

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15730 from liewegas/wip-make-check

ceph-object-corpus: mark MMDSSlaveRequest incompat change

ceph-object-corpus: mark MMDSSlaveRequest incompat change

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15251 from joscollin/wip-cleanup-redundant-headers-4

common: Remove redundant includes - 4

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15310 from liewegas/wip-qa-scrubs

qa/tasks/ceph: raise exceptions if scrubbing fails or cannot proceed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15657 from xiexingguo/wip-reshard-allocation

os/bluestore: fix potential access violation

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15706 from tchaikov/wip-20115

ceph-rest-api: be more tolerant on network failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15714 from liewegas/wip-20323

messages/MOSDPing: initialize MOSDPing padding

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15561 from zy751713126/add-create-rbd-default-info

rbd: add default note info to size (create and resize)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15611 from tchaikov/wip-20175

test/librbd: decouple ceph_test_librbd_api from libceph-common

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test/osd/osd-dup.sh: use wait_for_clean

Rather than a onetime 'ceph -s' test, which might be too early
on a loaded machine, use wait_for_clean; that's its job

Signed-off-by: Dan Mick <dan.mick@redhat.com>

os/bluestore:remove invalid statement

Signed-off-by: zhanglei <243290414@qq.com>

test/unittest_librbd: Useless variables

Signed-off-by: shiqi <1454927420@qq.com>

qa/tasks/ceph.py: tolerate active+clean+something

where something is, say, snaptrim.  or maybe scrubbing.
or whatever.

Signed-off-by: Sage Weil <sage@redhat.com>

messages/MOSDPing: initialize MOSDPing padding

This memory must be initialized or else valgrind will be very unhappy.

Avoid the cost of zeroing (or even allocating) the buffer for normal
padding values by (re)using a static zero buffer.

Fixes: http://tracker.ceph.com/issues/20323
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15622 from joscollin/wip-CID1412576-divide-by-zero

mon: DIVIDE_BY_ZERO in PGMapDigest::dump_pool_stats_full()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15596 from smithfarm/wip-copying

build/ops: rpm: package COPYING, move sample ceph.conf to ceph-common

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15668 from Liuchang0812/wip-fix-mgr-dashboard

mgr/dashboard: fix type error in get_rate function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15650 from gmayyyha/filestore-zfs

os/filestore: zfs add get_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15680 from xiexingguo/wip-bluestore-20170614

os/bluestore: cleanup bluestore_types

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15704 from tchaikov/wip-mgr-raise-in-send-command

mgr: raise python exception on failure in send_command()

Reviewed-by: Sage Weil <sage@redhat.com>

mgr: fix MgrStandby eating messages

It was failing to pass up the Mgr::ms_dispatch
return value, so messages were being consumed
even if Mgr was returning false.  Mgr returns
false to enable Client to see FSMap, so the
symptom was the Client not working.

While we're here, refactor switch into if, because
there were only two paths in the switch and we
don't expect to be adding more.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15654 from lixiaoy1/lisa_bid

bluestore: wrap blob id when it reaches maximum value of int16_t

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15666 from xiexingguo/wip-fix-shard-finishers

os/bluestore: initialize finishers properly

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15675 from gregsfortytwo/wip-bad-command

mon,mgr: fix "ceph osd df", add some tools to find untested commands

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15687 from xiexingguo/wip-fix-decode-oi

osd/OSD: bump up current version; conditionally encoding manifest into oi

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15696 from liewegas/wip-19947

osd/PrimaryLogPG: fix oi reset during trim_object

Merge pull request #15407 from david-z/wip-fix-mds-balancer

mds/MDBalancer: remove useless check_targets and hit_targets logic from MDS balancer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15095 from gregsfortytwo/wip-doc-mark-event

doc: describe mark_events logging available via the OSD's OpTracker

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15698 from Wilhelmshaven/remove_unused_func

rgw: remove unused func in rgw_file.h

Merge pull request #15707 from tchaikov/wip-20239

doc: document the setup of restful and dashboard plugins

qa/tasks/thrashosds: wait before wait_for_recovery

Make sure OSDs are up *and* they have flushed their PG stats before
waiting for recovery to ensure that we do not see a stale 'clean' state.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash/worklaods/radosbench: use less disk

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic-luminous/scrub_test: skip final scrub

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15125 from ukernel/wip-multimds-misc2

mds: miscellaneous multimds fixes part2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15440 from ukernel/wip-20170

mds: multimds flock fixes

Reviewed-by: John Spray <john.spray@redhat.com>

qa/suites/rados/basic/repair_test: skip final scrub

OSD is deliberately damaged; scrub will fail.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: raise exception if scrubs time out

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: raise an exception if pgs are not clean

If this happens the preceding test should have cleaned
up (e.g., ceph.healthy:).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: try a bit longer

I just saw a test fail that was still waiting for
scrubs to complete.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15308 from jcsp/wip-19706

mon: don't kill MDSs unless some beacons are getting through

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15411 from jcsp/wip-fs-suite

qa: misc cephfs test improvements

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15708 from markhpc/wip-bluestore-shard_target

os/bluestore: target_bytes should scale with meta/data ratios.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15082 from joke-lee/fix-website-redirect-request-to-master

rgw-multisite: fix the problem of rgw website configure request not redirect to metadata master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15617 from joke-lee/fix_put_bucket_policy_when_set_again

rgw: fix RGWPutBucketPolicy error when set BucketPolicy again without delete pre set Policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/PrimaryLogPG: fix oi reset during trim_object

We partially fixed this in cd1271b4324431d16d3868a9f3b93cbefeeee057
but cleared the wrong oi (the clone's and not the
snap/head object's).

While we're here, drop the various bits where we clear
state bits; these are about to get clobbered when we reset
oi anyway.

Fixes: http://tracker.ceph.com/issues/19947
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15631 from xiexingguo/wip-bluestore-20170612

os/bluestore: stop calculating bound if we must reshard; narrow shard combination condition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15636 from tsg-/isal-ec-avx512

erasure-code: update ec_isa version + add missing AVX512 ISA-L sources

Merge pull request #15639 from liewegas/wip-20167

osd/PG: make non-empty PastIntervals non-fatal

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15641 from jecluis/wip-mon-deprecate-osd-create

mon: mark `osd create` as deprecated

Reviewed-by: Sage Weil <sage@redhat.com>

os/Bluestore: target_bytes should scale with meta/data ratios.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

build/ops: rpm: package COPYING, move conf example to ceph-common

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: drop sample.fetch_config

smithfarm: @liewegas Is src/fetch_config still needed?
liewegas: i don't think so

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: document the setup of restful and dashboard plugins

Fixes: http://tracker.ceph.com/issues/20239
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-rest-api: be more tolerant on network failure

* set timeout for json_command()
* retry on timeout

Fixes: http://tracker.ceph.com/issues/20115
Signed-off-by: Kefu Chai <kchai@redhat.com>