osd: do not try to boot until we've seen the first osdmap

This is actually not a change in behavior *except* we see this message
instead of

2017-06-15 18:05:29.760938 24a7f700 -1 osd.1 0 osdmap SORTBITWISE OSDMap flag is NOT set; please set it

because the OSDMap() epoch 0 does not have SORTBITWISE.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15672 from liewegas/wip-bluestore-cache-ratios

os/bluestore: handle rounding error in cache ratios

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #15718 from ztczll/zl

os/bluestore: remove unused variables

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15730 from liewegas/wip-make-check

ceph-object-corpus: mark MMDSSlaveRequest incompat change

ceph-object-corpus: mark MMDSSlaveRequest incompat change

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15251 from joscollin/wip-cleanup-redundant-headers-4

common: Remove redundant includes - 4

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15310 from liewegas/wip-qa-scrubs

qa/tasks/ceph: raise exceptions if scrubbing fails or cannot proceed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15657 from xiexingguo/wip-reshard-allocation

os/bluestore: fix potential access violation

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15706 from tchaikov/wip-20115

ceph-rest-api: be more tolerant on network failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15714 from liewegas/wip-20323

messages/MOSDPing: initialize MOSDPing padding

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15561 from zy751713126/add-create-rbd-default-info

rbd: add default note info to size (create and resize)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15611 from tchaikov/wip-20175

test/librbd: decouple ceph_test_librbd_api from libceph-common

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore:remove invalid statement

Signed-off-by: zhanglei <243290414@qq.com>

messages/MOSDPing: initialize MOSDPing padding

This memory must be initialized or else valgrind will be very unhappy.

Avoid the cost of zeroing (or even allocating) the buffer for normal
padding values by (re)using a static zero buffer.

Fixes: http://tracker.ceph.com/issues/20323
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15622 from joscollin/wip-CID1412576-divide-by-zero

mon: DIVIDE_BY_ZERO in PGMapDigest::dump_pool_stats_full()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15596 from smithfarm/wip-copying

build/ops: rpm: package COPYING, move sample ceph.conf to ceph-common

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15668 from Liuchang0812/wip-fix-mgr-dashboard

mgr/dashboard: fix type error in get_rate function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15650 from gmayyyha/filestore-zfs

os/filestore: zfs add get_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15680 from xiexingguo/wip-bluestore-20170614

os/bluestore: cleanup bluestore_types

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15704 from tchaikov/wip-mgr-raise-in-send-command

mgr: raise python exception on failure in send_command()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15654 from lixiaoy1/lisa_bid

bluestore: wrap blob id when it reaches maximum value of int16_t

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15666 from xiexingguo/wip-fix-shard-finishers

os/bluestore: initialize finishers properly

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15675 from gregsfortytwo/wip-bad-command

mon,mgr: fix "ceph osd df", add some tools to find untested commands

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15687 from xiexingguo/wip-fix-decode-oi

osd/OSD: bump up current version; conditionally encoding manifest into oi

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15696 from liewegas/wip-19947

osd/PrimaryLogPG: fix oi reset during trim_object

Merge pull request #15407 from david-z/wip-fix-mds-balancer

mds/MDBalancer: remove useless check_targets and hit_targets logic from MDS balancer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15095 from gregsfortytwo/wip-doc-mark-event

doc: describe mark_events logging available via the OSD's OpTracker

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15698 from Wilhelmshaven/remove_unused_func

rgw: remove unused func in rgw_file.h

Merge pull request #15707 from tchaikov/wip-20239

doc: document the setup of restful and dashboard plugins

qa/tasks/thrashosds: wait before wait_for_recovery

Make sure OSDs are up *and* they have flushed their PG stats before
waiting for recovery to ensure that we do not see a stale 'clean' state.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash/worklaods/radosbench: use less disk

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic-luminous/scrub_test: skip final scrub

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15125 from ukernel/wip-multimds-misc2

mds: miscellaneous multimds fixes part2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15440 from ukernel/wip-20170

mds: multimds flock fixes

Reviewed-by: John Spray <john.spray@redhat.com>

qa/suites/rados/basic/repair_test: skip final scrub

OSD is deliberately damaged; scrub will fail.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: raise exception if scrubs time out

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: raise an exception if pgs are not clean

If this happens the preceding test should have cleaned
up (e.g., ceph.healthy:).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: try a bit longer

I just saw a test fail that was still waiting for
scrubs to complete.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15308 from jcsp/wip-19706

mon: don't kill MDSs unless some beacons are getting through

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15411 from jcsp/wip-fs-suite

qa: misc cephfs test improvements

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15708 from markhpc/wip-bluestore-shard_target

os/bluestore: target_bytes should scale with meta/data ratios.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15082 from joke-lee/fix-website-redirect-request-to-master

rgw-multisite: fix the problem of rgw website configure request not redirect to metadata master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15617 from joke-lee/fix_put_bucket_policy_when_set_again

rgw: fix RGWPutBucketPolicy error when set BucketPolicy again without delete pre set Policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/PrimaryLogPG: fix oi reset during trim_object

We partially fixed this in cd1271b4324431d16d3868a9f3b93cbefeeee057
but cleared the wrong oi (the clone's and not the
snap/head object's).

While we're here, drop the various bits where we clear
state bits; these are about to get clobbered when we reset
oi anyway.

Fixes: http://tracker.ceph.com/issues/19947
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15631 from xiexingguo/wip-bluestore-20170612

os/bluestore: stop calculating bound if we must reshard; narrow shard combination condition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15636 from tsg-/isal-ec-avx512

erasure-code: update ec_isa version + add missing AVX512 ISA-L sources

Merge pull request #15639 from liewegas/wip-20167

osd/PG: make non-empty PastIntervals non-fatal

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15641 from jecluis/wip-mon-deprecate-osd-create

mon: mark `osd create` as deprecated

Reviewed-by: Sage Weil <sage@redhat.com>

os/Bluestore: target_bytes should scale with meta/data ratios.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

build/ops: rpm: package COPYING, move conf example to ceph-common

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: drop sample.fetch_config

smithfarm: @liewegas Is src/fetch_config still needed?
liewegas: i don't think so

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: document the setup of restful and dashboard plugins

Fixes: http://tracker.ceph.com/issues/20239
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-rest-api: be more tolerant on network failure

* set timeout for json_command()
* retry on timeout

Fixes: http://tracker.ceph.com/issues/20115
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: add sanity check when appending pextents of blob

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: dedup is_allocated() and is_unallocated() of bluestore_blob_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15646 from tchaikov/wip-20258

pybind/mgr/restful: use list to pass hooks to create a `Pecan` instance

Reviewed-by: Boris Ranto <branto@redhat.com>

mon/OSDMonitor: more assert()

this helps with silencing the clang analyzer also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: raise python exception on failure in send_command()

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: return 400 on bad request

we should return 500 on that case. it's the client's fault not feeding us
with expected requests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15694 from cbodley/wip-qa-rgw-thrash-encryption

qa/rgw: add encryption config for s3tests under thrash

Merge pull request #15684 from badone/wip-refactor-of-osdmap-methods

osd/OSDMap: Change *pg_to_* to return void

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15692 from joscollin/wip-test-mon-Wreorder-warning

test/mon: silence warnings from -Wreorder

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: remove unused func in rgw_file.h

Signed-off-by: lihongjie <lihongjie@cmss.chinamobile.com>

Revert "qa/suites/rados/rest/mgr-restful: test on centos"

This reverts commit be88220965c37f2f36c61368f465a197ac76bed3.
we can workaround the limit of old requests by using the backward
compatible way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: set headers for requests

i think it's a bug in httplib shipped in old distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: use list to pass hooks to create a `Pecan` instance

pecan 0.3.2 introduced the new way to pass the hooks to construct a
Pecan instance, but the "callable" builder for hooks is not compatible
with pecan < 0.3.2. and ubuntu trusty ships pecan 0.3.0, so we need to
do this the old way: pass a list of hooks instead of a callable.

see https://github.com/pecan/pecan/blame/0.3.2/pecan/core.py and
https://github.com/pecan/pecan/commit/fb2cf166aaf5b5af461879ec5b7a46806e8aee45

Fixes: http://tracker.ceph.com/issues/20258
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/mon: silence warnings from -Wreorder

The following warning appears during make:

ceph/src/test/mon/test-mon-msg.cc: In constructor ‘MonMsgTest::MonMsgTest()’:
ceph/src/test/mon/test-mon-msg.cc:222:12: warning: ‘MonMsgTest::reply_msg’ will be initialized after [-Wreorder]
   Message *reply_msg;
            ^~~~~~~~~
ceph/src/test/mon/test-mon-msg.cc:229:16: warning:   base ‘MonClientHelper’ [-Wreorder]
     lock("lock") { }
                ^
ceph/src/test/mon/test-mon-msg.cc:226:3: warning:   when initialized here [-Wreorder]
   MonMsgTest() :

Signed-off-by: Jos Collin <jcollin@redhat.com>

os/bluestore: kill dead fall_back_to_per_au() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD: bump up current version of encoding; conditionally encoding manifest into oi

These versions shall match the corresponding encoding versions.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa: test 'osd df' in cephtool/test.sh

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

qa: add a check_commands.sh script which looks for commands with no tests

This isn't run automatically by anything yet. Note that it's also a best-effort
thing; passing doesn't guarantee there are tests. It can be pretty easily fooled
if the command is a common word which shows up in specifying other things,
for instance.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: mgr: enable "osd df" on the manager

Fixes: http://tracker.ceph.com/issues/20256
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

move the OSDUtilizationDumper code into OSDMap

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: inherit PGMonStatService from the PGMapStatService

Adventures in multiple inheritance!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: mgr: move 'osd df' handling to manager

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: move most PGMapStatService into PGMap; rename PGMon's to PGMonStatService

Most of this is independent of the PGMonitor, so move it into
PGMap and strip out those few bits. It'll come in handy shortly
when I move "ceph osd df" into the mgr.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: move creating_pgs and reweight_by_utilization into new MonPGStatService

Use that throughout the monitor code, so that we can use the
PGStatService elsewhere.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

rgw-multisite: fix the problem of rgw website configure request not redirect to metadata master

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

doc: describe mark_events logging available via the OSD's OpTracker

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

qa/rgw: add encryption config for s3tests under thrash

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15381 from xiexingguo/wip-noout-osdlist

mon/OSDMonitor: batch OSDs nodown/noout support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

bluestore: wrap blob id when it reaches maximum value of int16_t

Blob id exceeds maximum value, this patch is to prevent it.
Fixes: http://tracker.ceph.com/issues/19555
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

Merge pull request #15535 from gregsfortytwo/wip-jumbo-ping

osd: heartbeat with packets large enough to require working jumbo frames

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15645 from yuyuyu101/wip-20230

msg/async: fix deleted_conn is out of sync with conns

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Alexey Sheplyakov <asheplyakov@mirantis.com>

Merge pull request #15376 from tchaikov/wip-remove-boost-submodule

cmake: build boost as an external project

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15269 from theanalyst/rgw/env-fix

rgw: make RGWEnv return a const ref. to its map

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15690 from jcsp/wip-mgr-doc

doc: freshen mgr docs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: do not link libmon with mon_common_objs

ceph-common includes them already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move common/util.cc into ceph-common

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include Monitor.cc in osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: #include <atomic> in headers where atomic<> is used

so these headers are self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: include boost before default system directory

so ceph will prefer the local boost installation to the one in system if
any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build boost as an external project

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15607 from joke-lee/fix_bucket_policy_IfExists

rgw: fix rgw bucket policy IfExists position

Reviewed By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15686 from mikulely/drop-header-ended

rgw: drop unused var header_ended

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr: update for mgr_beacon_period change

This setting got renamed.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: flesh out introductory text

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: fix auth keys instruction

This was out of date.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/install: reference mgr manual setup

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15688 from rzarzynski/wip-rgw-fix-auth-for-clang

rgw: don't use strlen in constexprs to not brake Clang builds.
Reviewed-by : Orit Wasserman <owasserm@redhat.com>