Merge pull request #15622 from joscollin/wip-CID1412576-divide-by-zero

mon: DIVIDE_BY_ZERO in PGMapDigest::dump_pool_stats_full()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15596 from smithfarm/wip-copying

build/ops: rpm: package COPYING, move sample ceph.conf to ceph-common

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15668 from Liuchang0812/wip-fix-mgr-dashboard

mgr/dashboard: fix type error in get_rate function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15650 from gmayyyha/filestore-zfs

os/filestore: zfs add get_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15680 from xiexingguo/wip-bluestore-20170614

os/bluestore: cleanup bluestore_types

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15704 from tchaikov/wip-mgr-raise-in-send-command

mgr: raise python exception on failure in send_command()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15654 from lixiaoy1/lisa_bid

bluestore: wrap blob id when it reaches maximum value of int16_t

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15666 from xiexingguo/wip-fix-shard-finishers

os/bluestore: initialize finishers properly

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15675 from gregsfortytwo/wip-bad-command

mon,mgr: fix "ceph osd df", add some tools to find untested commands

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15687 from xiexingguo/wip-fix-decode-oi

osd/OSD: bump up current version; conditionally encoding manifest into oi

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15696 from liewegas/wip-19947

osd/PrimaryLogPG: fix oi reset during trim_object

Merge pull request #15407 from david-z/wip-fix-mds-balancer

mds/MDBalancer: remove useless check_targets and hit_targets logic from MDS balancer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15095 from gregsfortytwo/wip-doc-mark-event

doc: describe mark_events logging available via the OSD's OpTracker

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15698 from Wilhelmshaven/remove_unused_func

rgw: remove unused func in rgw_file.h

Merge pull request #15707 from tchaikov/wip-20239

doc: document the setup of restful and dashboard plugins

Merge pull request #15125 from ukernel/wip-multimds-misc2

mds: miscellaneous multimds fixes part2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15440 from ukernel/wip-20170

mds: multimds flock fixes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15308 from jcsp/wip-19706

mon: don't kill MDSs unless some beacons are getting through

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15411 from jcsp/wip-fs-suite

qa: misc cephfs test improvements

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15708 from markhpc/wip-bluestore-shard_target

os/bluestore: target_bytes should scale with meta/data ratios.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15082 from joke-lee/fix-website-redirect-request-to-master

rgw-multisite: fix the problem of rgw website configure request not redirect to metadata master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15617 from joke-lee/fix_put_bucket_policy_when_set_again

rgw: fix RGWPutBucketPolicy error when set BucketPolicy again without delete pre set Policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/PrimaryLogPG: fix oi reset during trim_object

We partially fixed this in cd1271b4324431d16d3868a9f3b93cbefeeee057
but cleared the wrong oi (the clone's and not the
snap/head object's).

While we're here, drop the various bits where we clear
state bits; these are about to get clobbered when we reset
oi anyway.

Fixes: http://tracker.ceph.com/issues/19947
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15631 from xiexingguo/wip-bluestore-20170612

os/bluestore: stop calculating bound if we must reshard; narrow shard combination condition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15636 from tsg-/isal-ec-avx512

erasure-code: update ec_isa version + add missing AVX512 ISA-L sources

Merge pull request #15639 from liewegas/wip-20167

osd/PG: make non-empty PastIntervals non-fatal

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15641 from jecluis/wip-mon-deprecate-osd-create

mon: mark `osd create` as deprecated

Reviewed-by: Sage Weil <sage@redhat.com>

os/Bluestore: target_bytes should scale with meta/data ratios.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

build/ops: rpm: package COPYING, move conf example to ceph-common

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: drop sample.fetch_config

smithfarm: @liewegas Is src/fetch_config still needed?
liewegas: i don't think so

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: document the setup of restful and dashboard plugins

Fixes: http://tracker.ceph.com/issues/20239
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: add sanity check when appending pextents of blob

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: dedup is_allocated() and is_unallocated() of bluestore_blob_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15646 from tchaikov/wip-20258

pybind/mgr/restful: use list to pass hooks to create a `Pecan` instance

Reviewed-by: Boris Ranto <branto@redhat.com>

mon/OSDMonitor: more assert()

this helps with silencing the clang analyzer also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: raise python exception on failure in send_command()

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: return 400 on bad request

we should return 500 on that case. it's the client's fault not feeding us
with expected requests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15694 from cbodley/wip-qa-rgw-thrash-encryption

qa/rgw: add encryption config for s3tests under thrash

Merge pull request #15684 from badone/wip-refactor-of-osdmap-methods

osd/OSDMap: Change *pg_to_* to return void

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15692 from joscollin/wip-test-mon-Wreorder-warning

test/mon: silence warnings from -Wreorder

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: remove unused func in rgw_file.h

Signed-off-by: lihongjie <lihongjie@cmss.chinamobile.com>

Revert "qa/suites/rados/rest/mgr-restful: test on centos"

This reverts commit be88220965c37f2f36c61368f465a197ac76bed3.
we can workaround the limit of old requests by using the backward
compatible way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: set headers for requests

i think it's a bug in httplib shipped in old distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: use list to pass hooks to create a `Pecan` instance

pecan 0.3.2 introduced the new way to pass the hooks to construct a
Pecan instance, but the "callable" builder for hooks is not compatible
with pecan < 0.3.2. and ubuntu trusty ships pecan 0.3.0, so we need to
do this the old way: pass a list of hooks instead of a callable.

see https://github.com/pecan/pecan/blame/0.3.2/pecan/core.py and
https://github.com/pecan/pecan/commit/fb2cf166aaf5b5af461879ec5b7a46806e8aee45

Fixes: http://tracker.ceph.com/issues/20258
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/mon: silence warnings from -Wreorder

The following warning appears during make:

ceph/src/test/mon/test-mon-msg.cc: In constructor ‘MonMsgTest::MonMsgTest()’:
ceph/src/test/mon/test-mon-msg.cc:222:12: warning: ‘MonMsgTest::reply_msg’ will be initialized after [-Wreorder]
   Message *reply_msg;
            ^~~~~~~~~
ceph/src/test/mon/test-mon-msg.cc:229:16: warning:   base ‘MonClientHelper’ [-Wreorder]
     lock("lock") { }
                ^
ceph/src/test/mon/test-mon-msg.cc:226:3: warning:   when initialized here [-Wreorder]
   MonMsgTest() :

Signed-off-by: Jos Collin <jcollin@redhat.com>

os/bluestore: kill dead fall_back_to_per_au() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSD: bump up current version of encoding; conditionally encoding manifest into oi

These versions shall match the corresponding encoding versions.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa: test 'osd df' in cephtool/test.sh

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

qa: add a check_commands.sh script which looks for commands with no tests

This isn't run automatically by anything yet. Note that it's also a best-effort
thing; passing doesn't guarantee there are tests. It can be pretty easily fooled
if the command is a common word which shows up in specifying other things,
for instance.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: mgr: enable "osd df" on the manager

Fixes: http://tracker.ceph.com/issues/20256
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

move the OSDUtilizationDumper code into OSDMap

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: inherit PGMonStatService from the PGMapStatService

Adventures in multiple inheritance!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: mgr: move 'osd df' handling to manager

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: move most PGMapStatService into PGMap; rename PGMon's to PGMonStatService

Most of this is independent of the PGMonitor, so move it into
PGMap and strip out those few bits. It'll come in handy shortly
when I move "ceph osd df" into the mgr.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: move creating_pgs and reweight_by_utilization into new MonPGStatService

Use that throughout the monitor code, so that we can use the
PGStatService elsewhere.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

rgw-multisite: fix the problem of rgw website configure request not redirect to metadata master

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

doc: describe mark_events logging available via the OSD's OpTracker

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

qa/rgw: add encryption config for s3tests under thrash

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15381 from xiexingguo/wip-noout-osdlist

mon/OSDMonitor: batch OSDs nodown/noout support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

bluestore: wrap blob id when it reaches maximum value of int16_t

Blob id exceeds maximum value, this patch is to prevent it.
Fixes: http://tracker.ceph.com/issues/19555
Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

Merge pull request #15535 from gregsfortytwo/wip-jumbo-ping

osd: heartbeat with packets large enough to require working jumbo frames

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15645 from yuyuyu101/wip-20230

msg/async: fix deleted_conn is out of sync with conns

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Alexey Sheplyakov <asheplyakov@mirantis.com>

Merge pull request #15376 from tchaikov/wip-remove-boost-submodule

cmake: build boost as an external project

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15269 from theanalyst/rgw/env-fix

rgw: make RGWEnv return a const ref. to its map

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15690 from jcsp/wip-mgr-doc

doc: freshen mgr docs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: do not link libmon with mon_common_objs

ceph-common includes them already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move common/util.cc into ceph-common

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include Monitor.cc in osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: #include <atomic> in headers where atomic<> is used

so these headers are self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: include boost before default system directory

so ceph will prefer the local boost installation to the one in system if
any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build boost as an external project

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15607 from joke-lee/fix_bucket_policy_IfExists

rgw: fix rgw bucket policy IfExists position

Reviewed By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15686 from mikulely/drop-header-ended

rgw: drop unused var header_ended

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr: update for mgr_beacon_period change

This setting got renamed.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: flesh out introductory text

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: fix auth keys instruction

This was out of date.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/install: reference mgr manual setup

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15688 from rzarzynski/wip-rgw-fix-auth-for-clang

rgw: don't use strlen in constexprs to not brake Clang builds.
Reviewed-by : Orit Wasserman <owasserm@redhat.com>

rgw: make RGWEnv return a const ref. to its map

We already have a public method `set` for setting values in the RGW env
map, making the map as a read only. In addition:

- Added const to other  methods in `RGWEnv` which are getters
- `req_info::init_meta` also reused the same `iter` name for an internal
  find, changed the var name
- `add_grants_headers` now accepts an RGWEnv instead of the map
- use range based for loops wherever the code was changed
- req_info holds a ptr to a const RGWEnv, since we seem to only read the
  values from it

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15399 from smithfarm/wip-20127

build/ops: miscellaneous cleanups and fixes (run-make-check.sh, ceph.spec.in)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

rgw: don't use strlen in constexprs to not brake Clang builds.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

build/ops: rpm: one License line to rule them all

The SUSE and Fedora packaging guidelines specify that subpackages _may_ have
their own License line if it is necessary. (Hopefully it's not, because
maintaining one License line is much easier.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm, COPYING: drop GPL 2.0 with Autoconf Exception

This license is no longer used.

Fixes: http://tracker.ceph.com/issues/20091
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: Enable ppc64le and s390x builds in SLE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: ceph.spec.in: drop redundant defattrs

The defattr(-,root,root,-) has been the default for a long time.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: add which as a build dependency, unconditionally

Although "which" is only used in the tests, we need the dependency
to be picked up by install-deps.sh.

This is a stopgap measure until we can get rid of the dependency entirely by
replacing "which" with "type -P" in the tests.

Fixes: http://tracker.ceph.com/issues/20127
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: make sure which is installed in run-make-check.sh

Fixes: http://tracker.ceph.com/issues/20127
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15664 from zealoussnow/wip-fix-doc

doc: remove some non-existent and fix the default value according to …

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15681 from zealoussnow/wip-option-comment-error

common: fix typo in option of rados_mon_op_timeout's comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: drop unused var header_ended

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

osd: Change *pg_to_* to return void

Returning the number of items in the vector is not required and can be
calculated since the caller has access to the vector.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

common: fix option "rados_mon_op_timeout" comment spell error

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

cmake: use boost target instead of libary path while linking against them

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14885 from rzarzynski/wip-rgw-auth-rework-cont-3-awsv4

rgw: continuation of the auth rework -- AWSv4

osd: heartbeat with packets large enough to require working jumbo frames.

We get periodic reports that users somehow misconfigure one of their switches
so that it drops jumbo frames, yet the servers are still passing them along. In
that case, MOSDOp messages generally don't get through because they are much
larger than the 1500-byte non-jumbo limit, but the MOSDPing messages have kept
going (as they are very small and dispatched independently, even when the
server is willing to make jumbo frames). This means peer OSDs won't mark down
the ones behind the broken switch, despite all IO hanging.
Push the MOSDPing message size over the 1500-byte limit so that anybody in
this scenario will see the OSDs stuck behind a bad switch get marked down.

Fixes: http://tracker.ceph.com/issues/20087
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14344 from rzarzynski/wip-rgw-repect-negative-refacls

rgw: respect Swift's negative, HTTP referer-based ACL grants.

Merge pull request #15586 from adamemerson/wip-ex-ML

rgw: Remove spurious XML header for GetBucketPolicy

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: Remove spurious XML header for GetBucketPolicy

Fixes: http://tracker.ceph.com/issues/20247
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15564 from Jing-Scott/fix-formpost-auth

rgw: fix the signature mismatch of FormPost in swift API

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15581 from neha-ojha/wip-fix-objclass

objclass-sdk: use namespace ceph for bufferlist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>