qa/tasks/thrashosds: wait before wait_for_recovery

Make sure OSDs are up *and* they have flushed their PG stats before
waiting for recovery to ensure that we do not see a stale 'clean' state.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash/worklaods/radosbench: use less disk

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic-luminous/scrub_test: skip final scrub

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/basic/repair_test: skip final scrub

OSD is deliberately damaged; scrub will fail.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: raise exception if scrubs time out

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: raise an exception if pgs are not clean

If this happens the preceding test should have cleaned
up (e.g., ceph.healthy:).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: osd_scrub_pgs: try a bit longer

I just saw a test fail that was still waiting for
scrubs to complete.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15308 from jcsp/wip-19706

mon: don't kill MDSs unless some beacons are getting through

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15411 from jcsp/wip-fs-suite

qa: misc cephfs test improvements

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15708 from markhpc/wip-bluestore-shard_target

os/bluestore: target_bytes should scale with meta/data ratios.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15082 from joke-lee/fix-website-redirect-request-to-master

rgw-multisite: fix the problem of rgw website configure request not redirect to metadata master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15617 from joke-lee/fix_put_bucket_policy_when_set_again

rgw: fix RGWPutBucketPolicy error when set BucketPolicy again without delete pre set Policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15631 from xiexingguo/wip-bluestore-20170612

os/bluestore: stop calculating bound if we must reshard; narrow shard combination condition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15636 from tsg-/isal-ec-avx512

erasure-code: update ec_isa version + add missing AVX512 ISA-L sources

Merge pull request #15639 from liewegas/wip-20167

osd/PG: make non-empty PastIntervals non-fatal

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15641 from jecluis/wip-mon-deprecate-osd-create

mon: mark `osd create` as deprecated

Reviewed-by: Sage Weil <sage@redhat.com>

os/Bluestore: target_bytes should scale with meta/data ratios.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #15646 from tchaikov/wip-20258

pybind/mgr/restful: use list to pass hooks to create a `Pecan` instance

Reviewed-by: Boris Ranto <branto@redhat.com>

pybind/mgr/restful: return 400 on bad request

we should return 500 on that case. it's the client's fault not feeding us
with expected requests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15694 from cbodley/wip-qa-rgw-thrash-encryption

qa/rgw: add encryption config for s3tests under thrash

Merge pull request #15684 from badone/wip-refactor-of-osdmap-methods

osd/OSDMap: Change *pg_to_* to return void

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15692 from joscollin/wip-test-mon-Wreorder-warning

test/mon: silence warnings from -Wreorder

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "qa/suites/rados/rest/mgr-restful: test on centos"

This reverts commit be88220965c37f2f36c61368f465a197ac76bed3.
we can workaround the limit of old requests by using the backward
compatible way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: set headers for requests

i think it's a bug in httplib shipped in old distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: use list to pass hooks to create a `Pecan` instance

pecan 0.3.2 introduced the new way to pass the hooks to construct a
Pecan instance, but the "callable" builder for hooks is not compatible
with pecan < 0.3.2. and ubuntu trusty ships pecan 0.3.0, so we need to
do this the old way: pass a list of hooks instead of a callable.

see https://github.com/pecan/pecan/blame/0.3.2/pecan/core.py and
https://github.com/pecan/pecan/commit/fb2cf166aaf5b5af461879ec5b7a46806e8aee45

Fixes: http://tracker.ceph.com/issues/20258
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/mon: silence warnings from -Wreorder

The following warning appears during make:

ceph/src/test/mon/test-mon-msg.cc: In constructor ‘MonMsgTest::MonMsgTest()’:
ceph/src/test/mon/test-mon-msg.cc:222:12: warning: ‘MonMsgTest::reply_msg’ will be initialized after [-Wreorder]
   Message *reply_msg;
            ^~~~~~~~~
ceph/src/test/mon/test-mon-msg.cc:229:16: warning:   base ‘MonClientHelper’ [-Wreorder]
     lock("lock") { }
                ^
ceph/src/test/mon/test-mon-msg.cc:226:3: warning:   when initialized here [-Wreorder]
   MonMsgTest() :

Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw-multisite: fix the problem of rgw website configure request not redirect to metadata master

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

qa/rgw: add encryption config for s3tests under thrash

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15381 from xiexingguo/wip-noout-osdlist

mon/OSDMonitor: batch OSDs nodown/noout support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #15535 from gregsfortytwo/wip-jumbo-ping

osd: heartbeat with packets large enough to require working jumbo frames

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15645 from yuyuyu101/wip-20230

msg/async: fix deleted_conn is out of sync with conns

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Alexey Sheplyakov <asheplyakov@mirantis.com>

Merge pull request #15376 from tchaikov/wip-remove-boost-submodule

cmake: build boost as an external project

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15269 from theanalyst/rgw/env-fix

rgw: make RGWEnv return a const ref. to its map

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15690 from jcsp/wip-mgr-doc

doc: freshen mgr docs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: do not link libmon with mon_common_objs

ceph-common includes them already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move common/util.cc into ceph-common

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include Monitor.cc in osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: #include <atomic> in headers where atomic<> is used

so these headers are self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: include boost before default system directory

so ceph will prefer the local boost installation to the one in system if
any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build boost as an external project

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15607 from joke-lee/fix_bucket_policy_IfExists

rgw: fix rgw bucket policy IfExists position

Reviewed By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15686 from mikulely/drop-header-ended

rgw: drop unused var header_ended

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr: update for mgr_beacon_period change

This setting got renamed.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: flesh out introductory text

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: fix auth keys instruction

This was out of date.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/install: reference mgr manual setup

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15688 from rzarzynski/wip-rgw-fix-auth-for-clang

rgw: don't use strlen in constexprs to not brake Clang builds.
Reviewed-by : Orit Wasserman <owasserm@redhat.com>

rgw: make RGWEnv return a const ref. to its map

We already have a public method `set` for setting values in the RGW env
map, making the map as a read only. In addition:

- Added const to other  methods in `RGWEnv` which are getters
- `req_info::init_meta` also reused the same `iter` name for an internal
  find, changed the var name
- `add_grants_headers` now accepts an RGWEnv instead of the map
- use range based for loops wherever the code was changed
- req_info holds a ptr to a const RGWEnv, since we seem to only read the
  values from it

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15399 from smithfarm/wip-20127

build/ops: miscellaneous cleanups and fixes (run-make-check.sh, ceph.spec.in)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

rgw: don't use strlen in constexprs to not brake Clang builds.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

build/ops: rpm: one License line to rule them all

The SUSE and Fedora packaging guidelines specify that subpackages _may_ have
their own License line if it is necessary. (Hopefully it's not, because
maintaining one License line is much easier.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm, COPYING: drop GPL 2.0 with Autoconf Exception

This license is no longer used.

Fixes: http://tracker.ceph.com/issues/20091
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: Enable ppc64le and s390x builds in SLE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: ceph.spec.in: drop redundant defattrs

The defattr(-,root,root,-) has been the default for a long time.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: add which as a build dependency, unconditionally

Although "which" is only used in the tests, we need the dependency
to be picked up by install-deps.sh.

This is a stopgap measure until we can get rid of the dependency entirely by
replacing "which" with "type -P" in the tests.

Fixes: http://tracker.ceph.com/issues/20127
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: make sure which is installed in run-make-check.sh

Fixes: http://tracker.ceph.com/issues/20127
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15664 from zealoussnow/wip-fix-doc

doc: remove some non-existent and fix the default value according to …

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15681 from zealoussnow/wip-option-comment-error

common: fix typo in option of rados_mon_op_timeout's comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: drop unused var header_ended

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

osd: Change *pg_to_* to return void

Returning the number of items in the vector is not required and can be
calculated since the caller has access to the vector.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

common: fix option "rados_mon_op_timeout" comment spell error

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

cmake: use boost target instead of libary path while linking against them

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14885 from rzarzynski/wip-rgw-auth-rework-cont-3-awsv4

rgw: continuation of the auth rework -- AWSv4

osd: heartbeat with packets large enough to require working jumbo frames.

We get periodic reports that users somehow misconfigure one of their switches
so that it drops jumbo frames, yet the servers are still passing them along. In
that case, MOSDOp messages generally don't get through because they are much
larger than the 1500-byte non-jumbo limit, but the MOSDPing messages have kept
going (as they are very small and dispatched independently, even when the
server is willing to make jumbo frames). This means peer OSDs won't mark down
the ones behind the broken switch, despite all IO hanging.
Push the MOSDPing message size over the 1500-byte limit so that anybody in
this scenario will see the OSDs stuck behind a bad switch get marked down.

Fixes: http://tracker.ceph.com/issues/20087
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14344 from rzarzynski/wip-rgw-repect-negative-refacls

rgw: respect Swift's negative, HTTP referer-based ACL grants.

Merge pull request #15586 from adamemerson/wip-ex-ML

rgw: Remove spurious XML header for GetBucketPolicy

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: Remove spurious XML header for GetBucketPolicy

Fixes: http://tracker.ceph.com/issues/20247
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15564 from Jing-Scott/fix-formpost-auth

rgw: fix the signature mismatch of FormPost in swift API

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15581 from neha-ojha/wip-fix-objclass

objclass-sdk: use namespace ceph for bufferlist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15595 from ztczll/zl

os/bluestore/BlueFS: .slow should be compared with dirname

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15590 from Linbing/wip-osd-check-osdmap-twice

osd: the condition of last epoch <= superblock.newest_map epoch has been check twice

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15661 from tchaikov/wip-qa-upgrade-set-sortbitwise

qa/suites/upgrade: set "sortbitwise" for jewel clusters

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async: go to open new session when existing already closed

Fixes: http://tracker.ceph.com/issues/20230
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15613 from cbodley/wip-20244

rgw: only log metadata on metadata master zone

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15280 from ZVampirEM77/wip-bucket-policy-cleanup

rgw: fix get bucket policy s3 compatible issue

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15663 from zealoussnow/wip-fix-build

cmake: exclude *.css while generating ctags

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15655 from guihecheng/rgw_file-fix-mkdir

rgw_file: prevent conflict of mkdir between restarts

Merge pull request #15659 from lixiaoy1/fio_ceph

test/fio: Fix assert in set_cache_shards in bluestore fio

Fix assert in set_cache_shards in bluestore fio

The config osd_op_num_shards is changed so it needs to change in
fio_ceph_objectstore.cc.

Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

Merge pull request #15593 from chenfangxian/rbd-nbd-fix-unmap

rbd-nbd: ensure unmap returns error code

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Pan Liu <wanjun.lp@alibaba-inc.com>

doc: remove some non-existent and fix the default value according to source code

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

build: add css file to excludes while generate tags

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

rgw: fix the signature mismatch of FormPost in swift API

Fixes: http://tracker.ceph.com/issues/20220
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

qa/suites/upgrade: remove duplicated upgrade task

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade: set "sortbitwise" for jewel clusters

so ceph.healthy or wait-for-healthy won't be blocked by this warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw_file: prevent conflict of mkdir between restarts

Fixes: http://tracker.ceph.com/issues/20275
Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15648 from joscollin/wip-CID1412575-incorrect-expression

mon: Incorrect expression in PGMap::get_health()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15638 from liewegas/wip-19948

ceph_test_rados_api_*: wait for snap trim on ENOENT during cleanup

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: fix nbd do_unmap error handling

when nbd disconnect failed, ensure do_unmap return error logic.

Signed-off-by: chenfangxian <chenfangxian@cmss.chinamobile.com>
Signed-off-by: guojiannan <guojiannan@cmss.chinamobile.com>

Merge pull request #15519 from liewegas/wip-19595

osd/PrimaryLogPG: clear oi from trim_object()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon: Incorrect expression in PGMap::get_health()

Fixes:

CID 1412575:  Incorrect expression  (COPY_PASTE_ERROR)
ceph/src/mon/PGMap.cc: 2764 in PGMap::get_health()
"num_err" in "ss2 << num_err" looks like a copy-paste error.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15629 from Linbing/wip-msg-async-op-threads

msg/async: add assert of ms_async_op_threads > 0

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: fix accept_conn not remove entry in conns when lazy delete

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15426 from zhangsw/fix-rgw-gc-loglevel

rgw: lower some log's level in gc process.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15571 from ceph/wip-fix-fio-shard-hint

test/fio: fix lack of setting for Sequencer::shard_hint

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mon: mark `osd create` as deprecated

With the introduction of `osd new`, we are now deprecating `osd create`
with the intent of removing it in a future release.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

osd/PG: make non-empty PastIntervals non-fatal

Yes, we should not have past intervals in this case.  However, we don't
need to crash the OSD; the ERR in clog is sufficient to fail the teuthology
run, and users in production don't actually care.

See: http://tracker.ceph.com/issues/20167
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15603 from idryomov/wip-rados-df-typo

rados: fix typo in 'df' column name

Merge pull request #15615 from joscollin/wip-CID717210-Uninitialized-members

common: initialize _hash in LogEntryKey()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15627 from theanalyst/12.1.0/release-notes-2

doc: 12.1.0/release notes 2