Merge pull request #15694 from cbodley/wip-qa-rgw-thrash-encryption

qa/rgw: add encryption config for s3tests under thrash

Merge pull request #15684 from badone/wip-refactor-of-osdmap-methods

osd/OSDMap: Change *pg_to_* to return void

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15692 from joscollin/wip-test-mon-Wreorder-warning

test/mon: silence warnings from -Wreorder

Reviewed By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/mon: silence warnings from -Wreorder

The following warning appears during make:

ceph/src/test/mon/test-mon-msg.cc: In constructor ‘MonMsgTest::MonMsgTest()’:
ceph/src/test/mon/test-mon-msg.cc:222:12: warning: ‘MonMsgTest::reply_msg’ will be initialized after [-Wreorder]
   Message *reply_msg;
            ^~~~~~~~~
ceph/src/test/mon/test-mon-msg.cc:229:16: warning:   base ‘MonClientHelper’ [-Wreorder]
     lock("lock") { }
                ^
ceph/src/test/mon/test-mon-msg.cc:226:3: warning:   when initialized here [-Wreorder]
   MonMsgTest() :

Signed-off-by: Jos Collin <jcollin@redhat.com>

qa/rgw: add encryption config for s3tests under thrash

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15381 from xiexingguo/wip-noout-osdlist

mon/OSDMonitor: batch OSDs nodown/noout support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Merge pull request #15535 from gregsfortytwo/wip-jumbo-ping

osd: heartbeat with packets large enough to require working jumbo frames

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15645 from yuyuyu101/wip-20230

msg/async: fix deleted_conn is out of sync with conns

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Alexey Sheplyakov <asheplyakov@mirantis.com>

Merge pull request #15376 from tchaikov/wip-remove-boost-submodule

cmake: build boost as an external project

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15269 from theanalyst/rgw/env-fix

rgw: make RGWEnv return a const ref. to its map

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15690 from jcsp/wip-mgr-doc

doc: freshen mgr docs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: do not link libmon with mon_common_objs

ceph-common includes them already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move common/util.cc into ceph-common

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include Monitor.cc in osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: #include <atomic> in headers where atomic<> is used

so these headers are self-contained.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: include boost before default system directory

so ceph will prefer the local boost installation to the one in system if
any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build boost as an external project

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15607 from joke-lee/fix_bucket_policy_IfExists

rgw: fix rgw bucket policy IfExists position

Reviewed By: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15686 from mikulely/drop-header-ended

rgw: drop unused var header_ended

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr: update for mgr_beacon_period change

This setting got renamed.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: flesh out introductory text

Signed-off-by: John Spray <john.spray@redhat.com>

doc/mgr: fix auth keys instruction

This was out of date.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/install: reference mgr manual setup

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15688 from rzarzynski/wip-rgw-fix-auth-for-clang

rgw: don't use strlen in constexprs to not brake Clang builds.
Reviewed-by : Orit Wasserman <owasserm@redhat.com>

rgw: make RGWEnv return a const ref. to its map

We already have a public method `set` for setting values in the RGW env
map, making the map as a read only. In addition:

- Added const to other  methods in `RGWEnv` which are getters
- `req_info::init_meta` also reused the same `iter` name for an internal
  find, changed the var name
- `add_grants_headers` now accepts an RGWEnv instead of the map
- use range based for loops wherever the code was changed
- req_info holds a ptr to a const RGWEnv, since we seem to only read the
  values from it

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15399 from smithfarm/wip-20127

build/ops: miscellaneous cleanups and fixes (run-make-check.sh, ceph.spec.in)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

rgw: don't use strlen in constexprs to not brake Clang builds.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

build/ops: rpm: one License line to rule them all

The SUSE and Fedora packaging guidelines specify that subpackages _may_ have
their own License line if it is necessary. (Hopefully it's not, because
maintaining one License line is much easier.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm, COPYING: drop GPL 2.0 with Autoconf Exception

This license is no longer used.

Fixes: http://tracker.ceph.com/issues/20091
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: Enable ppc64le and s390x builds in SLE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: ceph.spec.in: drop redundant defattrs

The defattr(-,root,root,-) has been the default for a long time.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: add which as a build dependency, unconditionally

Although "which" is only used in the tests, we need the dependency
to be picked up by install-deps.sh.

This is a stopgap measure until we can get rid of the dependency entirely by
replacing "which" with "type -P" in the tests.

Fixes: http://tracker.ceph.com/issues/20127
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: make sure which is installed in run-make-check.sh

Fixes: http://tracker.ceph.com/issues/20127
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15664 from zealoussnow/wip-fix-doc

doc: remove some non-existent and fix the default value according to …

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15681 from zealoussnow/wip-option-comment-error

common: fix typo in option of rados_mon_op_timeout's comment

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: drop unused var header_ended

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

osd: Change *pg_to_* to return void

Returning the number of items in the vector is not required and can be
calculated since the caller has access to the vector.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

common: fix option "rados_mon_op_timeout" comment spell error

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

cmake: use boost target instead of libary path while linking against them

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14885 from rzarzynski/wip-rgw-auth-rework-cont-3-awsv4

rgw: continuation of the auth rework -- AWSv4

osd: heartbeat with packets large enough to require working jumbo frames.

We get periodic reports that users somehow misconfigure one of their switches
so that it drops jumbo frames, yet the servers are still passing them along. In
that case, MOSDOp messages generally don't get through because they are much
larger than the 1500-byte non-jumbo limit, but the MOSDPing messages have kept
going (as they are very small and dispatched independently, even when the
server is willing to make jumbo frames). This means peer OSDs won't mark down
the ones behind the broken switch, despite all IO hanging.
Push the MOSDPing message size over the 1500-byte limit so that anybody in
this scenario will see the OSDs stuck behind a bad switch get marked down.

Fixes: http://tracker.ceph.com/issues/20087
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14344 from rzarzynski/wip-rgw-repect-negative-refacls

rgw: respect Swift's negative, HTTP referer-based ACL grants.

Merge pull request #15586 from adamemerson/wip-ex-ML

rgw: Remove spurious XML header for GetBucketPolicy

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: Remove spurious XML header for GetBucketPolicy

Fixes: http://tracker.ceph.com/issues/20247
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15564 from Jing-Scott/fix-formpost-auth

rgw: fix the signature mismatch of FormPost in swift API

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15581 from neha-ojha/wip-fix-objclass

objclass-sdk: use namespace ceph for bufferlist

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15595 from ztczll/zl

os/bluestore/BlueFS: .slow should be compared with dirname

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15590 from Linbing/wip-osd-check-osdmap-twice

osd: the condition of last epoch <= superblock.newest_map epoch has been check twice

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15661 from tchaikov/wip-qa-upgrade-set-sortbitwise

qa/suites/upgrade: set "sortbitwise" for jewel clusters

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async: go to open new session when existing already closed

Fixes: http://tracker.ceph.com/issues/20230
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15613 from cbodley/wip-20244

rgw: only log metadata on metadata master zone

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15280 from ZVampirEM77/wip-bucket-policy-cleanup

rgw: fix get bucket policy s3 compatible issue

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #15663 from zealoussnow/wip-fix-build

cmake: exclude *.css while generating ctags

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15655 from guihecheng/rgw_file-fix-mkdir

rgw_file: prevent conflict of mkdir between restarts

Merge pull request #15659 from lixiaoy1/fio_ceph

test/fio: Fix assert in set_cache_shards in bluestore fio

Fix assert in set_cache_shards in bluestore fio

The config osd_op_num_shards is changed so it needs to change in
fio_ceph_objectstore.cc.

Signed-off-by: Xiaoyan Li <xiaoyan.li@intel.com>

Merge pull request #15593 from chenfangxian/rbd-nbd-fix-unmap

rbd-nbd: ensure unmap returns error code

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Pan Liu <wanjun.lp@alibaba-inc.com>

doc: remove some non-existent and fix the default value according to source code

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

build: add css file to excludes while generate tags

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

rgw: fix the signature mismatch of FormPost in swift API

Fixes: http://tracker.ceph.com/issues/20220
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

qa/suites/upgrade: remove duplicated upgrade task

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/upgrade: set "sortbitwise" for jewel clusters

so ceph.healthy or wait-for-healthy won't be blocked by this warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw_file: prevent conflict of mkdir between restarts

Fixes: http://tracker.ceph.com/issues/20275
Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15648 from joscollin/wip-CID1412575-incorrect-expression

mon: Incorrect expression in PGMap::get_health()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15638 from liewegas/wip-19948

ceph_test_rados_api_*: wait for snap trim on ENOENT during cleanup

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: fix nbd do_unmap error handling

when nbd disconnect failed, ensure do_unmap return error logic.

Signed-off-by: chenfangxian <chenfangxian@cmss.chinamobile.com>
Signed-off-by: guojiannan <guojiannan@cmss.chinamobile.com>

Merge pull request #15519 from liewegas/wip-19595

osd/PrimaryLogPG: clear oi from trim_object()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon: Incorrect expression in PGMap::get_health()

Fixes:

CID 1412575:  Incorrect expression  (COPY_PASTE_ERROR)
ceph/src/mon/PGMap.cc: 2764 in PGMap::get_health()
"num_err" in "ss2 << num_err" looks like a copy-paste error.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15629 from Linbing/wip-msg-async-op-threads

msg/async: add assert of ms_async_op_threads > 0

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: fix accept_conn not remove entry in conns when lazy delete

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15426 from zhangsw/fix-rgw-gc-loglevel

rgw: lower some log's level in gc process.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15571 from ceph/wip-fix-fio-shard-hint

test/fio: fix lack of setting for Sequencer::shard_hint

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15603 from idryomov/wip-rados-df-typo

rados: fix typo in 'df' column name

Merge pull request #15615 from joscollin/wip-CID717210-Uninitialized-members

common: initialize _hash in LogEntryKey()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15627 from theanalyst/12.1.0/release-notes-2

doc: 12.1.0/release notes 2

ceph_test_rados_api_*: wait for snap trim on ENOENT during cleanup

Assume that an ENOENT removing an object during cleanup means the
head is a whiteout because there are clones.  When that happens
just sleep for a bit and then retry in the hopes that snap
trimming has happened.

Time out (and fail) after retrying for 10 minutes' worth of tries.

Fixes: http://tracker.ceph.com/issues/19948
Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: clear oi from trim_object()

trim_object() is unique in that its resulting OpContxtUPtr is fed to
simple_opc_submit() but does not first pass through finish_ctx().  Almost
none of what finish_ctx is helpful for us because we are the only place
where we delete a clone, and for that reason it is not terribly useful to
put what we do into finish_ctx.  However, *one* thing that finish_ctx()
does do that we don't (didn't) is clear out the object_info_t io on
delete.  This would leave things like version with a value that would
bleed into future instantiations of the object and break things like
the prior_version in pg_log_entry_t's and, much later, break asserts
when loading the missing set off of disk.

Fix by clearing oi.

Fixes: http://tracker.ceph.com/issues/19947
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15604 from liewegas/wip-rest-test-qa

qa/suites/rados/rest: test restful mgr module

Reviewed-by: Boris Ranto <branto@redhat.com>

qa/suites/rados/rest/mgr-restful: test on centos

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15584 from liewegas/wip-20219

mon/MgrStatMonitor: keep mgrstat version ahead of pgmon

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15620 from tchaikov/wip-20249

test/vstart_wrapper.sh: display_log on test failure

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rest/test_mgr_rest_api.py: tolerate old distros

Work with older requests and/or missing urllib3.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/rest/mgr-restful: test restful module

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15634 from liewegas/wip-pyformatter

mgr/PyFormatter: implement dump_format_va

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15580 from liewegas/wip-bluestore-memory

os/bluestore: configure rocksdb cache via bluestore_cache_kv_ratio

Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/PyFormatter: implement dump_format_va

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15624 from Adirl/wip

msg/async/rdma: check if fin message completed

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: check if fin message completed

Change-Id: I9cb1e523be7b21127503a9252b09331a37bea204
Signed-off-by: Alexander Mikheev <alexm@mellanox.com>
Signed-off-by: Adir Lev <adirl@mellanox.com>

mon/PGMonitor: assert PGMonitor stops proposing after luminous

One hacky exception here for the rebuild-mondb case.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: keep mgrstat version ahead of pgmon

Pre-luminous, statfs and getpoolstats come from pgmon, and clients will
see and track the paxos version for that.  After the switchover, those
move to mgrstatmon.  If a clients sends a version ahead of the current
mgrstat version, it will block until the paxos version catches up, which
may be years for an aged cluster.

Fix by keeping mgrstat version ahead of pgmon's every time it updates.
This only really matters right at the transition point (we don't update
before that, and after that pgmon doesn't update).

Fixes: http://tracker.ceph.com/issues/20219
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14753 from theanalyst/v12.0.2/mailmap

doc: mailmap for v12.0.2

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15328 from jimifm/wip-RGWPutObj

rgw/rgw_op:Prevents memory leaks when calling func swift_versioning_copy() fails

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15323 from fangyuxiangGL/compress_bug_fix

rgw: get wrong content when download object with specific range with compression

Reviewed-by: Adam Kupczyk <akupczyk@mirantis.com>
Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

msg/async: add assert of ms_async_op_threads > 0

Signed-off-by: linbing <linbing@t2cloud.net>

qa: timeout if flush_pg_stats() takes too long

a "timeout" which defaults to 300 seconds is added to flush_pg_stats()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mailmap: Yunfei Guan affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Andrey Parfenov affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Yao Zongyou affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Patrick Dinnen affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mailmap: Henrik Korkuc affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>