rgw: Check bucket CORS operations in policy

Add code to check s3:GetCORS and s3:PutCORS operations against bucket
policy.

Fixes: http://tracker.ceph.com/issues/21578
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494140
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17978 from alram/rbdmap-fix

rbdmap: fix umount when multiple mounts use the same RBD

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17962 from joscollin/wip-test-signed-unsigned-warning

test: silence warnings from -Wsign-compare

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17988 from wumingqiao/wip-rados-update

include/rados: fix typo in librados.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

include/rados: fix typo in librados.h

opeprate -> operate

Signed-off-by: wumingqiao <wumingqiao@inspur.com>

rbdmap: fix umount when multiple mounts use the same RBD

When a Kubernetes Pod consumes a RBD it is mounted two
times on the same host. When the host shutdown umount will
fail leading to a hung system

Signed-off-by: Alexandre Marangone <amarango@redhat.com>

Merge pull request #17063 from bspark8/wip-dynamic-cli-info-f

osd: use dmclock library client_info_f function dynamically

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17335 from Carudy/cxw

crush: safe check for 'ceph osd crush swap-bucket'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17410 from zmedico/PGPool-update-optimize-removed_snaps-comparison-when-possible

PGPool::update: optimize removed_snaps comparison when possible

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #17980 from liewegas/wip-boost-url

make-dist,cmake: move boost tarball location to download.ceph.com

Reviewed-by: Jeff Layton <jlayton@redhat.com>

make-dist,cmake: move boost tarball location to download.ceph.com

Sourceforge is down.  Also, we can hammer our servers instead of
theirs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17589 from myENA/feature/20883-adminapi_sync_stats_master

rgw: admin api - add ability to sync user stats from admin api

Merge pull request #17433 from C2python/check_action

rgw: Return Error if Bucket Policy Contians Undefined Action

Merge pull request #17969 from ceph/rbd-mirror-reboot

systemd: rbd-mirror does not start on reboot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17929 from renhwztetecs/renhw-wip-mgrmon-command_descs_prefix

mon/mgr: sync "mgr_command_descs","osd_metadata" and "mgr_metadata" prefixes to new mons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17932 from jcsp/wip-21197

mgr: store declared_types in MgrSession

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17933 from jcsp/wip-17737

mgr: fix crashable DaemonStateIndex::get calls

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17944 from thmour/tmour_upmap

osd/OSDMap: upmap should respect the osd reweights

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17880 from amitkumar50/cov-rgw-9

rgw: Error check on return of read_line()

rbd-mirorr: does not start on reboot

The current systemd unit file misses 'PartOf=ceph-rbd-mirror.target',
which results in the unit not starting after reboot.
If you have ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0, it won't start
after reboot even if enabled.
Adding 'PartOf=ceph-rbd-mirror.target' will enable
ceph-rbd-mirror.target when ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0
gets enabled.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #17961 from tchaikov/wip-doc-local-pool-in-toc

doc/mgr: add "local pool" plugin to toc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17949 from asomers/nproc_freebsd

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17940 from jcsp/wip-21534

mon: update get_store_prefixes implementations

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: silence warnings from -Wsign-compare

Fixed the warnings:

ceph/src/test/bufferlist.cc:2150:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

ceph/src/test/bufferlist.cc:2151:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/mgr: add "local pool" plugin to toc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17842 from MahatiC/fix_example

examples: fix link order in librados example Makefile

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17633 from yonghengdexin735/wip-zzz-doc

doc/rbd: add info for rbd group

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17803 from amitkumar50/cov-librbd-5

test: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17800 from amitkumar50/cov-misc-8

test/librbd: initialize on_finish,locker,force,snap_id

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Use sysctl(8) instead.  Also, there's no longer any need for /bin/bash
on FreeBSD.

Signed-off-by: Alan Somers <asomers@gmail.com>

Merge pull request #17813 from liewegas/wip-21410-note

PendingReleaseNotes: note about upmap mapping change in luminous release notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: note about upmap mapping change in luminous release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17809 from yaozongyou/fix-rgw-file-write-error

rgw_file: fix write error when the write offset overlaps

Merge pull request #17947 from batrick/ceph-osd-core-dbg

ceph-debug-docker.sh: add ceph-osd-dbg package

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-debug-docker.sh: add ceph-osd-dbg package

So we can debug ceph-osd cores.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: upmap should respect osd reweights

Modify OSDMap::calc_pg_upmaps to take the osd reweight into account when
computing the size of each OSD.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>
Fixes: http://tracker.ceph.com/issues/21538

test/cli/osdmaptool: test upmap with an out osd

Test upmap with one osd marked out.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #17810 from hjwsm1989/wip-21294

qa/ceph_manager: check pg state again before timedout

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17661 from yunfeiguan/master

librados: Fix a potential risk of buffer::list::claim_prepend(list& b…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17790 from joscollin/wip-test-warning-signed-unsigned

test: silence warning from -Wsign-compare

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17928 from xiexingguo/wip-more-interval-set-size

osd/PrimaryLogPG: calc clone_overlap size in a more efficient and concise way

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17528 from liewegas/wip-mgr-localpool

pybind/mgr/localpool: module to automagically create localized pools

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/mgr/test_localpool: simple test for localpool mode

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: add --mark-out option

Add a --mark-down <id> option which sets the reweight for a given
osd id to 0.0.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

examples: fix link order in librados example Makefile

The library link order in librados example Makefile is incorrect
and as a result 'make' throws an error. This change fixes the order.

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

mon/mgr: sync mgr_command_descs table and mgr_metadata table

sync mgr_command_descs table and mgr_metadata table to new mons

Fixes: http://tracker.ceph.com/issues/21527
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon/osd_metadata: sync osd_metadata table

sync osd_metadata table to new mons when add new mons

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mgr: fix crashable DaemonStateIndex::get calls

This function was recently fixed to return null
on missing entries: handle that properly.

Fixes: http://tracker.ceph.com/issues/17737
Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement MDSMonitor::get_store_prefixes

Fixes: http://tracker.ceph.com/issues/21534
Signed-off-by: John Spray <john.spray@redhat.com>

mon/OSDMonitor: tidy prefix definitions

We should define them in one place to make it easy
when updating get_store_prefixes.

Fixes: http://tracker.ceph.com/issues/21534
Signed-off-by: John Spray <john.spray@redhat.com>

mon/osd_metadata: sync osd_metadata table

sync osd_metadata table to new mons when add new mons

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #17831 from badone/wip-require-osd-release-with-no-up-osds

mon: osd feature checks with 0 up osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17848 from liewegas/wip-ruleset-errors

mon/OSDMonitor: error out if setting ruleset-* ec profile property

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #17902 from tchaikov/wip-21474

qa/tasks: prolong revive_osd() timeout to 6 min

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17903 from lixiaoy1/add_func_log

os/bluestore: Add missing __func__ in dout

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17905 from tchaikov/wip-osd_types-cleanup

osd/osd_types: object_info_t: remove unused function

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17927 from liewegas/wip-21511

qa/standalone/scrub/osd-scrub-snaps: adjust test for lack of snapdir objects

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #17919 into master

* refs/remotes/upstream/pull/17919/head:
qa: whitelist cache too large in client-limits

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17918 into master

* refs/remotes/upstream/pull/17918/head:
qa: whitelist failover log warnings

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17917 into master

* refs/remotes/upstream/pull/17917/head:
qa: ignore evicted client in client-recovery

Reviewed-by: Zheng Yan <zyan@redhat.com>

mon/OSDMonitor: error out if setting ruleset-* ec profile property

We change ruleset -> crush back in dc7a2aaf7a34b1e6af0c7b79dc44a69974c1da23.
If someone tries to use the old property, error out early, instead of
silently not doing the thing they thought they told us to do.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file: fix write error when the write offset overlaps.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

mgr: store declared_types in MgrSession

Because we don't (yet) properly prevent multiple sessions
from daemons reporting the same name (e.g. rgws), storing
it in the DaemonPerfCounters meant that one daemon's report
was referring to another daemon's set of reported types.

This should always have been a property of the session.

The behaviour will still be ugly when multiple daemons
are using the same name (stomping on each other's stats/statsu)
but it shouldn't crash.

Fixes: http://tracker.ceph.com/issues/21197
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #17850 from batrick/vstart-confg-path

qa: revert "qa: use config_path property instead of literal"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17881 from joscollin/wip-mgr-signed-unsigned-warning

mgr: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: calc clone_overlap size in a more efficient and concise way

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/standalone/scrub/osd-scrub-snaps: adjust test for lack of snapdir objects

The head_exists stuff is totally gone; those test failures go away.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17756 from jcsp/wip-19291

mgr: set explicit thread name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17808 from xiexingguo/wip-eqduot-vs-enospc

osd/PrimaryLogPG: prepare_transaction - fix EDQUOT vs ENOSPC

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17819 from neha-ojha/wip-cbt-teuthology-fix-pg-num

qa/suites/rados/perf: create pool with lower pg_num

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17827 from tchaikov/wip-mono-heartbeat

common: use mono clock for HeartbeatMap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17844 from liewegas/wip-21259

os/bluestore: debug bluestore cache shutdown

Merge pull request #17845 from liewegas/wip-21408

os/bluestore: ignore 0x2000~2000 extent oddity from luminous upgrade

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge PR #17853 into master

* refs/remotes/upstream/pull/17853/head:
mds: check ongoing catter-gather process before capping log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17851 into master

* refs/remotes/upstream/pull/17851/head:
client: use unique_ptr for locks
client: use unique_ptr for Formatter

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #17843 into master

* refs/remotes/upstream/pull/17843/head:
ceph_volume_client: perform snapshot operations in

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17757 from ceph/wip-rgw-ansible-var

qa/tests: Add additional required ceph-ansible vars due to upstream changes

qa: whitelist cache too large in client-limits

Fixes: http://tracker.ceph.com/issues/21510
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: whitelist failover log warnings

Fixes: http://tracker.ceph.com/issues/21509
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore evicted client in client-recovery

Fixes: http://tracker.ceph.com/issues/21508
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17910 from smithfarm/wip-21499

tests: point-to-point-x: upgrade client.1 to -x along with cluster nodes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17904 from carltonf/wip-fix-21498

ceph-disk: fix '--runtime' omission for ceph-osd service

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17770 from aclamk/fix-bluestore-fio-hash

bluestore/fio: Fixed problem with all objects having the same hash

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17901 from tchaikov/wip-cmake-gperftools

cmake: update the error message for gperftools bug

Add missed __func__ in dout

With __func__ it makes users to know where it prints the logs in source code.

Signed-off-by: Xiaoyan Li xiaoyan.li@intel.com

qa: point-to-point-x: upgrade client.1 to -x along with cluster nodes

The client.1 rgw in workload_x had not been upgraded to -x.

Fixes: http://tracker.ceph.com/issues/21499
Signed-off-by: Nathan Cutler <ncutler@suse.com>

crush: safe check for 'ceph osd crush swap-bucket'
The swap method is just swap the two buckets'weight,son and name.So there is a problem when we swap parent and children. When swap two buckets,one should not be ancestor of the other,or there will be a loop,which would destory the structure.

Signed-off-by: Carudy <zhaoyang.han@easystack.cn>

osd/osd_types: object_info_t: remove unused function

legacy_object_locator_to_ps() is not used anymore, so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17806 from tchaikov/wip-osdc-new-options

osdc/Journaler: use new style options

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17823 from xiexingguo/wip-clone-bytes

osd/osd_type: get_clone_bytes - inline size() for overlapping size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17794 from hjwsm1989/wip-21425

qa/suites/rest/basic/tasks/rest_test: more whitelisting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17744 from xiexingguo/wip-blue-eio

os/bluestore: propagate read-EIO to high level callers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17460 from dengquan/master

java/native: fix milliseconds to mtime/atime conversion

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: mon osd feature checks with 0 up osds

get_up_osd_features() returns 0 if no osds are up which causes feature
checks to fail.

Fixes: http://tracker.ceph.com/issues/21471
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17422 from aclamk/fix-cmake-unsupported-allocator

cmake: in case of bad "ALLOCATOR" selected issue warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17374 from jcsp/wip-mgr-config-tidy

mgr: use new style config opts + add metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17309 from jcsp/wip-daemonserver-locks

mgr: locking fixes

Reviewed-by: Sage Weil <sage@redhat.com>

osdc/Journaler: use new style options

also use uint64_t for these options, as they are casted to uint64_t when
reading from conf.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17830 from Yan-waller/wip-walle-0920primarypg

osd/PrimaryLogPG: misc cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: fix '--runtime' omission for ceph-osd service

f425a127b introduces a regression that ceph-disk omits "--runtime" when
enabling ceph-osd@$ID.service units for device-backed OSDs.

Fixes: http://tracker.ceph.com/issues/21498
Signed-off-by: Carl Xiong <cxiong@suse.com>