ceph-disk: fix signed integer is greater than maximum when call major
  fix signed integer is greater than maximum when call os.major
  using python 2.7.5 in Centos 7

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

Merge PR #18041 into master

* refs/remotes/upstream/pull/18041/head:
qa: relax cap expected value check

Merge PR #17990 into master

* refs/remotes/upstream/pull/17990/head:
mds: open purge queue when transitioning out of standby replay
mds: always re-probe mds log when standby replay done

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #17935 into master

* refs/remotes/upstream/pull/17935/head:
ceph_volume_client: fix setting caps for IDs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17958 from gaosibei/branch1

RGW: fix a bug about inconsistent unit of comparison

Merge pull request #15199 from xiexingguo/wip-object-logic-size

osd: fine-grained statistics of logical object space usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16884 from liewegas/wip-20919

osd/PrimaryLogPG: send requests to primary on cache miss

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17955 from asomers/bin_bash2

test: fix bash path in shebangs (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17985 from dzafman/wip-21327

ceph-objectstore-tool: "$OBJ get-omaphdr" and "$OBJ list-omap" scan all pgs instead of using specific pg

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18005 from jcsp/wip-21577

tools: update monstore tool for fsmap, mgrmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18015 from tchaikov/wip-kill-warnings

osd,os/bluestore: kill clang analyzer warnings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18018 from tchaikov/wip-ceph-disk-cleanup

ceph-disk: more precise error message when a disk is specified

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18034 from tchaikov/wip-options

common/options: pass by reference and use user-literals for size

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17971 from idryomov/wip-krbd-exclude-shared-298

qa/suites/krbd: exclude shared/298

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17920 from dzafman/wip-21382

Erasure code recovery should send additional reads if necessary

Fixes: http://tracker.ceph.com/issues/21382
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: relax cap expected value check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18036 from ownedu/wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under heavy RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/PrimaryLogPG: allow trimmed read for OP_CHECKSUM

Normal reads support trimmed read length, and so shall checksums!

This fixes occasionally failure of rados/thrash test scripts, e.g.:
(1) create object using WriteOp with random generated length
(2) normal writes might accompany with TruncOp of randomized chosen truncate_size
(3) for ReadOp, pick a random 'length' to read, and do checksum simultaneously
    for the same range ([0, 'length']) to read too.

Since the 'length' for reading is randomized chosen, it might
exceed the current object size, and hence causing an EOVERFLOW error.

Related issues:
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_01:52:47-rados-wip-object-logic-size-distro-basic-smithi/1657337
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_14:14:19-rados-wip-object-logic-size-distro-basic-smithi/1658015

Fix the above problems by keeping pace with normal reads.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fine-grained statistics of logical object space usage

To test this change, we create an image of 5GB and do rbd bench write of 1GB:
./bin/rbd create bar -s 5120 && ./bin/rbd bench --io-type write --io-size 32K --io-total 100M --io-pattern rand  rbd/bar

Below is the test result.

Was:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27052M        3859M         12.49
POOLS:
    NAME                  ID     USED      %USED     MAX AVAIL     OBJECTS
    rbd                   0      3191M     26.36         8914M        1174
    cephfs_data_a         1          0         0         8914M           0
    cephfs_metadata_a     2       2246         0         8914M          21

Now:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27050M        3861M         12.49
POOLS:
    NAME                  ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                   0      101216k      1.10         8913M        1178
    cephfs_data_a         1            0         0         8913M           0
    cephfs_metadata_a     2          892         0         8913M          21

E.g., this change can make "osd pool set-quota max_bytes" work nicely.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17979 from dillaman/wip-21559

rbd-mirror: forced promotion can result in incorrect status

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18035 from tchaikov/wip-bit-cleanup

script/build-integration-branch: python3 compatible and pep8 clean

ceph_volume_client: fix setting caps for IDs

... that have empty OSD and MDS caps. Don't add a ',' at the
start of OSD and MDS caps.

Fixes: http://tracker.ceph.com/issues/21501
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #17846 from jecluis/wip-21300

mon/MgrMonitor: read cmd descs if empty on update_from_paxos()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: For recovery get all possible shards to read on errors

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Allow recovery to send additional reads

For now it doesn't include non-acting OSDs
Added test for this case

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Allow modified options to existing setup functions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use feature to get last array element

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge branch 'wip-fix-asyncrdma-coredump' of https://github.com/ownedu/ceph into wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under
heavy RDMA traffic, there are chances to access a current_chunk which can
be beyond the range of pre-allocated Tx buffer pool thus causes a coredump

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Fix a potential coredump when handling tx_buffers under heacy RDMA
traffic, there are chances to access a current_chunk which can be beyond the
range of pre-allocated Tx buffer pool thus causes a coredump.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

common/interval_set: override subset_of for given range

E.g.:
subset_of([5~10,20~5], 0, 100)  -> [5~10,20~5]
subset_of([5~10,20~5], 5, 25)   -> [5~10,20~5]
subset_of([5~10,20~5], 1, 10)   -> [5~5]
subset_of([5~10,20~5], 8, 24)   -> [8~7, 20~4]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

script/build-integration-branch: python3 compatible and pep8 clean

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: use user-defined literals for sizes

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: pass by reference

for better performance. and for better consistency, pass by
std::initializer_list.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: do not assign never read variable

actually "r" is always 0 in that branch. so it's a no-op.

this silences the clang analyzer warning of

Value stored to 'r' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not assign never read variable

shrink the lexical scope of "csum_order" and do not set it if it is
never read.

this silences the clang analyzer warning of:

Value stored to 'csum_order' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not assign never read variable

"git" is not read after being reset.

this silences clang analyzer warning of:

Value stored to 'gift' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: do not set never read variable

do not set "omap_fsid_fd" to -1 after ::close() it.

also move it's definition to where it is use for the first time for
better readability.

this silences the clang analyzer warning of:

Value stored to 'omap_fsid_fd' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: fail if unable to update device class in crush

device-class is supported since luminous, so we should fail the init()
if device-class is not updated in crush.

this reverts 244db740: if the device class changes then the user should
either follow the instruction to "ceph osd crush rm-device-class" or fix
CrushWrapper::update_device_class().

this silences clang analyzer warning of:

Value stored to 'r' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17697 into master

* refs/remotes/upstream/pull/17697/head:
pybind/ceph_volume_client: add get, put, and delete object interfaces
pybind/ceph_volume_client: remove 'compat_version'
pybind/ceph_volume_client: set the version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17604 into master

* refs/remotes/upstream/pull/17604/head:
client:fix a bug in lseek

Reviewed-by: Zheng Yan <zyan@redhat.com>

RGW: fix a bug about inconsistent unit of comparison

Fixes：http://tracker.ceph.com/issues/21590

Signed-off-by: gaosibei <gaosb@inspur.com>

Merge PR #16036 into HEAD

* refs/remotes/upstream/pull/16036/head:
mds: improve cap min/max ratio descriptions
mds: fix whitespace
mds: cap client recall to min caps per client
mds: fix conf types
mds: fix whitespace
doc/cephfs: add client min cache and max cache ratio describe
mds: adding tunable features for caps_per_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: improve cap min/max ratio descriptions

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cap client recall to min caps per client

Fixes: http://tracker.ceph.com/issues/21575
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17774 from amitkumar50/cov-misc-4

include,messages,rbd: Initialize counter,group_pool

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17820 from zmedico/PGPool-update-optimize-with-subset-of

osd/PGPool::update: optimize with subset_of

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17884 from kungf/stats_invalid_scrub

osd: make scrub right now when pg stats_invalid is true

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17984 from adamemerson/wip-lists-of-pointers-are-bad-in-options

options: Do not use linked lists of pointers!

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17987 from xiexingguo/wip-kick-recovery-for-degraded-obj

osd/PrimaryLogPG: kick off recovery on backoffing a degraded object

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17662 from shashalu/fix-set-bucket-versioning

rgw: set bucket versioninig donot change versioning status if missing status in xml

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17775 from theanalyst/rgw-keystone-logging

rgw: avoid logging keystone revocation messages when not configured

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17916 from cbodley/wip-21506

rgw: dont reuse stale RGWObjectCtx for get_bucket_info()

Merge pull request #17742 from adamemerson/wip-21389

rgw: Check payment operations in policy

Reviewed-by: Enming Zhang <enming.zhang@umcloud.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18023 from liewegas/wip-changelog

debian/rules: no more ChangeLog

Merge pull request #18002 from adamemerson/wip-21582

rgw: Check bucket GetBucketLocation in policy

Merge pull request #18024 from adamemerson/wip-21597

rgw: Check bucket Website operations in policy

Merge pull request #17839 from liewegas/wip-recovery-preemption

osd: allow PG recovery scheduling preemption

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

debian/rules: no more ChangeLog

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket Website operations in policy

Add code to check s3:GetBucketWebsite and s3:PutBucketWebsite
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21597
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493896
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/rados/singleton/all/recovery-preemption: add test

This mirrors what I was testing locally.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18014 from theanalyst/doc-12.2.1

doc: 12.2.1 release notes

Merge pull request #17973 from ligne/patch-1

doc: Fix typos in placement-groups.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: add 12.2.1 to releases table

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.2.1 release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #18000 from adamemerson/wip-21578

rgw: Check bucket CORS operations in policy

Merge pull request #17684 from majianpeng/bluefs-release-extents

os/bluestore/BlueFS: move release unused extents work in _flush_and_syn_log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18020 from theanalyst/doc-changelog-12.2.1

doc: add changelog for 12.2.1

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add changelog for 12.2.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #17769 from optimistyzy/97_new_opt

os/bluestore/NVMEDevice: Remove using dpdk thread

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18001 from liewegas/wip-21417

os/bluestore: put cached attrs in correct mempool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17755 from dillaman/wip-21353

doc/release-notes: ensure RBD users can blacklist prior to upgrade

Reviewed-By: Nathan Cutler <ncutler@suse.com>

os/bluestore: move several buffer{ptr,list}s into cache_other mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put new attrs in correct mempool too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17966 from xiexingguo/wip-bluefs-kill-block-total

os/bluestore/BlueFS: several cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

tools: update monstore tool for fsmap, mgrmap

Fixes: http://tracker.ceph.com/issues/21577
Signed-off-by: John Spray <john.spray@redhat.com>

tools: handle decode errors in monstore tool

Print a single line message instead of dumping
a backtrace.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph-disk: more precise error message when a disk is specified

we raise an exception if the path is not a block device, e.g. /dev/tty,
or it's disk, e.g, /dev/sda.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18013 from tchaikov/wip-dashboard-favicon

mgr/dashboard: use rel="icon" for favicon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17993 from jcsp/wip-21572

mgr/dashboard: sort servers and OSDs in OSD list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17937 from tchaikov/wip-automake-cleanup

automake: remove files required by automake

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mgr/dashboard: use rel="icon" for favicon

shortcut is not a standard compliant link type, see
https://www.w3.org/TR/html5/links.html#linkTypes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17991 from jcsp/wip-21570

mgr/dashboard: handle null in format_number

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17986 from tchaikov/wip-doc-rgw-config

doc: fix hyper link to radosgw/config-ref

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17926 from batrick/ptl-tool

scripts: add ptl-tool for scripting merges

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

rgw: Check bucket GetBucketLocation in policy

Add code to check s3:GetBucketLocation against bucket policy.

Fixes: http://tracker.ceph.com/issues/21582
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493934
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore: put attrs in mempool

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add ptr::[try_]reassign_to_mempool

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket CORS operations in policy

Add code to check s3:GetCORS and s3:PutCORS operations against bucket
policy.

Fixes: http://tracker.ceph.com/issues/21578
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494140
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd/PG: handle racy preemption

If we finish recovery/backfill and go active, but also get
preempted at the same time, we can ignore the event.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: allow local recovery reservations to be preempted

If a PG has a higher recovery priority and a lower-priority item is in
progress, allow it to be preempted.  This triggers the RecoveryCancel
or BackfillCancel event with a 0 delay, which means it will immediately
re-request a reservation (and presumably wait).

Signed-off-by: Sage Weil <sage@redhat.com>

common/AsyncReserver: support preemption

If an (optional) preemption context is provided, use that to preempt
and existing reservation and grant a higher-priority one.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: performance: Add --pgid for faster search for object by name

Fixes: http://tracker.ceph.com/issues/21327
Signed-off-by: David Zafman <dzafman@redhat.com>

rgw: Check bucket versioning operations in policy

Add code to check s3:GetBucketVersioning and s3:PutBucketVersioning
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Check payment operations in policy

Add code to check s3:GetBucketRequestPayment and
s3:PutBucketRequestPayment operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21389
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1490278
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17978 from alram/rbdmap-fix

rbdmap: fix umount when multiple mounts use the same RBD

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mds: fix conf types

This correct an assertion failure.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17962 from joscollin/wip-test-signed-unsigned-warning

test: silence warnings from -Wsign-compare

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>