osd/PG: add explicit CANCEL MBackfillReserve type

This lets us distinguish between REJECT and CANCEL for clarity.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: separate event for RemoteReservationCanceled

Right now we transparently map a RemoteReservationRejected into a
*Canceled event because this what peers send over the wire.  Even
once new peers start sending and explicit CANCEL, old peers will
still do so, so we'll maintain this mapping for a while.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: separate verb 'Reject' from passive 'rejected'

This reduces pg->reject_reservation() callsites from 2 to 1 and
makes the state transitions a bit more explicit.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: make note about when we get MBackfillReserve REJECT messages

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: handle RecoveryReservationRejected in RepWaitRecoveryReserved

This state is analogous to RepWaitBackfillReserved; just like we do there
we want to handle the REJECT from the primary by canceling our local
remote_reservation.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: ignore RemoteReservationRejected if we are RepNotRecoverying

The primary may send us a REJECT (meaning cancel) if recovery/backfill is
preempted there.  That can happen even if the recovery isn't reserved or
requested here (e.g., because the primary is still waiting for the local
reservation).  Just ignore it and remain in RepNotRecovering.

Fixes: http://tracker.ceph.com/issues/21613
Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: cancel local reservation in RemoteReservationRejected handler

We can get a RemoteReservationRejected event either because *we* decide
to reject, or because we get a REJECT from the primary that means "cancel"
(e.g., because recovery/backfill was preempted there).  In both cases we
want to cancel our remote_reservation.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move reject_reservation out of RemoteReservationRejected reaction

The RemoteReservationRejected event is also submitted when we are a
replica or backfill target and get a MBackfillReserve REJECT message
because the primary canceled or was preempted.  In that case, we don't
want to send a REJECT back to the primary; we only need to send it in the
cases where *we*, locally, decide to reject.  Move the call to those call
sites.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18043 from dillaman/wip-21567

librbd: snapshots should be created/removed against data pool

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18042 from dillaman/wip-21529

librbd: image-meta config overrides should be dynamically refreshed

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18026 from dillaman/wip-21535

rbd-mirror: sync image metadata when transfering remote image

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17992 from dillaman/wip-21558

librbd: hold cache_lock while clearing cache nonexistence flags

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17989 from mkogan1/wip-rgw-21151

rgw: update vstart.sh to support rgw ssl port notation : '--rgw_port 443s'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #18041 into master

* refs/remotes/upstream/pull/18041/head:
qa: relax cap expected value check

Merge PR #17990 into master

* refs/remotes/upstream/pull/17990/head:
mds: open purge queue when transitioning out of standby replay
mds: always re-probe mds log when standby replay done

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #17935 into master

* refs/remotes/upstream/pull/17935/head:
ceph_volume_client: fix setting caps for IDs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17958 from gaosibei/branch1

RGW: fix a bug about inconsistent unit of comparison

Merge pull request #15199 from xiexingguo/wip-object-logic-size

osd: fine-grained statistics of logical object space usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16884 from liewegas/wip-20919

osd/PrimaryLogPG: send requests to primary on cache miss

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17955 from asomers/bin_bash2

test: fix bash path in shebangs (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17985 from dzafman/wip-21327

ceph-objectstore-tool: "$OBJ get-omaphdr" and "$OBJ list-omap" scan all pgs instead of using specific pg

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18005 from jcsp/wip-21577

tools: update monstore tool for fsmap, mgrmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18015 from tchaikov/wip-kill-warnings

osd,os/bluestore: kill clang analyzer warnings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18018 from tchaikov/wip-ceph-disk-cleanup

ceph-disk: more precise error message when a disk is specified

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18034 from tchaikov/wip-options

common/options: pass by reference and use user-literals for size

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17971 from idryomov/wip-krbd-exclude-shared-298

qa/suites/krbd: exclude shared/298

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: snapshots should be created/removed against data pool

Fixes: http://tracker.ceph.com/issues/21567
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid dynamically refreshing non-atomic configuration settings

Fixes: http://tracker.ceph.com/issues/21529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17920 from dzafman/wip-21382

Erasure code recovery should send additional reads if necessary

Fixes: http://tracker.ceph.com/issues/21382
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: relax cap expected value check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: notify watcher when updating image metadata

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #18036 from ownedu/wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under heavy RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/PrimaryLogPG: allow trimmed read for OP_CHECKSUM

Normal reads support trimmed read length, and so shall checksums!

This fixes occasionally failure of rados/thrash test scripts, e.g.:
(1) create object using WriteOp with random generated length
(2) normal writes might accompany with TruncOp of randomized chosen truncate_size
(3) for ReadOp, pick a random 'length' to read, and do checksum simultaneously
    for the same range ([0, 'length']) to read too.

Since the 'length' for reading is randomized chosen, it might
exceed the current object size, and hence causing an EOVERFLOW error.

Related issues:
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_01:52:47-rados-wip-object-logic-size-distro-basic-smithi/1657337
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_14:14:19-rados-wip-object-logic-size-distro-basic-smithi/1658015

Fix the above problems by keeping pace with normal reads.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fine-grained statistics of logical object space usage

To test this change, we create an image of 5GB and do rbd bench write of 1GB:
./bin/rbd create bar -s 5120 && ./bin/rbd bench --io-type write --io-size 32K --io-total 100M --io-pattern rand  rbd/bar

Below is the test result.

Was:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27052M        3859M         12.49
POOLS:
    NAME                  ID     USED      %USED     MAX AVAIL     OBJECTS
    rbd                   0      3191M     26.36         8914M        1174
    cephfs_data_a         1          0         0         8914M           0
    cephfs_metadata_a     2       2246         0         8914M          21

Now:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27050M        3861M         12.49
POOLS:
    NAME                  ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                   0      101216k      1.10         8913M        1178
    cephfs_data_a         1            0         0         8913M           0
    cephfs_metadata_a     2          892         0         8913M          21

E.g., this change can make "osd pool set-quota max_bytes" work nicely.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17979 from dillaman/wip-21559

rbd-mirror: forced promotion can result in incorrect status

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18035 from tchaikov/wip-bit-cleanup

script/build-integration-branch: python3 compatible and pep8 clean

ceph_volume_client: fix setting caps for IDs

... that have empty OSD and MDS caps. Don't add a ',' at the
start of OSD and MDS caps.

Fixes: http://tracker.ceph.com/issues/21501
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #17846 from jecluis/wip-21300

mon/MgrMonitor: read cmd descs if empty on update_from_paxos()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: For recovery get all possible shards to read on errors

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Allow recovery to send additional reads

For now it doesn't include non-acting OSDs
Added test for this case

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Allow modified options to existing setup functions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use feature to get last array element

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge branch 'wip-fix-asyncrdma-coredump' of https://github.com/ownedu/ceph into wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under
heavy RDMA traffic, there are chances to access a current_chunk which can
be beyond the range of pre-allocated Tx buffer pool thus causes a coredump

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Fix a potential coredump when handling tx_buffers under heacy RDMA
traffic, there are chances to access a current_chunk which can be beyond the
range of pre-allocated Tx buffer pool thus causes a coredump.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

common/interval_set: override subset_of for given range

E.g.:
subset_of([5~10,20~5], 0, 100)  -> [5~10,20~5]
subset_of([5~10,20~5], 5, 25)   -> [5~10,20~5]
subset_of([5~10,20~5], 1, 10)   -> [5~5]
subset_of([5~10,20~5], 8, 24)   -> [8~7, 20~4]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

script/build-integration-branch: python3 compatible and pep8 clean

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: use user-defined literals for sizes

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: pass by reference

for better performance. and for better consistency, pass by
std::initializer_list.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: do not assign never read variable

actually "r" is always 0 in that branch. so it's a no-op.

this silences the clang analyzer warning of

Value stored to 'r' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not assign never read variable

shrink the lexical scope of "csum_order" and do not set it if it is
never read.

this silences the clang analyzer warning of:

Value stored to 'csum_order' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not assign never read variable

"git" is not read after being reset.

this silences clang analyzer warning of:

Value stored to 'gift' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: do not set never read variable

do not set "omap_fsid_fd" to -1 after ::close() it.

also move it's definition to where it is use for the first time for
better readability.

this silences the clang analyzer warning of:

Value stored to 'omap_fsid_fd' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: fail if unable to update device class in crush

device-class is supported since luminous, so we should fail the init()
if device-class is not updated in crush.

this reverts 244db740: if the device class changes then the user should
either follow the instruction to "ceph osd crush rm-device-class" or fix
CrushWrapper::update_device_class().

this silences clang analyzer warning of:

Value stored to 'r' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17697 into master

* refs/remotes/upstream/pull/17697/head:
pybind/ceph_volume_client: add get, put, and delete object interfaces
pybind/ceph_volume_client: remove 'compat_version'
pybind/ceph_volume_client: set the version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17604 into master

* refs/remotes/upstream/pull/17604/head:
client:fix a bug in lseek

Reviewed-by: Zheng Yan <zyan@redhat.com>

RGW: fix a bug about inconsistent unit of comparison

Fixes：http://tracker.ceph.com/issues/21590

Signed-off-by: gaosibei <gaosb@inspur.com>

Merge PR #16036 into HEAD

* refs/remotes/upstream/pull/16036/head:
mds: improve cap min/max ratio descriptions
mds: fix whitespace
mds: cap client recall to min caps per client
mds: fix conf types
mds: fix whitespace
doc/cephfs: add client min cache and max cache ratio describe
mds: adding tunable features for caps_per_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: improve cap min/max ratio descriptions

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: cap client recall to min caps per client

Fixes: http://tracker.ceph.com/issues/21575
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17774 from amitkumar50/cov-misc-4

include,messages,rbd: Initialize counter,group_pool

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17820 from zmedico/PGPool-update-optimize-with-subset-of

osd/PGPool::update: optimize with subset_of

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17884 from kungf/stats_invalid_scrub

osd: make scrub right now when pg stats_invalid is true

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17984 from adamemerson/wip-lists-of-pointers-are-bad-in-options

options: Do not use linked lists of pointers!

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17987 from xiexingguo/wip-kick-recovery-for-degraded-obj

osd/PrimaryLogPG: kick off recovery on backoffing a degraded object

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17662 from shashalu/fix-set-bucket-versioning

rgw: set bucket versioninig donot change versioning status if missing status in xml

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17775 from theanalyst/rgw-keystone-logging

rgw: avoid logging keystone revocation messages when not configured

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Amit Kumar amitkuma@redhat.com

Merge pull request #17916 from cbodley/wip-21506

rgw: dont reuse stale RGWObjectCtx for get_bucket_info()

Merge pull request #17742 from adamemerson/wip-21389

rgw: Check payment operations in policy

Reviewed-by: Enming Zhang <enming.zhang@umcloud.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18023 from liewegas/wip-changelog

debian/rules: no more ChangeLog

Merge pull request #18002 from adamemerson/wip-21582

rgw: Check bucket GetBucketLocation in policy

Merge pull request #18024 from adamemerson/wip-21597

rgw: Check bucket Website operations in policy

rbd-mirror: sync image metadata when transfering remote image

Fixes: http://tracker.ceph.com/issues/21535
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17839 from liewegas/wip-recovery-preemption

osd: allow PG recovery scheduling preemption

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

debian/rules: no more ChangeLog

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Check bucket Website operations in policy

Add code to check s3:GetBucketWebsite and s3:PutBucketWebsite
operations against bucket policy.

Fixes: http://tracker.ceph.com/issues/21597
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1493896
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/rados/singleton/all/recovery-preemption: add test

This mirrors what I was testing locally.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18014 from theanalyst/doc-12.2.1

doc: 12.2.1 release notes

Merge pull request #17973 from ligne/patch-1

doc: Fix typos in placement-groups.rst

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc: add 12.2.1 to releases table

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

doc: 12.2.1 release notes

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #18000 from adamemerson/wip-21578

rgw: Check bucket CORS operations in policy

rgw: update vstart.sh to support ssl port notation : '--rgw_port 443s'
reference: https://github.com/civetweb/civetweb/blob/master/docs/OpenSSL.md

Fixes: http://tracker.ceph.com/issues/21151
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #17684 from majianpeng/bluefs-release-extents

os/bluestore/BlueFS: move release unused extents work in _flush_and_syn_log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18020 from theanalyst/doc-changelog-12.2.1

doc: add changelog for 12.2.1

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add changelog for 12.2.1

Signed-off-by: Abhishek Lekshmanan <alekshmanan@suse.com>

Merge pull request #17769 from optimistyzy/97_new_opt

os/bluestore/NVMEDevice: Remove using dpdk thread

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18001 from liewegas/wip-21417

os/bluestore: put cached attrs in correct mempool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #17755 from dillaman/wip-21353

doc/release-notes: ensure RBD users can blacklist prior to upgrade

Reviewed-By: Nathan Cutler <ncutler@suse.com>

os/bluestore: move several buffer{ptr,list}s into cache_other mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: put new attrs in correct mempool too

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17966 from xiexingguo/wip-bluefs-kill-block-total

os/bluestore/BlueFS: several cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

tools: update monstore tool for fsmap, mgrmap

Fixes: http://tracker.ceph.com/issues/21577
Signed-off-by: John Spray <john.spray@redhat.com>

tools: handle decode errors in monstore tool

Print a single line message instead of dumping
a backtrace.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph-disk: more precise error message when a disk is specified

we raise an exception if the path is not a block device, e.g. /dev/tty,
or it's disk, e.g, /dev/sda.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18013 from tchaikov/wip-dashboard-favicon

mgr/dashboard: use rel="icon" for favicon

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17993 from jcsp/wip-21572

mgr/dashboard: sort servers and OSDs in OSD list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17937 from tchaikov/wip-automake-cleanup

automake: remove files required by automake

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

mgr/dashboard: use rel="icon" for favicon

shortcut is not a standard compliant link type, see
https://www.w3.org/TR/html5/links.html#linkTypes.

Signed-off-by: Kefu Chai <kchai@redhat.com>