ceph-volume tests ensure Volume.as_dict gets populated correctly

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.listing initial take on listing command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api be able to return a dictionary repr of a Volume object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process allows to fully mute terminal output on commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api use a helper to get an lv from an arg

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm add listing module to main

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18053 from ownedu/wip-fix-async-rdma-tx-buffer-leakage

msg/async/rdma: fix Tx buffer leakage that can introduce "heartbeat no reply"

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Alex Mikheev <alexm@mellanox.com>

Merge pull request #17997 from ivancich/wip-bring-in-latest-dmclock

osd: bring in latest dmclock library updates

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18003 from cbodley/wip-build-options-move

common: use move instead of copy in build_options()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18029 from jdurgin/wip-kvstore-tool

ceph-kvstore-tool: copy to different store type and cleanup properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17999 from cbodley/wip-21576

rgw: include SSE-KMS headers in encrypted upload response

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #18063 from oritwas/wip-rgw-21610

doc: replace region with zonegroup in configure bucket sharding section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18077 from joscollin/wip-drop-unused-function-declaration

os/kstore: Drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18046 from mdw-at-linuxbox/master-awsv4-sort

radosgw: fix awsv4 header line sort order.

Merge pull request #18075 from dragonylffly/wip-remove-unused-declaration

os/bluestore: drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18067 from trociny/wip-qa-pg_num

qa/workunits/rbd: pool create may fail for small cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18076 from joscollin/wip-contributor-list-maintenance-guide

mailmap: Fixed maintenance guide URL

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17735 from jcsp/wip-mgr-perf-interface

mgr: common interface for TSDB modules

Reviewed-by: My Do <mhdo@umich.edu>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18069 from trociny/wip-affiliation

mailmap: update affiliation for Mykola Golub

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

os/kstore: Drop unused function declaration

Dropped unnecessary function declaration _do_transaction() in KStore.h

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap: Fixed maintenance guide URL

Fixed the Contributors List Maintenance Guide URL in .mailmap, .organizationmap, .githubmap and .peoplemap

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap, organizationmap: update affiliation for Mykola Golub

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

os/bluestore/BlueStore: remove unused declaration

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #18022 from neha-ojha/wip-21566

osd: hold lock while accessing recovery_needs_sleep

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18059 from Liuchang0812/remove-duplicated-func

osd: remove duplicated function ec_pool in pg_pool_t

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17913 from rzarzynski/wip-bs-bulky_disk_releases

os/bluestore: release disk extents in bulky manner

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed: Sage Weil <sage@redhat.com>

Merge pull request #17752 from XinzeChi/wip-force-recovery

osd: set min_version to newest version in maybe_force_recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17728 from ygtzf/cleanup-unused-parameter

mon: cleanup unused option mon_health_data_update_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16843 from amitkumar50/cov-radosstriper

rados: Intializing members class StriperTest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17912 from ceph/docs-min-size

docs: Do not use "min size = 1" as an example

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17931 from jcsp/wip-21518

mgr/zabbix: fix div by zero

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #18040 from joscollin/wip-guide-submitting-patches-typo

doc: Fix typo and URL

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18060 from dragonylffly/wip-revise-organization

doc: update Li Wang Affiliation

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: pool create may fail for small cluster

ceph osd pool create test 100
Error ERANGE:  pg_num 100 size 3 would mean 648 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from alex-mikheev (Alex Mikheev), to use a single
atomic counter for inflight Tx CQEs.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #18043 from dillaman/wip-21567

librbd: snapshots should be created/removed against data pool

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18042 from dillaman/wip-21529

librbd: image-meta config overrides should be dynamically refreshed

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18026 from dillaman/wip-21535

rbd-mirror: sync image metadata when transfering remote image

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17992 from dillaman/wip-21558

librbd: hold cache_lock while clearing cache nonexistence flags

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from tchaikov (Kefu Chai).

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #17989 from mkogan1/wip-rgw-21151

rgw: update vstart.sh to support rgw ssl port notation : '--rgw_port 443s'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: replace region with zonegroup in configure bucket sharding section

Fixes: http://tracker.ceph.com/issues/21610
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #18041 into master

* refs/remotes/upstream/pull/18041/head:
qa: relax cap expected value check

Merge PR #17990 into master

* refs/remotes/upstream/pull/17990/head:
mds: open purge queue when transitioning out of standby replay
mds: always re-probe mds log when standby replay done

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #17935 into master

* refs/remotes/upstream/pull/17935/head:
ceph_volume_client: fix setting caps for IDs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17958 from gaosibei/branch1

RGW: fix a bug about inconsistent unit of comparison

doc: revise organization

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

osd: remove duplicated function ec_pool in pg_pool_t

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #15199 from xiexingguo/wip-object-logic-size

osd: fine-grained statistics of logical object space usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16884 from liewegas/wip-20919

osd/PrimaryLogPG: send requests to primary on cache miss

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17955 from asomers/bin_bash2

test: fix bash path in shebangs (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17985 from dzafman/wip-21327

ceph-objectstore-tool: "$OBJ get-omaphdr" and "$OBJ list-omap" scan all pgs instead of using specific pg

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18005 from jcsp/wip-21577

tools: update monstore tool for fsmap, mgrmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18015 from tchaikov/wip-kill-warnings

osd,os/bluestore: kill clang analyzer warnings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18018 from tchaikov/wip-ceph-disk-cleanup

ceph-disk: more precise error message when a disk is specified

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18034 from tchaikov/wip-options

common/options: pass by reference and use user-literals for size

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

msg/async/rdma: fix Tx buffer leakage which can introduce "heartbeat no
reply" due to out of Tx buffers, this can be reproduced by marking some
OSDs down in a big Ceph cluster, say 300+ OSDs.

rootcause: when RDMAStack wants to delete faulty connections there are
chances that those QPs still have inflight CQEs, thus inflight Tx
buffers; without waiting for them to complete, Tx buffer pool will run
out of buffers finally.

fix: ideally the best way to fix this bug is to destroy QPs gracefully
such as to_dead(), we now just reply on the number of Tx WQE and CQE to
avoid buffer leakage; RDMAStack polling is always running so we are safe
to simply bypass some QPs that are not in 'complete' state.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

doc: Fix URL in Licensing

Fixed the unnecessary URL format in the text. Modify the URL formatting to highlight only the file name seems better.

Signed-off-by: Jos Collin <jcollin@redhat.com>

radosgw: fix awsv4 header line sort order.

The awsv4 signature calculation includes a list of header lines, which
are supposed to be sorted.  The existing code sorts by header name, but
it appears that in fact it is necessary to sort the whole header *line*,
not just the field name.  Sorting by just the field name usually works,
but not always.  The s3-tests teuthology suite includes
s3tests.functional.test_s3.test_object_header_acl_grants
s3tests.functional.test_s3.test_bucket_header_acl_grants
which include the following header lines,

x-amz-grant-read-acp:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-read:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-write-acp:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-write:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234

in this case, note that ':' needs to sort after '-'.

Fixes: http://tracker.ceph.com/issues/21607
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #17971 from idryomov/wip-krbd-exclude-shared-298

qa/suites/krbd: exclude shared/298

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: snapshots should be created/removed against data pool

Fixes: http://tracker.ceph.com/issues/21567
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: drop support for non-bulky extent release.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: release txc's extents in bulky manner.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: BlueFS releases disk extents in bulky manner.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: avoid dynamically refreshing non-atomic configuration settings

Fixes: http://tracker.ceph.com/issues/21529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17920 from dzafman/wip-21382

Erasure code recovery should send additional reads if necessary

Fixes: http://tracker.ceph.com/issues/21382
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: relax cap expected value check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: notify watcher when updating image metadata

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #18036 from ownedu/wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under heavy RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore: make the BitMapAllocator aware about bulk releases.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: make the StupidAllocator aware about bulk releases.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: extend the Allocator interface with bulk releases.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: Fix typo and URL in Submitting patches

Dropped the repeated 'the' in the paragraph and Fixed the unnecessary URL format in the text.

Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/PrimaryLogPG: allow trimmed read for OP_CHECKSUM

Normal reads support trimmed read length, and so shall checksums!

This fixes occasionally failure of rados/thrash test scripts, e.g.:
(1) create object using WriteOp with random generated length
(2) normal writes might accompany with TruncOp of randomized chosen truncate_size
(3) for ReadOp, pick a random 'length' to read, and do checksum simultaneously
    for the same range ([0, 'length']) to read too.

Since the 'length' for reading is randomized chosen, it might
exceed the current object size, and hence causing an EOVERFLOW error.

Related issues:
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_01:52:47-rados-wip-object-logic-size-distro-basic-smithi/1657337
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_14:14:19-rados-wip-object-logic-size-distro-basic-smithi/1658015

Fix the above problems by keeping pace with normal reads.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fine-grained statistics of logical object space usage

To test this change, we create an image of 5GB and do rbd bench write of 1GB:
./bin/rbd create bar -s 5120 && ./bin/rbd bench --io-type write --io-size 32K --io-total 100M --io-pattern rand  rbd/bar

Below is the test result.

Was:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27052M        3859M         12.49
POOLS:
    NAME                  ID     USED      %USED     MAX AVAIL     OBJECTS
    rbd                   0      3191M     26.36         8914M        1174
    cephfs_data_a         1          0         0         8914M           0
    cephfs_metadata_a     2       2246         0         8914M          21

Now:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27050M        3861M         12.49
POOLS:
    NAME                  ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                   0      101216k      1.10         8913M        1178
    cephfs_data_a         1            0         0         8913M           0
    cephfs_metadata_a     2          892         0         8913M          21

E.g., this change can make "osd pool set-quota max_bytes" work nicely.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17979 from dillaman/wip-21559

rbd-mirror: forced promotion can result in incorrect status

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18035 from tchaikov/wip-bit-cleanup

script/build-integration-branch: python3 compatible and pep8 clean

ceph_volume_client: fix setting caps for IDs

... that have empty OSD and MDS caps. Don't add a ',' at the
start of OSD and MDS caps.

Fixes: http://tracker.ceph.com/issues/21501
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #17846 from jecluis/wip-21300

mon/MgrMonitor: read cmd descs if empty on update_from_paxos()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: For recovery get all possible shards to read on errors

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Allow recovery to send additional reads

For now it doesn't include non-acting OSDs
Added test for this case

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Allow modified options to existing setup functions

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use feature to get last array element

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge branch 'wip-fix-asyncrdma-coredump' of https://github.com/ownedu/ceph into wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under
heavy RDMA traffic, there are chances to access a current_chunk which can
be beyond the range of pre-allocated Tx buffer pool thus causes a coredump

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Fix a potential coredump when handling tx_buffers under heacy RDMA
traffic, there are chances to access a current_chunk which can be beyond the
range of pre-allocated Tx buffer pool thus causes a coredump.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

common/interval_set: override subset_of for given range

E.g.:
subset_of([5~10,20~5], 0, 100)  -> [5~10,20~5]
subset_of([5~10,20~5], 5, 25)   -> [5~10,20~5]
subset_of([5~10,20~5], 1, 10)   -> [5~5]
subset_of([5~10,20~5], 8, 24)   -> [8~7, 20~4]

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

script/build-integration-branch: python3 compatible and pep8 clean

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: use user-defined literals for sizes

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: pass by reference

for better performance. and for better consistency, pass by
std::initializer_list.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/rados: do not assign never read variable

actually "r" is always 0 in that branch. so it's a no-op.

this silences the clang analyzer warning of

Value stored to 'r' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not assign never read variable

shrink the lexical scope of "csum_order" and do not set it if it is
never read.

this silences the clang analyzer warning of:

Value stored to 'csum_order' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: do not assign never read variable

"git" is not read after being reset.

this silences clang analyzer warning of:

Value stored to 'gift' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/filestore: do not set never read variable

do not set "omap_fsid_fd" to -1 after ::close() it.

also move it's definition to where it is use for the first time for
better readability.

this silences the clang analyzer warning of:

Value stored to 'omap_fsid_fd' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: fail if unable to update device class in crush

device-class is supported since luminous, so we should fail the init()
if device-class is not updated in crush.

this reverts 244db740: if the device class changes then the user should
either follow the instruction to "ceph osd crush rm-device-class" or fix
CrushWrapper::update_device_class().

this silences clang analyzer warning of:

Value stored to 'r' is never read

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17697 into master

* refs/remotes/upstream/pull/17697/head:
pybind/ceph_volume_client: add get, put, and delete object interfaces
pybind/ceph_volume_client: remove 'compat_version'
pybind/ceph_volume_client: set the version

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17604 into master

* refs/remotes/upstream/pull/17604/head:
client:fix a bug in lseek

Reviewed-by: Zheng Yan <zyan@redhat.com>

RGW: fix a bug about inconsistent unit of comparison

Fixes：http://tracker.ceph.com/issues/21590

Signed-off-by: gaosibei <gaosb@inspur.com>

Merge PR #16036 into HEAD

* refs/remotes/upstream/pull/16036/head:
mds: improve cap min/max ratio descriptions
mds: fix whitespace
mds: cap client recall to min caps per client
mds: fix conf types
mds: fix whitespace
doc/cephfs: add client min cache and max cache ratio describe
mds: adding tunable features for caps_per_client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: improve cap min/max ratio descriptions

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>