os/bluestore: define PREFIX_ALLOC_BITMAP

This was hard-coded to 'b'.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: simplify CF merge operator routing

Merge operators always map to a prefix, just like a CF, so we can link
directly to the operator without doing any lookup.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: add CF support

- detect and use existing CFs on open

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

kv/RocksDBStore: move to_bufferlist to .cc

Signed-off-by: Sage Weil <sage@redhat.com>

kv/KeyValueDB: add column family interface

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>
Signed-off-by: Sage Weil <sage@redhat.com>

kv/KineticStore: override

Signed-off-by: Sage Weil <sage@redhat.com>

kv/KeyValueDB: simplify iterator overloads

get_wholespace_iterator() and get_iterator().  This gets rid of
the goofy _get_iterator() indirection and avoids colliding on names w/
virtual methods.

Signed-off-by: Sage Weil <sage@redhat.com>

kv/KeyValueDB: use public iterator interface simple; hide prefixing one

This uses the most general class as the type for the Iterator
shared_ptr typedef, and makes the class that filters by prefix
private.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectMap: tie ObjectMap::ObjectMapIterator to superclass

These iterfaces are similar but ultimately unrelated! Unfortunately
we are stuck with a joined class hiearchy because of DBObjectMap.
Someday it would be nice to break this link.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18095 from ceph/wip-bz1491250

ceph-volume lvm list

Reviewed by: Andrew Schoen <aschoen@redhat.com>

doc/ceph-volume add lvm/list to the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the reference for the list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add create and list to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a unit test module for listing features

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18006 from adamwg/awg/rbd-mirror-default-data-pool

rbd-mirror: Improve data pool selection when creating images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17721 from amitkumar50/cov-librbd-4

librbd: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17696 from amitkumar50/cov-librbd-3

librbd: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17660 from gmayyyha/striping-feature-21360

librbd: fix don't send get_stripe_unit_count if striping is not enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17461 from PCzhangPC/bench_rw_test

 rbd:introduce rbd bench rw(for read and write mix) test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17576 from ceph/wip-rm-1-minsize

qa/tests/rados: Remove unsupported 2-size-1-min-size config

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume tests create a small factory fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests ensure Volume.as_dict gets populated correctly

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.listing initial take on listing command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api be able to return a dictionary repr of a Volume object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process allows to fully mute terminal output on commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api use a helper to get an lv from an arg

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm add listing module to main

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18053 from ownedu/wip-fix-async-rdma-tx-buffer-leakage

msg/async/rdma: fix Tx buffer leakage that can introduce "heartbeat no reply"

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Alex Mikheev <alexm@mellanox.com>

Merge pull request #17997 from ivancich/wip-bring-in-latest-dmclock

osd: bring in latest dmclock library updates

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18003 from cbodley/wip-build-options-move

common: use move instead of copy in build_options()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18029 from jdurgin/wip-kvstore-tool

ceph-kvstore-tool: copy to different store type and cleanup properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17999 from cbodley/wip-21576

rgw: include SSE-KMS headers in encrypted upload response

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #18063 from oritwas/wip-rgw-21610

doc: replace region with zonegroup in configure bucket sharding section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18077 from joscollin/wip-drop-unused-function-declaration

os/kstore: Drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18046 from mdw-at-linuxbox/master-awsv4-sort

radosgw: fix awsv4 header line sort order.

Merge pull request #18075 from dragonylffly/wip-remove-unused-declaration

os/bluestore: drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18067 from trociny/wip-qa-pg_num

qa/workunits/rbd: pool create may fail for small cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18076 from joscollin/wip-contributor-list-maintenance-guide

mailmap: Fixed maintenance guide URL

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17735 from jcsp/wip-mgr-perf-interface

mgr: common interface for TSDB modules

Reviewed-by: My Do <mhdo@umich.edu>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18069 from trociny/wip-affiliation

mailmap: update affiliation for Mykola Golub

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

os/kstore: Drop unused function declaration

Dropped unnecessary function declaration _do_transaction() in KStore.h

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap: Fixed maintenance guide URL

Fixed the Contributors List Maintenance Guide URL in .mailmap, .organizationmap, .githubmap and .peoplemap

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap, organizationmap: update affiliation for Mykola Golub

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

os/bluestore/BlueStore: remove unused declaration

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #18022 from neha-ojha/wip-21566

osd: hold lock while accessing recovery_needs_sleep

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18059 from Liuchang0812/remove-duplicated-func

osd: remove duplicated function ec_pool in pg_pool_t

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17913 from rzarzynski/wip-bs-bulky_disk_releases

os/bluestore: release disk extents in bulky manner

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed: Sage Weil <sage@redhat.com>

Merge pull request #17752 from XinzeChi/wip-force-recovery

osd: set min_version to newest version in maybe_force_recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17728 from ygtzf/cleanup-unused-parameter

mon: cleanup unused option mon_health_data_update_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16843 from amitkumar50/cov-radosstriper

rados: Intializing members class StriperTest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17912 from ceph/docs-min-size

docs: Do not use "min size = 1" as an example

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17931 from jcsp/wip-21518

mgr/zabbix: fix div by zero

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #18040 from joscollin/wip-guide-submitting-patches-typo

doc: Fix typo and URL

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18060 from dragonylffly/wip-revise-organization

doc: update Li Wang Affiliation

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: pool create may fail for small cluster

ceph osd pool create test 100
Error ERANGE:  pg_num 100 size 3 would mean 648 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from alex-mikheev (Alex Mikheev), to use a single
atomic counter for inflight Tx CQEs.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #18043 from dillaman/wip-21567

librbd: snapshots should be created/removed against data pool

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18042 from dillaman/wip-21529

librbd: image-meta config overrides should be dynamically refreshed

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18026 from dillaman/wip-21535

rbd-mirror: sync image metadata when transfering remote image

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17992 from dillaman/wip-21558

librbd: hold cache_lock while clearing cache nonexistence flags

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from tchaikov (Kefu Chai).

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #17989 from mkogan1/wip-rgw-21151

rgw: update vstart.sh to support rgw ssl port notation : '--rgw_port 443s'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: replace region with zonegroup in configure bucket sharding section

Fixes: http://tracker.ceph.com/issues/21610
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #18041 into master

* refs/remotes/upstream/pull/18041/head:
qa: relax cap expected value check

Merge PR #17990 into master

* refs/remotes/upstream/pull/17990/head:
mds: open purge queue when transitioning out of standby replay
mds: always re-probe mds log when standby replay done

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #17935 into master

* refs/remotes/upstream/pull/17935/head:
ceph_volume_client: fix setting caps for IDs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17958 from gaosibei/branch1

RGW: fix a bug about inconsistent unit of comparison

doc: revise organization

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

osd: remove duplicated function ec_pool in pg_pool_t

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #15199 from xiexingguo/wip-object-logic-size

osd: fine-grained statistics of logical object space usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16884 from liewegas/wip-20919

osd/PrimaryLogPG: send requests to primary on cache miss

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17955 from asomers/bin_bash2

test: fix bash path in shebangs (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17985 from dzafman/wip-21327

ceph-objectstore-tool: "$OBJ get-omaphdr" and "$OBJ list-omap" scan all pgs instead of using specific pg

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18005 from jcsp/wip-21577

tools: update monstore tool for fsmap, mgrmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18015 from tchaikov/wip-kill-warnings

osd,os/bluestore: kill clang analyzer warnings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18018 from tchaikov/wip-ceph-disk-cleanup

ceph-disk: more precise error message when a disk is specified

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18034 from tchaikov/wip-options

common/options: pass by reference and use user-literals for size

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

msg/async/rdma: fix Tx buffer leakage which can introduce "heartbeat no
reply" due to out of Tx buffers, this can be reproduced by marking some
OSDs down in a big Ceph cluster, say 300+ OSDs.

rootcause: when RDMAStack wants to delete faulty connections there are
chances that those QPs still have inflight CQEs, thus inflight Tx
buffers; without waiting for them to complete, Tx buffer pool will run
out of buffers finally.

fix: ideally the best way to fix this bug is to destroy QPs gracefully
such as to_dead(), we now just reply on the number of Tx WQE and CQE to
avoid buffer leakage; RDMAStack polling is always running so we are safe
to simply bypass some QPs that are not in 'complete' state.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

doc: Fix URL in Licensing

Fixed the unnecessary URL format in the text. Modify the URL formatting to highlight only the file name seems better.

Signed-off-by: Jos Collin <jcollin@redhat.com>

radosgw: fix awsv4 header line sort order.

The awsv4 signature calculation includes a list of header lines, which
are supposed to be sorted.  The existing code sorts by header name, but
it appears that in fact it is necessary to sort the whole header *line*,
not just the field name.  Sorting by just the field name usually works,
but not always.  The s3-tests teuthology suite includes
s3tests.functional.test_s3.test_object_header_acl_grants
s3tests.functional.test_s3.test_bucket_header_acl_grants
which include the following header lines,

x-amz-grant-read-acp:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-read:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-write-acp:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-write:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234

in this case, note that ':' needs to sort after '-'.

Fixes: http://tracker.ceph.com/issues/21607
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #17971 from idryomov/wip-krbd-exclude-shared-298

qa/suites/krbd: exclude shared/298

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: snapshots should be created/removed against data pool

Fixes: http://tracker.ceph.com/issues/21567
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: drop support for non-bulky extent release.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: release txc's extents in bulky manner.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: BlueFS releases disk extents in bulky manner.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: avoid dynamically refreshing non-atomic configuration settings

Fixes: http://tracker.ceph.com/issues/21529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17920 from dzafman/wip-21382

Erasure code recovery should send additional reads if necessary

Fixes: http://tracker.ceph.com/issues/21382
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: relax cap expected value check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: notify watcher when updating image metadata

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #18036 from ownedu/wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under heavy RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: Update rbd-mirror docs to reflect data pool selection changes

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

rbd-mirror: Improve data pool selection when creating images

Previously we used the source image's data pool name
unconditionally. There were two problems with that:

1. If a pool with the same name didn't exist locally, creation of the
   local image would fail.
2. If the local pool had a default data pool configured it would be
   ignored.

Change local image creation so it uses the default pool if configured,
and uses the remote pool name only if a pool with that name exists
locally. If neither of those is true, leave the data pool unset.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

os/bluestore: make the BitMapAllocator aware about bulk releases.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: make the StupidAllocator aware about bulk releases.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: extend the Allocator interface with bulk releases.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: Fix typo and URL in Submitting patches

Dropped the repeated 'the' in the paragraph and Fixed the unnecessary URL format in the text.

Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/PrimaryLogPG: allow trimmed read for OP_CHECKSUM

Normal reads support trimmed read length, and so shall checksums!

This fixes occasionally failure of rados/thrash test scripts, e.g.:
(1) create object using WriteOp with random generated length
(2) normal writes might accompany with TruncOp of randomized chosen truncate_size
(3) for ReadOp, pick a random 'length' to read, and do checksum simultaneously
    for the same range ([0, 'length']) to read too.

Since the 'length' for reading is randomized chosen, it might
exceed the current object size, and hence causing an EOVERFLOW error.

Related issues:
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_01:52:47-rados-wip-object-logic-size-distro-basic-smithi/1657337
http://qa-proxy.ceph.com/teuthology/xxg-2017-09-22_14:14:19-rados-wip-object-logic-size-distro-basic-smithi/1658015

Fix the above problems by keeping pace with normal reads.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fine-grained statistics of logical object space usage

To test this change, we create an image of 5GB and do rbd bench write of 1GB:
./bin/rbd create bar -s 5120 && ./bin/rbd bench --io-type write --io-size 32K --io-total 100M --io-pattern rand  rbd/bar

Below is the test result.

Was:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27052M        3859M         12.49
POOLS:
    NAME                  ID     USED      %USED     MAX AVAIL     OBJECTS
    rbd                   0      3191M     26.36         8914M        1174
    cephfs_data_a         1          0         0         8914M           0
    cephfs_metadata_a     2       2246         0         8914M          21

Now:

GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    30911M     27050M        3861M         12.49
POOLS:
    NAME                  ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                   0      101216k      1.10         8913M        1178
    cephfs_data_a         1            0         0         8913M           0
    cephfs_metadata_a     2          892         0         8913M          21

E.g., this change can make "osd pool set-quota max_bytes" work nicely.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17979 from dillaman/wip-21559

rbd-mirror: forced promotion can result in incorrect status

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #18035 from tchaikov/wip-bit-cleanup

script/build-integration-branch: python3 compatible and pep8 clean