librbd: list_children should not attempt to refresh image

The snap_lock is being held when this method is invoked, which can
result in a deadlock.

Fixes: http://tracker.ceph.com/issues/21670
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18089 from liewegas/wip-localpool-min-size

mgr/localpool: default to 3x; allow min_size adjustment

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/localpool: fix rule selection

The 'osd pool create' arg parsing is broken; the rule name for
'ceph osd pool create $name $numpgs replicated $rulename' is passed
via the erasure_code_profile param.  Too many req=false options
without a way to disambiguate them.

Work around it by passing both 'rule' and 'erasure_code_profile'
keys, so that if/when the hack in OSDMonitor.cc is removed it will
still work.  Blech.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18048 from badone/wip-boost-mirrors

make-dist,cmake: Try multiple URLs to download boost before failing

Reviewed-by: Kefu Chai <kchai@redhat.com>

make-dist,cmake: Try multiple URLs to download boost before failing

Remove SPOF during boost download for make-dist and later cmake > 3.7

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #18097 from dillaman/wip-21663

qa/workunits/rbd: fixed variable name for resync image id

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18052 from xiexingguo/wip-clear-pin-stats-invalid

osd/PrimaryLogPG: clear pin_stats_invalid bit properly on scrub-repair completion

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18061 from xiexingguo/wip-crc

osd/PrimaryLogPG: do not set data/omap digest blindly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18066 from shun-s/wip-objectstore-tool

tool: change default objectstore from filestore to bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18078 from liewegas/wip-21614

qa/suites/rados/singleton/all/recover-preemption: handle slow starting osd

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18095 from ceph/wip-bz1491250

ceph-volume lvm list

Reviewed by: Andrew Schoen <aschoen@redhat.com>

doc/ceph-volume add lvm/list to the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the reference for the list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add create and list to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a unit test module for listing features

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rbd: fixed variable name for resync image id

Fixes: http://tracker.ceph.com/issues/21663
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18006 from adamwg/awg/rbd-mirror-default-data-pool

rbd-mirror: Improve data pool selection when creating images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17721 from amitkumar50/cov-librbd-4

librbd: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17696 from amitkumar50/cov-librbd-3

librbd: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17660 from gmayyyha/striping-feature-21360

librbd: fix don't send get_stripe_unit_count if striping is not enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17461 from PCzhangPC/bench_rw_test

 rbd:introduce rbd bench rw(for read and write mix) test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17576 from ceph/wip-rm-1-minsize

qa/tests/rados: Remove unsupported 2-size-1-min-size config

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume tests create a small factory fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests ensure Volume.as_dict gets populated correctly

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.listing initial take on listing command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api be able to return a dictionary repr of a Volume object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process allows to fully mute terminal output on commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api use a helper to get an lv from an arg

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm add listing module to main

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18053 from ownedu/wip-fix-async-rdma-tx-buffer-leakage

msg/async/rdma: fix Tx buffer leakage that can introduce "heartbeat no reply"

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Alex Mikheev <alexm@mellanox.com>

Merge pull request #17997 from ivancich/wip-bring-in-latest-dmclock

osd: bring in latest dmclock library updates

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18003 from cbodley/wip-build-options-move

common: use move instead of copy in build_options()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18029 from jdurgin/wip-kvstore-tool

ceph-kvstore-tool: copy to different store type and cleanup properly

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/localpool: optionally adjust min_size too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/localpool: default to 3x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17999 from cbodley/wip-21576

rgw: include SSE-KMS headers in encrypted upload response

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #18063 from oritwas/wip-rgw-21610

doc: replace region with zonegroup in configure bucket sharding section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18077 from joscollin/wip-drop-unused-function-declaration

os/kstore: Drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18046 from mdw-at-linuxbox/master-awsv4-sort

radosgw: fix awsv4 header line sort order.

Merge pull request #18075 from dragonylffly/wip-remove-unused-declaration

os/bluestore: drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/recover-preemption: handle slow starting osd

The OSD may not be marked up yet; set the config via the admin
socket.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18067 from trociny/wip-qa-pg_num

qa/workunits/rbd: pool create may fail for small cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18076 from joscollin/wip-contributor-list-maintenance-guide

mailmap: Fixed maintenance guide URL

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17735 from jcsp/wip-mgr-perf-interface

mgr: common interface for TSDB modules

Reviewed-by: My Do <mhdo@umich.edu>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18069 from trociny/wip-affiliation

mailmap: update affiliation for Mykola Golub

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

os/kstore: Drop unused function declaration

Dropped unnecessary function declaration _do_transaction() in KStore.h

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap: Fixed maintenance guide URL

Fixed the Contributors List Maintenance Guide URL in .mailmap, .organizationmap, .githubmap and .peoplemap

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap, organizationmap: update affiliation for Mykola Golub

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

os/bluestore/BlueStore: remove unused declaration

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #18022 from neha-ojha/wip-21566

osd: hold lock while accessing recovery_needs_sleep

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18059 from Liuchang0812/remove-duplicated-func

osd: remove duplicated function ec_pool in pg_pool_t

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17913 from rzarzynski/wip-bs-bulky_disk_releases

os/bluestore: release disk extents in bulky manner

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed: Sage Weil <sage@redhat.com>

Merge pull request #17752 from XinzeChi/wip-force-recovery

osd: set min_version to newest version in maybe_force_recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17728 from ygtzf/cleanup-unused-parameter

mon: cleanup unused option mon_health_data_update_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16843 from amitkumar50/cov-radosstriper

rados: Intializing members class StriperTest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17912 from ceph/docs-min-size

docs: Do not use "min size = 1" as an example

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17931 from jcsp/wip-21518

mgr/zabbix: fix div by zero

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #18040 from joscollin/wip-guide-submitting-patches-typo

doc: Fix typo and URL

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18060 from dragonylffly/wip-revise-organization

doc: update Li Wang Affiliation

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: pool create may fail for small cluster

ceph osd pool create test 100
Error ERANGE:  pg_num 100 size 3 would mean 648 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from alex-mikheev (Alex Mikheev), to use a single
atomic counter for inflight Tx CQEs.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #18043 from dillaman/wip-21567

librbd: snapshots should be created/removed against data pool

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18042 from dillaman/wip-21529

librbd: image-meta config overrides should be dynamically refreshed

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18026 from dillaman/wip-21535

rbd-mirror: sync image metadata when transfering remote image

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17992 from dillaman/wip-21558

librbd: hold cache_lock while clearing cache nonexistence flags

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from tchaikov (Kefu Chai).

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

tool: change default objectstore from filestore to bluestore
  as Luminous gets released, it's better to switch default objectstore from filestore to bluestore.

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

Merge pull request #17989 from mkogan1/wip-rgw-21151

rgw: update vstart.sh to support rgw ssl port notation : '--rgw_port 443s'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: replace region with zonegroup in configure bucket sharding section

Fixes: http://tracker.ceph.com/issues/21610
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #18041 into master

* refs/remotes/upstream/pull/18041/head:
qa: relax cap expected value check

Merge PR #17990 into master

* refs/remotes/upstream/pull/17990/head:
mds: open purge queue when transitioning out of standby replay
mds: always re-probe mds log when standby replay done

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #17935 into master

* refs/remotes/upstream/pull/17935/head:
ceph_volume_client: fix setting caps for IDs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17958 from gaosibei/branch1

RGW: fix a bug about inconsistent unit of comparison

osd/PrimaryLogPG: do not set data/omap digest blindly

As bluestore has bulitin csum, we generally no longer generate
object data digest for now. The consequence is that we should
handle data/omap digest more carefully to make certain ops,
such as copy_from/promote, to work properly since they heavily
relies on data digest for data transfer correctness.

Example of failure:
http://pulpito.ceph.com/xxg-2017-09-30_11:46:34-rbd-master-distro-basic-mira/1690609/

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: revise organization

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

osd: remove duplicated function ec_pool in pg_pool_t

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #15199 from xiexingguo/wip-object-logic-size

osd: fine-grained statistics of logical object space usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16884 from liewegas/wip-20919

osd/PrimaryLogPG: send requests to primary on cache miss

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17955 from asomers/bin_bash2

test: fix bash path in shebangs (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17985 from dzafman/wip-21327

ceph-objectstore-tool: "$OBJ get-omaphdr" and "$OBJ list-omap" scan all pgs instead of using specific pg

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18005 from jcsp/wip-21577

tools: update monstore tool for fsmap, mgrmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18015 from tchaikov/wip-kill-warnings

osd,os/bluestore: kill clang analyzer warnings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18018 from tchaikov/wip-ceph-disk-cleanup

ceph-disk: more precise error message when a disk is specified

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18034 from tchaikov/wip-options

common/options: pass by reference and use user-literals for size

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

msg/async/rdma: fix Tx buffer leakage which can introduce "heartbeat no
reply" due to out of Tx buffers, this can be reproduced by marking some
OSDs down in a big Ceph cluster, say 300+ OSDs.

rootcause: when RDMAStack wants to delete faulty connections there are
chances that those QPs still have inflight CQEs, thus inflight Tx
buffers; without waiting for them to complete, Tx buffer pool will run
out of buffers finally.

fix: ideally the best way to fix this bug is to destroy QPs gracefully
such as to_dead(), we now just reply on the number of Tx WQE and CQE to
avoid buffer leakage; RDMAStack polling is always running so we are safe
to simply bypass some QPs that are not in 'complete' state.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

doc: Fix URL in Licensing

Fixed the unnecessary URL format in the text. Modify the URL formatting to highlight only the file name seems better.

Signed-off-by: Jos Collin <jcollin@redhat.com>

osd/PrimaryLogPG: clear pin_stats_invalid bit properly on scrub-repair completion

We have done audit of stats and the numbers should be all ok by then.
Actually the pin_stats_invalid bit is never set true, so forgetting
to clear pin_stats_invalid here generally does harm. Also we could simply
kill the pin_stats_invalid bit instead but let's not bother with that
complexity either.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

radosgw: fix awsv4 header line sort order.

The awsv4 signature calculation includes a list of header lines, which
are supposed to be sorted.  The existing code sorts by header name, but
it appears that in fact it is necessary to sort the whole header *line*,
not just the field name.  Sorting by just the field name usually works,
but not always.  The s3-tests teuthology suite includes
s3tests.functional.test_s3.test_object_header_acl_grants
s3tests.functional.test_s3.test_bucket_header_acl_grants
which include the following header lines,

x-amz-grant-read-acp:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-read:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-write-acp:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
x-amz-grant-write:id=56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234

in this case, note that ':' needs to sort after '-'.

Fixes: http://tracker.ceph.com/issues/21607
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #17971 from idryomov/wip-krbd-exclude-shared-298

qa/suites/krbd: exclude shared/298

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: snapshots should be created/removed against data pool

Fixes: http://tracker.ceph.com/issues/21567
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: drop support for non-bulky extent release.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: release txc's extents in bulky manner.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: BlueFS releases disk extents in bulky manner.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: avoid dynamically refreshing non-atomic configuration settings

Fixes: http://tracker.ceph.com/issues/21529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17920 from dzafman/wip-21382

Erasure code recovery should send additional reads if necessary

Fixes: http://tracker.ceph.com/issues/21382
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: relax cap expected value check

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: notify watcher when updating image metadata

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #18036 from ownedu/wip-fix-asyncrdma-coredump

msg/async/rdma: fix a potential coredump when handling tx_buffers under heavy RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: Update rbd-mirror docs to reflect data pool selection changes

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>

rbd-mirror: Improve data pool selection when creating images

Previously we used the source image's data pool name
unconditionally. There were two problems with that:

1. If a pool with the same name didn't exist locally, creation of the
   local image would fail.
2. If the local pool had a default data pool configured it would be
   ignored.

Change local image creation so it uses the default pool if configured,
and uses the remote pool name only if a pool with that name exists
locally. If neither of those is true, leave the data pool unset.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>