Merge pull request #18082 from Liuchang0812/wip-21624

os/bluestore: make BlueStore opened by start_kv_only umountable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18083 from Liuchang0812/wip-21625

tool: ceph-kvstore-tool doesn't umount BlueStore properly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18110 from ceph/wip-volume-api-change

ceph-volume lvm api refactor/move

Reviewed By: Andrew Schoen <aschoen@redhat.com>

Merge pull request #17380 from ceph/wip-21189

mgr/status: format byte quantities in base 2 multiples

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17746 from elliswuuuuuuuu/wip-prometheus

ceph-mgr: can not change prometheus port for mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18089 from liewegas/wip-localpool-min-size

mgr/localpool: default to 3x; allow min_size adjustment

Reviewed-by: John Spray <john.spray@redhat.com>

mgr/localpool: fix rule selection

The 'osd pool create' arg parsing is broken; the rule name for
'ceph osd pool create $name $numpgs replicated $rulename' is passed
via the erasure_code_profile param.  Too many req=false options
without a way to disambiguate them.

Work around it by passing both 'rule' and 'erasure_code_profile'
keys, so that if/when the hack in OSDMonitor.cc is removed it will
still work.  Blech.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests move lvm api tests into its new test module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import for test_listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import for api tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import for activate tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests refactor lvm api import in conftest

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume devices.lvm refactor importing api to come from api/lvm.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume move lvm/api.py to api/lvm.py so disk can consume it

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume create an api module to hold common api modules

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18048 from badone/wip-boost-mirrors

make-dist,cmake: Try multiple URLs to download boost before failing

Reviewed-by: Kefu Chai <kchai@redhat.com>

make-dist,cmake: Try multiple URLs to download boost before failing

Remove SPOF during boost download for make-dist and later cmake > 3.7

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #18097 from dillaman/wip-21663

qa/workunits/rbd: fixed variable name for resync image id

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18052 from xiexingguo/wip-clear-pin-stats-invalid

osd/PrimaryLogPG: clear pin_stats_invalid bit properly on scrub-repair completion

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18061 from xiexingguo/wip-crc

osd/PrimaryLogPG: do not set data/omap digest blindly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18066 from shun-s/wip-objectstore-tool

tool: change default objectstore from filestore to bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18078 from liewegas/wip-21614

qa/suites/rados/singleton/all/recover-preemption: handle slow starting osd

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18095 from ceph/wip-bz1491250

ceph-volume lvm list

Reviewed by: Andrew Schoen <aschoen@redhat.com>

doc/ceph-volume add lvm/list to the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume create the reference for the list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume add create and list to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a unit test module for listing features

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rbd: fixed variable name for resync image id

Fixes: http://tracker.ceph.com/issues/21663
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18006 from adamwg/awg/rbd-mirror-default-data-pool

rbd-mirror: Improve data pool selection when creating images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17721 from amitkumar50/cov-librbd-4

librbd: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17696 from amitkumar50/cov-librbd-3

librbd: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17660 from gmayyyha/striping-feature-21360

librbd: fix don't send get_stripe_unit_count if striping is not enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17461 from PCzhangPC/bench_rw_test

 rbd:introduce rbd bench rw(for read and write mix) test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17576 from ceph/wip-rm-1-minsize

qa/tests/rados: Remove unsupported 2-size-1-min-size config

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume tests create a small factory fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests ensure Volume.as_dict gets populated correctly

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.listing initial take on listing command

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api be able to return a dictionary repr of a Volume object

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process allows to fully mute terminal output on commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.api use a helper to get an lv from an arg

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm add listing module to main

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18053 from ownedu/wip-fix-async-rdma-tx-buffer-leakage

msg/async/rdma: fix Tx buffer leakage that can introduce "heartbeat no reply"

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Alex Mikheev <alexm@mellanox.com>

tool: ceph-kvstore-tool doesn't umount BlueStore properly

Fixes: http://tracker.ceph.com/issues/21625
Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #17997 from ivancich/wip-bring-in-latest-dmclock

osd: bring in latest dmclock library updates

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18003 from cbodley/wip-build-options-move

common: use move instead of copy in build_options()

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18029 from jdurgin/wip-kvstore-tool

ceph-kvstore-tool: copy to different store type and cleanup properly

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/localpool: optionally adjust min_size too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/localpool: default to 3x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17999 from cbodley/wip-21576

rgw: include SSE-KMS headers in encrypted upload response

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

os/bluestore: make BlueStore opened by start_kv_only umountable

ceph-kvstore-tool use start_kv_only to debug the kvstore. we
will get a crash when we try to umount bluestore in kvstore-tool.

Fixes: http://tracker.ceph.com/issues/21624
Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #18063 from oritwas/wip-rgw-21610

doc: replace region with zonegroup in configure bucket sharding section

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18077 from joscollin/wip-drop-unused-function-declaration

os/kstore: Drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18046 from mdw-at-linuxbox/master-awsv4-sort

radosgw: fix awsv4 header line sort order.

Merge pull request #18075 from dragonylffly/wip-remove-unused-declaration

os/bluestore: drop unused function declaration

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/recover-preemption: handle slow starting osd

The OSD may not be marked up yet; set the config via the admin
socket.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18067 from trociny/wip-qa-pg_num

qa/workunits/rbd: pool create may fail for small cluster

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18076 from joscollin/wip-contributor-list-maintenance-guide

mailmap: Fixed maintenance guide URL

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #17735 from jcsp/wip-mgr-perf-interface

mgr: common interface for TSDB modules

Reviewed-by: My Do <mhdo@umich.edu>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18069 from trociny/wip-affiliation

mailmap: update affiliation for Mykola Golub

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

os/kstore: Drop unused function declaration

Dropped unnecessary function declaration _do_transaction() in KStore.h

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap: Fixed maintenance guide URL

Fixed the Contributors List Maintenance Guide URL in .mailmap, .organizationmap, .githubmap and .peoplemap

Signed-off-by: Jos Collin <jcollin@redhat.com>

mailmap, organizationmap: update affiliation for Mykola Golub

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

os/bluestore/BlueStore: remove unused declaration

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #18022 from neha-ojha/wip-21566

osd: hold lock while accessing recovery_needs_sleep

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18059 from Liuchang0812/remove-duplicated-func

osd: remove duplicated function ec_pool in pg_pool_t

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17913 from rzarzynski/wip-bs-bulky_disk_releases

os/bluestore: release disk extents in bulky manner

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed: Sage Weil <sage@redhat.com>

Merge pull request #17752 from XinzeChi/wip-force-recovery

osd: set min_version to newest version in maybe_force_recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17728 from ygtzf/cleanup-unused-parameter

mon: cleanup unused option mon_health_data_update_interval

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #16843 from amitkumar50/cov-radosstriper

rados: Intializing members class StriperTest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17912 from ceph/docs-min-size

docs: Do not use "min size = 1" as an example

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17931 from jcsp/wip-21518

mgr/zabbix: fix div by zero

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #18040 from joscollin/wip-guide-submitting-patches-typo

doc: Fix typo and URL

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18060 from dragonylffly/wip-revise-organization

doc: update Li Wang Affiliation

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: pool create may fail for small cluster

ceph osd pool create test 100
Error ERANGE:  pg_num 100 size 3 would mean 648 total pgs, which exceeds max 600 (mon_max_pg_per_osd 200 * num_in_osds 3)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from alex-mikheev (Alex Mikheev), to use a single
atomic counter for inflight Tx CQEs.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #18043 from dillaman/wip-21567

librbd: snapshots should be created/removed against data pool

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18042 from dillaman/wip-21529

librbd: image-meta config overrides should be dynamically refreshed

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18026 from dillaman/wip-21535

rbd-mirror: sync image metadata when transfering remote image

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #17992 from dillaman/wip-21558

librbd: hold cache_lock while clearing cache nonexistence flags

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Addressing CR comments from tchaikov (Kefu Chai).

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

tool: change default objectstore from filestore to bluestore
  as Luminous gets released, it's better to switch default objectstore from filestore to bluestore.

Signed-off-by: Song Shun <song.shun3@zte.com.cn>

Merge pull request #17989 from mkogan1/wip-rgw-21151

rgw: update vstart.sh to support rgw ssl port notation : '--rgw_port 443s'
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

doc: replace region with zonegroup in configure bucket sharding section

Fixes: http://tracker.ceph.com/issues/21610
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge PR #18041 into master

* refs/remotes/upstream/pull/18041/head:
qa: relax cap expected value check

Merge PR #17990 into master

* refs/remotes/upstream/pull/17990/head:
mds: open purge queue when transitioning out of standby replay
mds: always re-probe mds log when standby replay done

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #17935 into master

* refs/remotes/upstream/pull/17935/head:
ceph_volume_client: fix setting caps for IDs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17958 from gaosibei/branch1

RGW: fix a bug about inconsistent unit of comparison

osd/PrimaryLogPG: do not set data/omap digest blindly

As bluestore has bulitin csum, we generally no longer generate
object data digest for now. The consequence is that we should
handle data/omap digest more carefully to make certain ops,
such as copy_from/promote, to work properly since they heavily
relies on data digest for data transfer correctness.

Example of failure:
http://pulpito.ceph.com/xxg-2017-09-30_11:46:34-rbd-master-distro-basic-mira/1690609/

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: revise organization

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

osd: remove duplicated function ec_pool in pg_pool_t

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #15199 from xiexingguo/wip-object-logic-size

osd: fine-grained statistics of logical object space usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #16884 from liewegas/wip-20919

osd/PrimaryLogPG: send requests to primary on cache miss

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #17955 from asomers/bin_bash2

test: fix bash path in shebangs (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17985 from dzafman/wip-21327

ceph-objectstore-tool: "$OBJ get-omaphdr" and "$OBJ list-omap" scan all pgs instead of using specific pg

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18005 from jcsp/wip-21577

tools: update monstore tool for fsmap, mgrmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18015 from tchaikov/wip-kill-warnings

osd,os/bluestore: kill clang analyzer warnings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18018 from tchaikov/wip-ceph-disk-cleanup

ceph-disk: more precise error message when a disk is specified

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18034 from tchaikov/wip-options

common/options: pass by reference and use user-literals for size

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

msg/async/rdma: fix Tx buffer leakage which can introduce "heartbeat no
reply" due to out of Tx buffers, this can be reproduced by marking some
OSDs down in a big Ceph cluster, say 300+ OSDs.

rootcause: when RDMAStack wants to delete faulty connections there are
chances that those QPs still have inflight CQEs, thus inflight Tx
buffers; without waiting for them to complete, Tx buffer pool will run
out of buffers finally.

fix: ideally the best way to fix this bug is to destroy QPs gracefully
such as to_dead(), we now just reply on the number of Tx WQE and CQE to
avoid buffer leakage; RDMAStack polling is always running so we are safe
to simply bypass some QPs that are not in 'complete' state.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

doc: Fix URL in Licensing

Fixed the unnecessary URL format in the text. Modify the URL formatting to highlight only the file name seems better.

Signed-off-by: Jos Collin <jcollin@redhat.com>