Merge pull request #17539 from tchaikov/wip-21204

mon,option: set default value for mon_dns_srv_name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17549 from iliul/fix-texttable-align

tools: Modify "rados df" header's alignment

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17096 from ceph/wip-ansbile-fixes

[qa/tests]: misc ceph-ansible fixes and udpate

Merge pull request #17513 from Liuchang0812/wip-max-avail-in-df

mon: incorrect MAX AVAIL in "ceph df"

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #17533 into master

* refs/remotes/upstream/pull/17533/head:
qa: log ceph-fuse kill/cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17506 into master

* refs/remotes/upstream/pull/17506/head:
mds: reset Formatter on error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17466 into master

* refs/remotes/upstream/pull/17466/head:
qa: whitelist expected MDS_READ_ONLY for kcephfs test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17456 into master

* refs/remotes/upstream/pull/17456/head:
mds: print subtrees only after merge

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17452 into master

* refs/remotes/upstream/pull/17452/head:
Make standby-replay mds not migrate empty subtree when trimming cache
Make standby-replay mds avoid initiating subtree export

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17384 into master

* refs/remotes/upstream/pull/17384/head:
ceph.in: validate service glob

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16752 into master

* refs/remotes/upstream/pull/16752/head:
client: NULL check before append value to bufferlist

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

Stop the mgr on node which is not client, this will ensure
the client.0's mgr is active mgr for workunit to work.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #16272 from liewegas/wip-balancer

mgr/balancer: mgr module to automatically balance PGs across OSDs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17403 from ceph/wip-bz1485011

ceph-volume Persist non-lv devices for journals

Reviewed-by: Andrew Schoen aschoen@redhat.com

Merge pull request #17603 from liewegas/wip-migration-twiddle

doc: restructure bluestore migration insructions

Reviewed-by: David Galloway <dgallowa@redhat.com>

doc: restructure bluestore migration insructions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17602 from ceph/wip-blumig-doc

doc: Updates to bluestore migration doc

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rados/operations/bluestore-migration: Add instruction for evacuating
host

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #17011 from liewegas/wip-doc-bluestore-migration

doc/rados/operations/bluestore-migration: update docs a bit

Merge pull request #17582 from ivancich/wip-fix-mclock-unittests

test/common: unittest_mclock_priority_queue builds with "make" command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados/operations/bluestore-migration: Fix typo in bluestore migration doc

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #17569 from liewegas/wip-21293

Revert "os/bluestore: pass strict flag to bluestore_blob_use_tracker_t::equal()"

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17352 from liewegas/wip-21171

os/bluestore: drop deferred_submit_lock, fix aio leak

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #17593 from shashalu/fix-clean-up-apply_epoch

rgw: drop unused function apply_epoch

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17590 from xiaoxichen/minor_doc_fix

doc: fix typo on specify db block device

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17573 from jcsp/wip-21292

mgr/restful: cleaner message when not configured

Reviewed-by: Boris Ranto <branto@redhat.com>

rgw: drop unused function apply_epoch

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

doc: fix typo on specify db block device

Should use block.db instead of block-db

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

Test ceph-mgr RESTful api

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add additional variables as required by ceph-ansible for upstream
installation.

possible revert if those are made default
issue: https://github.com/ceph/ceph-ansible/issues/1834

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #17536 from ceph/wip-cd-rbd-pool

qa/tests: fix rbd pool creation for systemd tests

Merge pull request #16826 from ceph/wip-cd-upgrade

qa/tests: add ceph-deploy upgrade tests

test/common: unittest_mclock_priority_queue builds with "make" command

Although it would be built with "make tests",
unittest_mclock_priority_queue was not built with a simple "make".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17415 from wangshuguangbj/bug_mon_mdsmonitor_assert

mon/MDSMonitor: fix a bug at preprocess_beacon

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17454 from xiexingguo/wip-vs

src/vstart.sh: default os to filestore for FreeBSD

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #17522 from liupan1111/wip-fix-disk-error

os/bluestore: report error and quit correctly when disk error happens.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>

Merge pull request #17525 from jcsp/wip-21249

mon: fix `osd out` clog message

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/aio: handle short return from io_submit

io_submit may return a value less than nr, indicating that only some of
the provided iocbs were queued.  If that happens we should loop, not
return and silently drop those aios on the floor.

Signed-off-by: Sage Weil <sage@redhat.com>

Add workaround for http://tracker.ceph.com/issues/20950

mgr bootstrap key differs on disk, rewrite the new key
using auth get.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add kraken to luminous upgrade case

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add kernel tar/untar and systemd task

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Restart services after upgrade

This seems to be an issue and should be fixed in our systemd scripts,
need to discuss more on this.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

since the roles are mapped inside ceph-deploy, store the roles that
are mapped and use the new mapped role for upgrades during later
stage.

eg: mon.a is mapped to mon.mira002 during install, store this mapping
and durig upgrade map it back to appropriate name to find the hostname
with that role

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add doc string for missing ceph-deploy-branch

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use mon.a as ceph-admin node, the previous get_first_mon returns
a sorted(mons)[0] as first mon.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

use elif instead of else to handle cases for jewel install

jewel needs neither filestore or bluestore as an option, so provide none
when running with jewel branch.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Jewel to luminous upgrade scenario using ceph-deploy

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add upgrade functionality using ceph-deploy

This is to test for customer like upgrade scenarios and to find
any issues that may be related to systemd, packaging etc

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Fix old flake8 issues

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Add option to skip mgr install for old release

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #17581 from liewegas/wip-migration-fixups

doc/rados/operations/bluestore-migration: typos

doc/rados/operations/bluestore-migration: typos

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17322 from liewegas/wip-post-luminous-mon

mon: remove pre-luminous compat cruft (2 of many)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #17578 from liewegas/wip-integration

script/build-integration-branch: check errors

Merge pull request #17577 from ktdreyer/gitignore-debian-patches

.gitignore: allow debian .patch files

Reviewed-by: Sage Weil <sage@redhat.com>

script/build-integration-branch: check errors

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: allow debian .patch files

The Ubuntu packaging layout with git-buildpackage assumes a
"debian/patches/" directory with several .patch files in it.

When upstream's .gitignore tells Git to ignore .patch files, we have to
edit that line out downstream. When we forget to do that downstream, it
can lead to missing patches and broken downstream builds.

Allow patches in the /debian/patches directory so it's easier to
maintain an Ubuntu package based on upstream's Git repo.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #17548 from yangdongsheng/mailmap

mailmap, organizationmap: add Dongsheng, Liuzhong, Pengcheng, Yang Affiliation

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

organization: add Liuzhong Chen, Pengcheng Zhang, Yang Wang

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

mailmap: add Dongsheng Yang, Liuzhong Chen, Pengcheng Zhang, Yang Wang

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #17540 from heyingstar/wip-update-organization

.mailmap, .organizationmap: add Fufei, Mingqiao and Ying Affiliation

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mgr/restful: cleaner message when not configured

RuntimeError+backtrace prints should be for
genuinely exceptional exceptions.  For these
known cases, we should just print the message cleanly.

http://tracker.ceph.com/issues/21292
Signed-off-by: John Spray <john.spray@redhat.com>

Revert "os/bluestore: pass strict flag to bluestore_blob_use_tracker_t::equal()"

This reverts commit 5070999169b5733983fb74d955cb764a9066c3d8.

The existing blob may have a vector-based au because it was, say, split, while the local one
may not (because it not needed).  Use a sloppy comparison.

Fixes: http://tracker.ceph.com/issues/21293
Signed-off-by: Sage Weil <sage@redhat.com>

add Ying He,Mingqiao Wu,Fufei Shang to mailmap and organizationmap

Signed-off-by: Ying He <heyingbj@inspur.com>

qa/workunits/*: require-osd-release mimic, not luminous

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #16998 from trociny/wip-20860

rbd-mirror: update asok hook name on image rename

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17300 from Songweibin/wip-mirror-peer-set

librbd: should not set self as remote peer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17218 from PCzhangPC/benchrw

rbd: use different logic to disturb thread's offset in bench seq test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17187 from trociny/wip-mock-valgrind

test/librbd: fix valgrind memory leak warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #16597 from PCzhangPC/fix6

rbd: snap limit should't be set smaller than the number of  existing snaps

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

tools: Modify "rados df" header's alignment

Signed-off-by: iliul <liul124@chinaunicom.cn>

Merge pull request #17018 from alex-mikheev/wip_rdma_rxpool_2

msg/async/rdma: refactor rx buffer pool allocator

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #17504 from dillaman/wip-21251

qa/suites/rbd: miscellaneous test fixes

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17532 from dillaman/wip-21179

rbd: support iterating over metadata items when listing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17509 from dillaman/wip-21217

test/cls_rbd: mask newer feature bits to support upgrade tests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17499 from dillaman/wip-21247

cls/rbd: metadata_list not honoring max_return parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mon,monmap: use new style config opts

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon,option: set default value for mon_dns_srv_name

Fixes: http://tracker.ceph.com/issues/21204
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #17340 into master

* refs/remotes/upstream/pull/17340/head:
mds: void sending cap import message when inode is frozen
client: fix message order check in handle_cap_export()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #16778 into master

* refs/remotes/upstream/pull/16778/head:
mds: fix return value of MDCache::dump_cache
mds: new cap message flags indicate if there is pending capsnap
mds: properly do null snapflush part2
mds: track snap inodes through sorted map
mds: properly drop wrlock when finishing snapflush

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Add mgr create to fix single node test to reach healthy state

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

create the rbd pool right after install

rbd pool should exist for many rbd tests to work properly, create
the pool right after install is successful.

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #17291 into master

* refs/remotes/upstream/pull/17291/head:
mds: fix 'dirfrag end' check in Server::handle_client_readdir

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge PR #17289 into master

* refs/remotes/upstream/pull/17289/head:
osd, mds, tools: drop the invalid comment and some unused variables

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #17219 into master

* refs/remotes/upstream/pull/17219/head:
mds: fix StrayManager::truncate()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #17505 from liewegas/wip-20910

qa/objectstore/bluestore*: less debug output

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17459 from xiexingguo/wip-bs-tracked-key

os/bluestore: add bluestore_prefer_deferred_size_hdd/ssd to tracked keys

Reviewed-by: Pan Liu <liupan1111@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17463 from tchaikov/wip-ceph-tell-mds-star

ceph: fixes for "tell <service>.*" command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Chang Liu <liuchang0812@gmail.com>

Merge pull request #17503 from liewegas/wip-21250

os/bluestore/BlueFS: prevent _compact_log_async reentry

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Varada Kari <varada.kari@gmail.com>

pybind/mgr/balancer: make auto mode work

(with upmap at least)

Signed-off-by: Sage Weil <sage@redhat.com>

src/pybind/mgr/balancer/module.py: improve scoring method

* score lies in [0, 1), 0 being perfect distribution
* use shifted and scaled cdf of normal distribution
  to prioritize highly over-weighted device.
* consider only over-weighted devices to calculate score

Signed-off-by: Spandan Kumar Sahu <spandankumarsahu@gmail.com>

pybind/mgr/balancer: make 'crush-compat' sort of work

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/balancer: rough framework

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyOSDMap: OSDMap.map_pool_pgs_up, CRUSHMap.get_item_name

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/PyOSDMap: get_crush, find_takes, get_take_weight_osd_map

These let us identify distinct CRUSH hierarchies that rules distribute
data over, and create relative weight maps for the OSDs they map to.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: rule_has_take

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: fix output arg for find_{takes,roots}()

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: refactor get_rule_weight_osd_map to work with roots too

Allow us to specify a root node in the hierarchy instead of a rule.
This way we can use it in conjunction with find_takes().

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/balancer: do upmap by pool, in random order

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/balancer: add balancer module

- wake up every minute
- back off when unknown, inactive, degraded
- throttle against misplaced ratio
- apply some optimization step
  - initially implement 'upmap' only

Signed-off-by: Sage Weil <sage@redhat.com>