Merge pull request #23748 from pdvian/wip-26944-mimic

mimic: os/bluestore/BlueStore.cc: 1025: FAILED assert(buffer_bytes >= b->length) from ObjectStore/StoreTest.ColSplitTest2/2

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #23772 from ifed01/wip-ifed-perf-reset-mimic

mimic: os/bluestore: bluestore_buffer_hit_bytes perf counter doesn't reset

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23782 from smithfarm/wip-32082-mimic

mimic: mgr balancer does not save optimized plan but latest

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23865 from smithfarm/wip-26907-mimic

mimic: kv: MergeOperator name() returns string, and caller calls c_str() on the temporary

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23559 from dillaman/wip-25121-mimic

mimic: rbd: improved trash snapshot namespace handling

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #23560 from dillaman/wip-24945-mimic

mimic: librbd: validate data pool for self-managed snapshot support

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #23858 from smithfarm/wip-27060-mimic

mimic: run-rbd-unit-tests.sh test fails to finish in jenkin's make check run

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #23521 from pdvian/wip-26845-mimic

mimic: rgw: Limit the number of lifecycle rules on one bucket.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #23686 from pdvian/wip-26921-mimic

mimic: rgw: Fix log level of gc_iterate_entries

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #23855 from smithfarm/wip-26849-mimic

mimic: rgw: civetweb fails on urls with control characters

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23771 from pdvian/wip-26946-mimic

mimic: qa/rgw: patch keystone requirements.txt

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23856 from smithfarm/wip-26980-mimic

mimic: multisite: intermittent failures in test_bucket_sync_disable_enable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23857 from smithfarm/wip-35070-mimic

mimic: cls/rgw: add rgw_usage_log_entry type to ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

kv/KeyValueDB: return const char* from MergeOperator::name()

This avoids a c_str() on a temporary string later down the line, which
avoids a use-after-free.

Fixes: http://tracker.ceph.com/issues/26875
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e9be5dabf256c18a5575d1394e481e4f21ece97d)

librbd: potential race on image create request complete

Fixes: http://tracker.ceph.com/issues/24910
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit d9e5cfaf4a8cd090362892805e837f777b220474)

cls/rgw: ready rgw_usage_log_entry for extraction via ceph-dencoder

Signed-off-by: Vaibhav Bhembre vaibhav@digitalocean.com
(cherry picked from commit a503cec93853235f2f94dbc53eb307618f5a437a)

rgw: bucket sync transitions back to StateInit on OP_SYNCSTOP

the handling for OP_SYNCSTOP->OP_RESYNC is incorrect because it's not
safe to continue incremental sync after SYNCSTOP. any changes between
SYNCSTOP and RESYNC will not be written to the bilog, so incremental
sync cannot sync them

SYNCSTOP now transitions back to StateInit, where
RGWInitBucketShardSyncStatusCoroutine will query the remote bilog info
to see if it's still disabled. if so, it deletes the sync status object
and finishes. otherwise, StateInit will transition back to StateFull to
resync the bucket

Fixes: http://tracker.ceph.com/issues/26895
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d7725a091890cd8e52938c070cc15da3fe57a9e7)

rgw: bucket sync crs mutate rgw_bucket_shard_sync_info in place

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6c6917bca167819e890d1df3aa52b31a8f070e26)

rgw: return errors from RGWInitBucketShardSyncStatusCoroutine

return ENOENT so caller can detect whether sync is stopped

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 72ac0aa1bb5bdffb06b1df8469206b3967a59aa0)

civetweb: update for url validation fixes

The new civetweb 1.10 version in mimic and later is strict on control characters
being url encoded, making url validation more relaxed and passing these through
to rgw where the requisite url validation is done.

Fixes: http://tracker.ceph.com/issues/24158
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 810c1fa8d3a253745d07c4c740296589fa43811a)

Merge pull request #23607 from joscollin/wip-26916-mimic

mimic: doc: Fix broken urls

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23826 from alfredodeza/mimic-rm34309

mimic: ceph-volume: update batch documentation to explain filestore strategies

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc/man/ceph-volume uncomment flags that are now fully supported in batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f8db9faf9836328b920133c5c9c47a6ffacea443)

doc/ceph-volume add filestore docs for batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 188f4ae80fefcdd8d95d719a88d8422b2c120f0c)

doc/ceph-volume uncomment section of newly supported batch operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 02652343c8501a144c5ae2c012f335e8d0f236cf)

Merge pull request #23812 from ceph/backport-mimic-23788

mimic: ceph-volume: tests.functional inherit SSH_ARGS from ansible

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional use Debian ansible os family in deploy.yml

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7a076f5e4f58bc3b346cde306ba949aabbb9b743)

ceph-volume lvm.batch use the abspath attribute for Device objects in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b2a62c2b13d3e079f8020865b7c41f0f6c4b5d71)

ceph-volume lvm.batch remove usage of non-existent method

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6f3094449f46d10260c0a505b39571c35235292e)

ceph-volume tests.functional inherit SSH_ARGS from ansible with synchronize module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8e6269336f6e177f415a51e9fedd36244014215a)

Merge pull request #23810 from alfredodeza/mimic-rm34310

mimic: ceph-volume lvm.batch remove non-existent sys_api property

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.batch remove non-existent sys_api property

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a76fccba5ff0058d0dd1ba47f48b34dba27dd51f)

Merge pull request #23806 from alfredodeza/mimic-ceph-volume-testing

mimic: ceph-volume: run tests without waiting on ceph repos

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23807 from ceph/backport-mimic-23731

mimic: ceph-volume expand auto engine for multiple devices on filestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: reset sys_info.devices in the device_info fixture

In certain environments it was noticed that when running the full test
suite sys_info.devices would get populated correctly from previous
tests. This caused the tests to fail because they did not have the
expected mock data.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit cd63fb0df11c43b429c3025a6be2b8bba10d6e8d)

ceph-volume tests check uuid_name when creating an lv

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 494ece30994c9d80c9f4afc72213515f8ed63698)

ceph-volume tests add verifications for create_vg helper

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 997d922bbd68f59ebbacc3cef2ca3231f44339eb)

ceph-volume tests add verifications for extend_vg helper

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb1e1c61bd5944ad1fd93e86385d26e5b0af38cb)

ceph-volume lvm.batch update module to use Device object on strategy detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ebbb571d2f62dd6c6a38ced51a838ad4a099b817)

ceph-volume lvm.batch.filestore use new Device objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 369bb740a0c52b075a81fb26b368ddf936319e24)

ceph-volume lvm.batch.bluestore use new Device objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c6f4e107a1660dfee194c4344532bd9560ec7b37)

ceph-volume tests.util verify ValidDevice validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 649903bdabe3ea04c499c3dbe6d1e9e7b8f82a23)

ceph-volume util.templates allow more space for long paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 57f76e36abffee0f8f7d81b7d16e72de148a065c)

ceph-volume lvm.strategies create mixed type filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e4902ced1bb26a7f55639c747e6aed7a9fd97b05)

ceph-volume lvm.batch add new strategy validators for LVM devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f67732cd9fb11830cc134deaa3ec7ebc107fa1fe)

ceph-volume util add a new Device class validator for args

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5be2794b7fe42e3a4505ec9f406dd5ae46595bd9)

ceph-volume lvm.batch use new Device class and validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d74ce8e54da4878c01a68adf598685045fafba72)

ceph-volume api.lvm new utility to extend a volume group for one or more devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 21df60294977f4dd2dab218df77a04b326fd8b19)

ceph-volume api.lvm allow uuid suffixes in vg/lv creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 028316414227103457d96ba3ab88fbebeb27c1dc)

ceph-volume: allow `lvm zap` to zap many devices at once

Supports passing many devices, partitions or lvs to
ceph-volume lvm zap.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit d1b929395b4db3c021d3e6df02e020d72962ab3d)

ceph-volume devices.lvm zap should skip when lv_uuid doesn't exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 447be1c9cc3a8f83687f8c0d2378dfd175e03959)

ceph-volume: device.Device should ignore pvs with no vg_name

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9b4f9471ed258bc5a06ec48e1d2ebd56f02a68cd)

ceph-volume: tests for util.device.Device

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 947266e16ae3a5e9bdaa7fbf326a0e1b96f16e0f)

ceph-volume: adds sys_info constant to hold system information

This is only currently used to hold information about devices
from /sys/block in the util.Device class in the sys_api property.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 035f1a4092bba44cce95cabc1e5a12e3938e1150)

ceph-volume util create a device module for identifying devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f955a7e86404a879ae2bdfb696bdcefa51cbdff)

ceph-volume: retry when 'vagrant up' fails

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2c88e70877dcc2a04a69be4ab9c4834d5ced6198)

ceph-volume: run tests without waiting on ceph repos

This provides a new playbook that installs ceph on all nodes and
generates the ceph config. Then it will rsync ceph-volume from the
control node, or jenkins slave, to all testing vms before completing
the deployment. This means we can run tests on PRs without waiting
for repos to be built for the branch in the PR.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 6d712ef959f7ea5385bc6bf9b9ade60662940369)

Merge pull request #23786 from ceph/backport-mimic-23532

mimic: ceph-volume: expand auto engine for single type devices on filestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23784 from ceph/backport-mimic-23489

mimic: ceph-volume tests/functional add mgrs daemons to lvm tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23780 from ceph/backport-mimic-23448

mimic: ceph-volume: `lvm batch` allow extra flags (like dmcrypt) for bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap loop over all vgs from a pv to destroy them

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9066792764edcab1d0c2b3178d6f8f9784e30f85)

ceph-volume tests.util ensure dmcrypt_close will skip non-existent paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7a5ee22437a05fc809bffde97c8e3401f7ba564d)

ceph-volume util.encryption if crypt path does not exist, skip it

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 635f8e9e2ecdad5c54a599029bdad74e88fad600)

ceph-volume lvm.zap unmount lvs and close dmcrypt on partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 549109c1de2b43f31747828cb5703f9db221dd1a)

ceph-volume tests.functional batch add filestore sceanrios to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ac00e722551e759fc3c1c157ddc89933fb3ac275)

ceph-volume lvm.zap iterate over multiple lvs on a pv

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 21d4f4afcefb6b2510ae67d1ca5e3bf1d510cdfb)

ceph-volume tests.functional add xenial batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfd1bfc2422318edb5a87739b70f93bfc057dc02)

ceph-volume tests.functional add bluestore,filestore playbooks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 26ca6c129a70cd28aa0ed2dae552fb91a251ad21)

ceph-volume tests.functional add centos7 filestore batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a4ff2bff502f1837f696e5f8ebe88f7586d1a83f)

ceph-volume tests.functional add centos7 bluestore batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 544e364e8c3c01633333ddc11c29302965e0616c)

ceph-volume tests.util assert new as_int,as_float helper methods

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d2c181fc70bf20fb17cf41437fb207afc5dd7656)

ceph-volume tests verify new utility for journal sizes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3adf13f868f270293fc307b58786155341470a9d)

ceph-volume util.prepare add a helper to get journal sizes from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4c0b4df1df2db85209c13ff66ae7a67524856c2d)

ceph-volume util.disk add as_int and as_float helpers for Size objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ff7054116c50cb149de207280b0469bf51655728)

ceph-volume lvm.strategies fix a bug where incorrect template was used

The template wouldn't render the total OSDs and the removed section was
repeated

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f473bce7addbd31c7c976d2f5f1aef82696d4da1)

ceph-volume lvm.strategies use get_journal_size helper for filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 01aebfc6a075bc91d0dc345f9cd953e3a4259015)

ceph-volume lvm.strategies update filestore use of Size objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f29ec4c6077e9ab36a9eb5ac030b17cd3a47567c)

ceph-volume lvm.strategies update bluestore use of Size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 75e9729b76594ce1b6504c10e613a91aea82477d)

ceph-volume lvm.strategies create single type filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit af99401cad2c25b02e6ea0df4554d80afbad2b9b)

ceph-volume tests.util verify that Size object don't change state

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5a54ab17602d159bdd29fe82a25f4f026b0fe6c8)

ceph-volume tests.util update to reflect Size object changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7816fa9f90985d03c17fda31093252fc66a48146)

ceph-volume util.disk do not modify self instance of Size objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 00dba2e83ad80cd41b6b29efa642e2da826c2124)

ceph-volume lvm.strategies use new template location in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8f7bca7ba1f74de125a3789133e06c6cf0c083c1)

ceph-volume util create a templates module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 783e58472c55d378773404050d95470d3bb8bbc2)

doc/rados update journal size defaults to 5GB

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 35a48728ae160a347471cf5b461c552b85323a1b)

ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2b7a22ef10e38a7bb4d08c9ebe9736e6452b005f)

Merge pull request #23109 from pdvian/wip-24931-mimic

mimic: client: add inst to asok status output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23180 from ukernel/mimic-24856

mimic: mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23196 from pdvian/wip-25035-mimic

mimic: mds: print mdsmap processed at low debug level

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23565 from pdvian/wip-26905-mimic

mimic: mds: mark beacons as high priority

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22670 from votdev/mimic_bug_24080

mimic: mgr/dashboard: Prevent RGW API user deletion

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23111 from smithfarm/wip-18163-mimic

mimic: tools: ceph-detect-init: support SLED

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #23335 from smithfarm/wip-25178-mimic

mimic: rados: not all exceptions accept keyargs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: balancer: deepcopy best plan - otherwise we get latest

Fixes: http://tracker.ceph.com/issues/27000
Signed-off-by: Stefan Priebe <s.priebe@profihost.ag>
(cherry picked from commit 9852b49dbb4be80b11e3f5fe172c6c9f7f3d4acd)

ceph-volume lvm.batch move import statement to the top

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c460f18d0930e3dbe1cfb97f2d399e4a8a2ceed9)

ceph-volume lvm.batch allow extra flags for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4149d664fa51eb7b2f900370c2bd000b1a5b1edc)

Merge pull request #23777 from ceph/backport-mimic-23075

mimic: ceph-volume batch command

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api update lvm sizing tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8d4525a59d1d7eb1d571848698c69cace29bfa53)

ceph-volume api.lvm convert and use extents for sizing calculations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 99e034d7eed9508ee016431c040e3fddca9bca69)

ceph-volume lvm.batch update help docs from auto to batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 226a5389ab3379943d3af35e3e1a86481a3980b2)