radosgw-admin: 'sync error trim' loops until complete

a single call to time_log_trim() will trim a maximum of 1000 entries.
now loops until a return code of -ENODATA indicates that all entries
in the given range were trimmed

an optional --trim-delay-ms option can be used to limit the frequency
of osd ops

Fixes: http://tracker.ceph.com/issues/24873
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 326971931d15250ed26d9213ad14e76e4ef402bc)

Conflicts:
src/rgw/rgw_admin.cc
- some control commands not backported from master
- added #include <thread> for std::this_thread and #include <chrono> for
  std::chrono::milliseconds (in master and mimic, rgw_admin.cc must be getting
  these includes via some other header)

Merge pull request #23216 from b-ranto/wip-mgr-prom-bp-luminous

luminous: Sync the prometheus module

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #23461 from pdvian/wip-26838-luminous

luminous: mgrc: enable disabling stats via mgr_stats_threshold

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #23566 from pdvian/wip-26908-luminous

luminous: kv/KeyValueDB: return const char* from MergeOperator::name()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23579 from VictorDenisov/backport_26934

luminous: osd/OSDMap: fix apply upmap segfault

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #24039 from ceph/wip-s3a-fix-mirror

luminous: qa/tasks:  s3a fix mirror

Merge pull request #23211 from neha-ojha/wip-24988-luminous

luminous: osd: Limit pg log length during recovery/backfill so that we don't run out of memory

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23761 from k0ste/luminous_backports3

luminous: docs: radosgw: ldap-auth: fixed option name 'rgw_ldap_searchfilter'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

qa/tasks: update mirror link for maven, the original mirror no longer exists

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
(cherry picked from commit 13e100259e3658fcfd2123474cb8ea8498ac3fb6)

Merge pull request #23621 from joscollin/wip-26917-luminous

luminous: doc: Fix broken urls

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23967 from alfredodeza/luminous-rm35535

luminous: ceph-volume: batch tests for mixed-type of devices

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

osd/PrimaryLogPG: avoid dereferencing invalid complete_to

For the auto-repair (EIO caused) case, we will not reinitialize
**complete_to** (because last_complete is equal to last_update!)
and hence there is chance that **complete_to** should aleady
point to **log.end()** before we call recover_got.

We could simply drop it here as we (already) logged the **complete_to**
iterator change in a more compatible way a few lines below.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 69a2cc35840939436da09691ca62476d7f599de4)

ceph-volume lvm.batch use 'ceph' as the cluster name with filestore

Custom cluster names are currently broken on ceph-volume, should get
addressed with http://tracker.ceph.com/issues/27210 which is out of
scope for these changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c1481799a254eceb05258313aafd532e062ea0bc)

ceph-volume tests/functional update filestore xenial test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a096a016ccd0acc86ce816aba8ce95f6e3b68c11)

ceph-volume tests/functional update bluestore xenial test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89e52dd19792072919bef6ba2f2ab50d7b0ff4ed)

ceph-volume tests/functional update filestore centos7 test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a5ec54207a02c4bb85dc8226890def18da45e2ff)

ceph-volume tests/functional update bluestore centos7 test vars

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2549d5737283b39ecf5b0af98c73ffeb18033c36)

ceph-volume lvm.api use double -f flags when calling pvremove

Fairly destructive, just like everything else when zapping a device.
This is required in the case of double UUIDs detected, something that
surfaced when testing with a loop device to create an nvme (the loop
device ends up with the same UUID as the nvme).

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c1cbc407c095e0bc3e25a8ff1519b4909ec519cc)

ceph-volume tests/functional remove duplicate test playbooks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a008fe1385e4440f510eab97470a5d4e56a69402)

ceph-volume tests/functional setup playbook for nvme devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 71db509df0c4a14598010568f8bb078f6b2841ab)

ceph-volume tests/functional single (test) playbook to zap and redeploy batch operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 23330c0107a9f9ee436338c112b1852fd36e34a3)

ceph-volume tests/functional create a noop playbook to allow symlinking always to setup.yml

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 860e66083fe769714a10076fa101b94507e47169)

ceph-volume tests/functional add mixed-type scenarios to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0d5b90bb1fbf30c6e33dbf7e10a3d0ba053297ae)

ceph-volume tests/functional add CentOS7 bluestore mixed-type scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 799ce4508d9ed166ef6447daa21e0564262fbafe)

ceph-volume tests/functional add CentOS7 filestore mixed-type scenarios

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89bbcff659eed8f2cfa26fb93a0c12c7ff83e83b)

Merge pull request #23913 from yuriw/wip-35541-luminous

qa/tests:  added v12.2.8 to the mix

Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/tests:  added v12.2.8 to the mix

Fixes: http://tracker.ceph.com/issues/35541
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

osd/PrimaryLogPG.cc: limit trimming at can_rollback_to

This change is motivated by the failures seen in the multimds suite,
where we hit assert(s <= can_rollback_to), while trimming the log in ec
pools.

This is due to the fact that we had removed limits on the trim_to value to
address https://tracker.ceph.com/issues/23979.

But, seems that this could be dangerous for ec pools. So, keep the
can_rollback_to limit, while calculating the trim_to value.

Fixes: http://tracker.ceph.com/issues/21416
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 4b5c6b88d444e2173e716fe4890717873c8dc8e5)

12.2.8

Merge pull request #23825 from alfredodeza/luminous-rm34309

luminous: ceph-volume: update batch documentation to explain filestore strategies

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc/man/ceph-volume uncomment flags that are now fully supported in batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f8db9faf9836328b920133c5c9c47a6ffacea443)

doc/ceph-volume add filestore docs for batch

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 188f4ae80fefcdd8d95d719a88d8422b2c120f0c)

doc/ceph-volume uncomment section of newly supported batch operations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 02652343c8501a144c5ae2c012f335e8d0f236cf)

Merge pull request #23813 from alfredodeza/luminous-rm34311

luminous: ceph-volume: tests.functional inherit SSH_ARGS from ansible

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional use Debian ansible os family in deploy.yml

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7a076f5e4f58bc3b346cde306ba949aabbb9b743)

ceph-volume lvm.batch use the abspath attribute for Device objects in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b2a62c2b13d3e079f8020865b7c41f0f6c4b5d71)

ceph-volume lvm.batch remove usage of non-existent method

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6f3094449f46d10260c0a505b39571c35235292e)

ceph-volume tests.functional inherit SSH_ARGS from ansible with synchronize module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8e6269336f6e177f415a51e9fedd36244014215a)

Merge pull request #23811 from alfredodeza/luminous-rm34310

luminous: ceph-volume lvm.batch remove non-existent sys_api property

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.batch remove non-existent sys_api property

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a76fccba5ff0058d0dd1ba47f48b34dba27dd51f)

Merge pull request #23809 from alfredodeza/luminous-ceph-volume-testing

luminous: ceph-volume: run tests without waiting on ceph repos

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: retry when 'vagrant up' fails

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2c88e70877dcc2a04a69be4ab9c4834d5ced6198)

ceph-volume: run tests without waiting on ceph repos

This provides a new playbook that installs ceph on all nodes and
generates the ceph config. Then it will rsync ceph-volume from the
control node, or jenkins slave, to all testing vms before completing
the deployment. This means we can run tests on PRs without waiting
for repos to be built for the branch in the PR.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 6d712ef959f7ea5385bc6bf9b9ade60662940369)

Merge pull request #23808 from ceph/backport-luminous-23731

luminous: ceph-volume expand auto engine for multiple devices on filestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: reset sys_info.devices in the device_info fixture

In certain environments it was noticed that when running the full test
suite sys_info.devices would get populated correctly from previous
tests. This caused the tests to fail because they did not have the
expected mock data.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit cd63fb0df11c43b429c3025a6be2b8bba10d6e8d)

ceph-volume tests check uuid_name when creating an lv

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 494ece30994c9d80c9f4afc72213515f8ed63698)

ceph-volume tests add verifications for create_vg helper

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 997d922bbd68f59ebbacc3cef2ca3231f44339eb)

ceph-volume tests add verifications for extend_vg helper

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb1e1c61bd5944ad1fd93e86385d26e5b0af38cb)

ceph-volume lvm.batch update module to use Device object on strategy detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ebbb571d2f62dd6c6a38ced51a838ad4a099b817)

ceph-volume lvm.batch.filestore use new Device objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 369bb740a0c52b075a81fb26b368ddf936319e24)

ceph-volume lvm.batch.bluestore use new Device objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c6f4e107a1660dfee194c4344532bd9560ec7b37)

ceph-volume tests.util verify ValidDevice validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 649903bdabe3ea04c499c3dbe6d1e9e7b8f82a23)

ceph-volume util.templates allow more space for long paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 57f76e36abffee0f8f7d81b7d16e72de148a065c)

ceph-volume lvm.strategies create mixed type filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e4902ced1bb26a7f55639c747e6aed7a9fd97b05)

ceph-volume lvm.batch add new strategy validators for LVM devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f67732cd9fb11830cc134deaa3ec7ebc107fa1fe)

ceph-volume util add a new Device class validator for args

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5be2794b7fe42e3a4505ec9f406dd5ae46595bd9)

ceph-volume lvm.batch use new Device class and validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d74ce8e54da4878c01a68adf598685045fafba72)

ceph-volume api.lvm new utility to extend a volume group for one or more devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 21df60294977f4dd2dab218df77a04b326fd8b19)

ceph-volume api.lvm allow uuid suffixes in vg/lv creation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 028316414227103457d96ba3ab88fbebeb27c1dc)

ceph-volume: allow `lvm zap` to zap many devices at once

Supports passing many devices, partitions or lvs to
ceph-volume lvm zap.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit d1b929395b4db3c021d3e6df02e020d72962ab3d)

ceph-volume devices.lvm zap should skip when lv_uuid doesn't exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 447be1c9cc3a8f83687f8c0d2378dfd175e03959)

ceph-volume: device.Device should ignore pvs with no vg_name

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9b4f9471ed258bc5a06ec48e1d2ebd56f02a68cd)

ceph-volume: tests for util.device.Device

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 947266e16ae3a5e9bdaa7fbf326a0e1b96f16e0f)

ceph-volume: adds sys_info constant to hold system information

This is only currently used to hold information about devices
from /sys/block in the util.Device class in the sys_api property.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 035f1a4092bba44cce95cabc1e5a12e3938e1150)

ceph-volume util create a device module for identifying devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f955a7e86404a879ae2bdfb696bdcefa51cbdff)

Merge pull request #23805 from ceph/backport-luminous-23532

luminous: ceph-volume: expand auto engine for single type devices on filestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap loop over all vgs from a pv to destroy them

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9066792764edcab1d0c2b3178d6f8f9784e30f85)

ceph-volume tests.util ensure dmcrypt_close will skip non-existent paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7a5ee22437a05fc809bffde97c8e3401f7ba564d)

ceph-volume util.encryption if crypt path does not exist, skip it

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 635f8e9e2ecdad5c54a599029bdad74e88fad600)

ceph-volume lvm.zap unmount lvs and close dmcrypt on partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 549109c1de2b43f31747828cb5703f9db221dd1a)

ceph-volume tests.functional batch add filestore sceanrios to tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ac00e722551e759fc3c1c157ddc89933fb3ac275)

ceph-volume lvm.zap iterate over multiple lvs on a pv

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 21d4f4afcefb6b2510ae67d1ca5e3bf1d510cdfb)

ceph-volume tests.functional add xenial batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfd1bfc2422318edb5a87739b70f93bfc057dc02)

ceph-volume tests.functional add bluestore,filestore playbooks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 26ca6c129a70cd28aa0ed2dae552fb91a251ad21)

ceph-volume tests.functional add centos7 filestore batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a4ff2bff502f1837f696e5f8ebe88f7586d1a83f)

ceph-volume tests.functional add centos7 bluestore batch tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 544e364e8c3c01633333ddc11c29302965e0616c)

ceph-volume tests.util assert new as_int,as_float helper methods

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d2c181fc70bf20fb17cf41437fb207afc5dd7656)

ceph-volume tests verify new utility for journal sizes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3adf13f868f270293fc307b58786155341470a9d)

ceph-volume util.prepare add a helper to get journal sizes from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4c0b4df1df2db85209c13ff66ae7a67524856c2d)

ceph-volume util.disk add as_int and as_float helpers for Size objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ff7054116c50cb149de207280b0469bf51655728)

ceph-volume lvm.strategies fix a bug where incorrect template was used

The template wouldn't render the total OSDs and the removed section was
repeated

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f473bce7addbd31c7c976d2f5f1aef82696d4da1)

ceph-volume lvm.strategies use get_journal_size helper for filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 01aebfc6a075bc91d0dc345f9cd953e3a4259015)

ceph-volume lvm.strategies update filestore use of Size objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f29ec4c6077e9ab36a9eb5ac030b17cd3a47567c)

ceph-volume lvm.strategies update bluestore use of Size

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 75e9729b76594ce1b6504c10e613a91aea82477d)

ceph-volume lvm.strategies create single type filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit af99401cad2c25b02e6ea0df4554d80afbad2b9b)

ceph-volume tests.util verify that Size object don't change state

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5a54ab17602d159bdd29fe82a25f4f026b0fe6c8)

ceph-volume tests.util update to reflect Size object changes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7816fa9f90985d03c17fda31093252fc66a48146)

ceph-volume util.disk do not modify self instance of Size objects

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 00dba2e83ad80cd41b6b29efa642e2da826c2124)

ceph-volume lvm.strategies use new template location in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8f7bca7ba1f74de125a3789133e06c6cf0c083c1)

ceph-volume util create a templates module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 783e58472c55d378773404050d95470d3bb8bbc2)

doc/rados update journal size defaults to 5GB

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 35a48728ae160a347471cf5b461c552b85323a1b)

Conflicts:
doc/rados/configuration/osd-config-ref.rst

Merge pull request #23783 from ceph/backport-luminous-23489

luminous: ceph-volume tests/functional add mgrs daemons to lvm tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23781 from ceph/backport-luminous-23448

luminous: ceph-volume: `lvm batch` allow extra flags (like dmcrypt) for bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23776 from dillaman/wip-26927-luminous

luminous: rbd: fix issues in IEC unit handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume tests/functional add mgrs daemons to lvm tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2b7a22ef10e38a7bb4d08c9ebe9736e6452b005f)

ceph-volume lvm.batch move import statement to the top

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c460f18d0930e3dbe1cfb97f2d399e4a8a2ceed9)

ceph-volume lvm.batch allow extra flags for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4149d664fa51eb7b2f900370c2bd000b1a5b1edc)

Merge pull request #23778 from ceph/backport-luminous-23075

luminous: ceph-volume batch command

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.api update lvm sizing tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8d4525a59d1d7eb1d571848698c69cace29bfa53)

ceph-volume api.lvm convert and use extents for sizing calculations

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 99e034d7eed9508ee016431c040e3fddca9bca69)