tools/ceph-objectstore-tool: split filestore directories offline to target hash level

Currently ceph-objectstore-tool can only split dirs that already
meet the usual object number criteria. It won't reduce the cost of
online split for those dirs that would meet the criteria online.

This change uses 'arg1' option for apply-layout-settings to specify
the target hash level wants to split to. Then dirs will be split
offline no matters how many objects they have. This also helps
those pools that forgot to do pre-splitting at their creation time.

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit e71659b11a54ea3abdb047d2ec480bd855acee4a)

Merge pull request #23369 from alfredodeza/luminous-cv-auto

luminous: ceph-volume initial take on auto sub-command

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23236 from dzafman/wip-25085-luminous

luminous: Fix  25085 and 24949

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22806 from smithfarm/wip-24697-luminous

luminous: ceph osd safe-to-destroy crashes the mgr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22753 from pdvian/wip-24498-luminous

luminous: librbd: force 'invalid object map' flag on-disk update

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-volume tests verify get_devices captures device metadata

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8deb088f101894763d27a6c31a3e8a2e96fc8320)

ceph-volume tests.conftest allow custom dirs in tmpfile fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9d820e82ac6bc0f46890df1cdaa104e88cab2a7e)

ceph-volume tests ensure human_readable_size works up to terabytes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4ad42f4244062995de003d8141cddcdc1d2b80fc)

ceph-volume tests verify mapper and /dev devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9cfd37344c69ebf1f3d8440f295d79ae05b545e9)

ceph-volume tests add tests for get_block_devs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 672c4c4d1c31b4631b1238ed0efb98fef34f2a3d)

ceph-volume tests verify _map_dev_paths mappings

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2c4d46955e8d7f52b090cd8b5b5561a2dba443e5)

ceph-volume devices.auto initial take on auto sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e3b04c9677c273f91e0a9de35dcd68c578d124db)

ceph-volume util.disk create a device metadata utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c6559538db28d6b8bc8529e6b2a62e8619c69bd6)

ceph-volume util.disk add dev mapping helpers for path resolution

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c06e63fed8a11e80444e8e7ec2e82373d75aa501)

ceph-volume tests verify get_file_contents utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0177f71caba3fd64de285d6808b5f859c55ed2fe)

ceph-volume util.system capture file contents

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 39792573dabac3b85a41760c965b60746ca1f059)

ceph-volume devices.auto initial take on auto sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5649bd724b5d300c7080052433f9b9d1c6774cff)

ceph-volume main include commented out auto subcommand

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ad1b8cae055ee6c822370d91027613680ccd3dd3)

ceph-volume tests ensure is_lv detection on LVs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 47060a1ef4562b29771bdfb5a375a225f18c228c)

ceph-volume api.lvm create helpers to detect an lv from a device path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 68ca7a691e6e333209b67eb5cea8594f37fb93c1)

Merge pull request #22844 from smithfarm/wip-24735-luminous

luminous: order rbdmap.service before remote-fs-pre.target

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22865 from pdvian/wip-24584-luminous

luminous: osdc: Fix the wrong BufferHead offset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22767 from pdvian/wip-24514-luminous

luminous: rgw: fix the bug of radowgw-admin zonegroup set requires realm

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21806 from ivancich/wip-rgw-unordered-luminous

luminous: rgw: ability to list bucket contents in unsorted order for efficiency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22817 from smithfarm/wip-24690-luminous

luminous: rgw-multisite: endless loop in RGWBucketShardIncrementalSyncCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22820 from smithfarm/wip-24692-luminous

luminous: rgw: index complete miss zones_trace set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22822 from smithfarm/wip-24693-luminous

luminous: rgw: meta and data notify thread miss stop cr manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22846 from smithfarm/wip-24737-luminous

luminous: add unit test for cls bi list command

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22942 from pdvian/wip-24782-luminous

luminous: rgw: set cr state if aio_read err return in RGWCloneMetaLogCoroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22985 from joscollin/wip-24844-luminous

luminous: rgw: require --yes-i-really-mean-it to run radosgw-admin orphans find

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23345 into luminous

* refs/pull/23345/head:
Revert "luminous: pybind/mgr/mgr_module: make rados handle available to all modules"

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23137 from smithfarm/wip-24068-luminous

luminous: tests: osd sends op_reply out of order

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23350 from ceph/backport-luminous-23348

luminous: ceph-volume: adds test for `ceph-volume lvm list /dev/sda`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: adds a unit test for `lvm list /dev/sda`

This test is to prove that the issue from
http://tracker.ceph.com/issues/24957 was fixed
by http://tracker.ceph.com/issues/24784

When running lvm list against a raw device it should handle
gracefully the situation where there are multiple PVs with the
name of the given device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit ef10886f1e061d4273d3c0763ab0827e7e58e9d7)

ceph-volume: move pvolumes fixture into conftest.py

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 37ed1be08b226a5b3a7a5bb70c4d5d1fd22db5d1)

Revert "luminous: pybind/mgr/mgr_module: make rados handle available to all modules"

Reverts: https://github.com/ceph/ceph/pull/23235
Reverts: 45c7170eb68571a3d690460f99392c6753994799

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #22883 from smithfarm/wip-24311-luminous

luminous: pjd: cd: too many arguments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22955 from joscollin/wip-24828-luminous

luminous: qa: fix ffsb cd argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22980 from joscollin/wip-24860-luminous

luminous: cephfs-journal-tool: Fix purging when importing an zero-length journal.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23012 from batrick/i23015

luminous: client: fixup parallel calls to ceph_ll_lookup_inode() in NFS FASL

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23107 from pdvian/wip-24932-luminous

luminous: client: add inst to asok status output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23222 from pdvian/wip-25038-luminous

luminous: mds: scrub doesn't always return JSON results

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22779 from batrick/i24538

luminous: common/DecayCounter: set last_decay to current time when decoding decay counter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22780 from batrick/i24295

luminous: repeated eviction of idle client until some IO happens

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22782 from ukernel/luminous-23989

luminous: mds: don't report slow request for blocked filelock request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22783 from ukernel/luminous-24540

luminous: client:  update inode fields according to issued caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22786 from pdvian/wip-24535-luminous

luminous: client: _ll_drop_pins may access invalid iterator

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22811 from smithfarm/wip-24694-luminous

luminous: PurgeQueue sometimes ignores Journaler errors

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23329 from ceph/backport-luminous-23234

luminous: ceph-volume: PVolumes.get() should return one PV when using name or uuid

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22814 from smithfarm/wip-24748-luminous

luminous: change default filestore_merge_threshold to -10

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22839 from vumrao/wip-vumrao-backport-d12ba37e0d0-to-luminous

luminous: osd: change log level when withholding pg creation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #22979 from VictorDenisov/backport_24033

luminous: rados.pyx: make all exceptions accept keyword arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23064 from pdvian/wip-24886-luminous

luminous: os/bluestore: fix races on SharedBlob::coll in ~SharedBlob.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23331 from alfredodeza/wip-rm25170

luminous: ceph-volume add a __release__ string, to help version-conditional calls

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23293 from smithfarm/wip-upgrade-cleanup-luminous

luminous: qa: drop upgrade/luminous-x

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

ceph-volume add a __release__ string, to help version-conditional calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: PVolumes.get() should return one PV when using name or uuid

It is possible to get duplicated pv entires from the 'pvs' lvm command.
If we're using PVolumes.get() with either pv_name or pv_uuid we can
safely return a single PVolume object. However, if we use pv_tags with
PVolumes.get() we must still raise a MultiplePVsError if many pvs
are found that have the tags, because they are not guaranteed to be the
same pv, as would be the case with pv_name or pv_uuid.

Fixes: http://tracker.ceph.com/issues/24784
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit e5e866e8ecaf0522f3ebac83fa6fbc5a666acd58)

Merge pull request #23238 from alfredodeza/luminous-rm24993

luminous ceph-volume process.call with stdin in Python 3 fix

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23296 from b-ranto/wip-volume-selinux-luminous

[luminous] ceph-volume: Restore SELinux context

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: Restore SELinux context

We need to reset the root context of the file system after mounting it.
Otherwise, the SELinux policy rules will not be preserved.

Fixes: https://tracker.ceph.com/issues/24785
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 61dbf4ac2af7d1e9da927bbc4006c9fbccb55478)

ceph-volume tests.util verify as_bytes behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1802c386da3bf1e77f20d8a5456edd810a631c6f)

ceph-volume tests catch utf-8 encoded stdin errors

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 31f565e90bb346c2be982f7daba31431b5368677)

ceph-volume process use the as_bytes util for stdin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6a9368c1db468e05944607f9814f19053de80b6b)

ceph-volume util create a helper for converting to bytes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb518165445479cf08eefc8a2755ad78dcafd07f)

ceph-volume tests validate the new utils for prompts

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2d988337a96fe6cc8fffb37e4641c86cd47e1139)

ceph-volume tests verify str_to_int utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 56b1bf633d3d16cf335043450df95f191e2907a4)

ceph-volume util create prompt helpers

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0283fcc38b13c2d090b1d25be732e6f5f43b834c)

ceph-volume util move the str_to_int utility out of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cb5b30ab43f37adc45fb6114452b37722c65501e)

ceph-volume tests run to log_descriptors, verifying utf-8

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process decode to utf-8 when logging descriptors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.process ensure that stdin works on py3 and py2.7

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process encode stdin with utf-8 to prevent bytes TypeError

Signed-off-by: Alfredo Deza <adeza@redhat.com>

luminous: qa: drop upgrade/luminous-x

This suite would typically be run from the mimic or master branches, not
luminous.

This commit is intentionally not a cherry-pick, since it is relevant to
luminous only.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23274 from yuriw/wip-yuriw-25123-luminous

qa/upgrade/luminous-p2p: cleaned up "#branch: luminous" lines

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23179 from smithfarm/wip-25066-luminous

luminous: build/ops: install-deps.sh fails on newest openSUSE Leap

Reviewed-by: Tim Serong <tserong@suse.com>

qa/tests: cleaned up "#branch: luminous" lines

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23265 from yuriw/wip-yuriw-25123-luminous

qa/tests: added 12.2.7 to the mix

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23150 from alfredodeza/luminous-rm24952

luminous: ceph-volume lvm.listing only include devices if they exist

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa/tests: added 12.2.7 to the mix
also added 'overall HEALTH_' to the whitelist

Fixes: https://tracker.ceph.com/issues/25123
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23177 from smithfarm/wip-25063-luminous

luminous: doc: ceph-bluestore-tool manpage not getting rendered correctly

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

test: Add test for automatic repair during scrub when distrust set

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Even when we are distrust data_digest, go ahead and set it when we can

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix digest if osd_distrust_data_digest set and replicas match each other

Fixes: https://tracker.ceph.com/issues/24949
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: be_select_auth_object() gives chksum objectstore priority if osd_distrust_data_digest true

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add test for repair of bad object info data_digest on all copies

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 67d9e44de6b52673a3e23d7fd7f2d66566f2268a)

osd: Fix pg repair to handle object info data_digest mismatched, but replicas match each other.

Fixes: https://tracker.ceph.com/issues/25085
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 61e995b2c1b866768151a405641463de067c210c)

tools: Add testing feature "corrupt-info" to ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 8e9c08524da0cf465553f99b2e160e59abc8e84b)

Conflicts:
    src/tools/ceph_objectstore_tool.cc (adjust for Luminous transactions)

osd: Improve logic in be_compare_scrubmaps()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 135d5d0209fd6441a470fd2301c6c0e2ec803091)

ceph-volume lvm.listing only include devices if they exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 37565f81634f47c389926d300ed5f92b223bddea)

ceph-volume: do not pin the testinfra version for the simple tests

For testinfra to work with a 2.6.x version of ansible it must use the
latest version.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 70b86012b56fd4b66fc0c7ce88a6cfbe895a0702)

Merge pull request #23264 from ceph/backport-luminous-23263

luminous: ceph-volume: update version of ansible to 2.6.x for simple tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: update version of ansible to 2.6.x for simple tests

ceph-ansible now requires a 2.5.x or 2.6.x version of ansible if you're
using the master branch. This updates our functional tests for the
simple subcommand to use a 2.6.x version of ansible.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 654578928dd840801967c5409b5b19ad6cf90317)

Merge pull request #23245 from alfredodeza/luminous-ansible-26

luminous ceph-volume tests/functional use Ansible 2.6

Reviewed-by: Andrew Schoen <aschoen@redhat.com

ceph-volume tests/functional use testinfra latest for ansible 2.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f48ceab975539c9ce989aade7a1f3ae7d5b51d5)

ceph-volume tests/functional use Ansible 2.6

The ceph-ansible project now refuses to use 2.4 even though it works
with that version

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9570864dd614d2442278e2ba3501788fefd753e3)

Merge pull request #23235 from rhcs-dashboard/wip-25102-luminous

pybind/mgr/mgr_module: make rados handle available to all modules

pybind/mgr/mgr_module: make rados handle available to all modules

Fixes: http://tracker.ceph.com/issues/25102
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 366905fb5779ecb110ec4216afe7cde14fbbb2d8)
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #23227 from liewegas/wip-25057-luminous

luminous: osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

test: validate empty json output during recursive scrub

Signed-off-by: Venky Shankar <vshankar@redhat.com>
Fixes: http://tracker.ceph.com/issues/23958
(cherry picked from commit f253e3c3117aad183b05b863474c6d92fb923ed4)

test: subclass test_scrub.Workload from CephFSTestCase

This allows usage of assert*() calls instead of writing
our own version.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit ba23f5aa707b306f35939325b48de79859c93bcc)