osd: do_sparse_read(): On Checksum mismatch try to repair

Fixes: http://tracker.ceph.com/issues/24875
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 929f78ab895e63667c36e57cc44d14a1402f4f9d)

osd: do_read(): Use actual bytes read to determine if CRC can be checked

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 48d7448d72834810cf13eca1425e0cf0e8570c7a)

Conflicts:
src/osd/PrimaryLogPG.cc (trivial)

osd/PrimaryLogPG: trigger auto-repair on full-object-size CRC error

There are still FileStore users and the full-object CRC might be able
to catch silently returned data errors. Since we support auto-repair now,
let's trigger an auto-repair process instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4667280f8afe6cd68dfffea61d7530581f3dd0eb)

Conflicts:
src/osd/PrimaryLogPG.cc (trivial)

Merge pull request #23236 from dzafman/wip-25085-luminous

luminous: Fix  25085 and 24949

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22806 from smithfarm/wip-24697-luminous

luminous: ceph osd safe-to-destroy crashes the mgr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22753 from pdvian/wip-24498-luminous

luminous: librbd: force 'invalid object map' flag on-disk update

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #22844 from smithfarm/wip-24735-luminous

luminous: order rbdmap.service before remote-fs-pre.target

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #22865 from pdvian/wip-24584-luminous

luminous: osdc: Fix the wrong BufferHead offset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22767 from pdvian/wip-24514-luminous

luminous: rgw: fix the bug of radowgw-admin zonegroup set requires realm

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21806 from ivancich/wip-rgw-unordered-luminous

luminous: rgw: ability to list bucket contents in unsorted order for efficiency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22817 from smithfarm/wip-24690-luminous

luminous: rgw-multisite: endless loop in RGWBucketShardIncrementalSyncCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22820 from smithfarm/wip-24692-luminous

luminous: rgw: index complete miss zones_trace set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22822 from smithfarm/wip-24693-luminous

luminous: rgw: meta and data notify thread miss stop cr manager

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22846 from smithfarm/wip-24737-luminous

luminous: add unit test for cls bi list command

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22942 from pdvian/wip-24782-luminous

luminous: rgw: set cr state if aio_read err return in RGWCloneMetaLogCoroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22985 from joscollin/wip-24844-luminous

luminous: rgw: require --yes-i-really-mean-it to run radosgw-admin orphans find

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #23345 into luminous

* refs/pull/23345/head:
Revert "luminous: pybind/mgr/mgr_module: make rados handle available to all modules"

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23137 from smithfarm/wip-24068-luminous

luminous: tests: osd sends op_reply out of order

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23350 from ceph/backport-luminous-23348

luminous: ceph-volume: adds test for `ceph-volume lvm list /dev/sda`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: adds a unit test for `lvm list /dev/sda`

This test is to prove that the issue from
http://tracker.ceph.com/issues/24957 was fixed
by http://tracker.ceph.com/issues/24784

When running lvm list against a raw device it should handle
gracefully the situation where there are multiple PVs with the
name of the given device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit ef10886f1e061d4273d3c0763ab0827e7e58e9d7)

ceph-volume: move pvolumes fixture into conftest.py

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 37ed1be08b226a5b3a7a5bb70c4d5d1fd22db5d1)

Revert "luminous: pybind/mgr/mgr_module: make rados handle available to all modules"

Reverts: https://github.com/ceph/ceph/pull/23235
Reverts: 45c7170eb68571a3d690460f99392c6753994799

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #22883 from smithfarm/wip-24311-luminous

luminous: pjd: cd: too many arguments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22955 from joscollin/wip-24828-luminous

luminous: qa: fix ffsb cd argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22980 from joscollin/wip-24860-luminous

luminous: cephfs-journal-tool: Fix purging when importing an zero-length journal.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23012 from batrick/i23015

luminous: client: fixup parallel calls to ceph_ll_lookup_inode() in NFS FASL

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23107 from pdvian/wip-24932-luminous

luminous: client: add inst to asok status output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23222 from pdvian/wip-25038-luminous

luminous: mds: scrub doesn't always return JSON results

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22779 from batrick/i24538

luminous: common/DecayCounter: set last_decay to current time when decoding decay counter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22780 from batrick/i24295

luminous: repeated eviction of idle client until some IO happens

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22782 from ukernel/luminous-23989

luminous: mds: don't report slow request for blocked filelock request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22783 from ukernel/luminous-24540

luminous: client:  update inode fields according to issued caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22786 from pdvian/wip-24535-luminous

luminous: client: _ll_drop_pins may access invalid iterator

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22811 from smithfarm/wip-24694-luminous

luminous: PurgeQueue sometimes ignores Journaler errors

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23329 from ceph/backport-luminous-23234

luminous: ceph-volume: PVolumes.get() should return one PV when using name or uuid

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22814 from smithfarm/wip-24748-luminous

luminous: change default filestore_merge_threshold to -10

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22839 from vumrao/wip-vumrao-backport-d12ba37e0d0-to-luminous

luminous: osd: change log level when withholding pg creation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #22979 from VictorDenisov/backport_24033

luminous: rados.pyx: make all exceptions accept keyword arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23064 from pdvian/wip-24886-luminous

luminous: os/bluestore: fix races on SharedBlob::coll in ~SharedBlob.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23331 from alfredodeza/wip-rm25170

luminous: ceph-volume add a __release__ string, to help version-conditional calls

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23293 from smithfarm/wip-upgrade-cleanup-luminous

luminous: qa: drop upgrade/luminous-x

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

ceph-volume add a __release__ string, to help version-conditional calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: PVolumes.get() should return one PV when using name or uuid

It is possible to get duplicated pv entires from the 'pvs' lvm command.
If we're using PVolumes.get() with either pv_name or pv_uuid we can
safely return a single PVolume object. However, if we use pv_tags with
PVolumes.get() we must still raise a MultiplePVsError if many pvs
are found that have the tags, because they are not guaranteed to be the
same pv, as would be the case with pv_name or pv_uuid.

Fixes: http://tracker.ceph.com/issues/24784
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit e5e866e8ecaf0522f3ebac83fa6fbc5a666acd58)

Merge pull request #23238 from alfredodeza/luminous-rm24993

luminous ceph-volume process.call with stdin in Python 3 fix

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23296 from b-ranto/wip-volume-selinux-luminous

[luminous] ceph-volume: Restore SELinux context

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: Restore SELinux context

We need to reset the root context of the file system after mounting it.
Otherwise, the SELinux policy rules will not be preserved.

Fixes: https://tracker.ceph.com/issues/24785
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 61dbf4ac2af7d1e9da927bbc4006c9fbccb55478)

ceph-volume tests.util verify as_bytes behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1802c386da3bf1e77f20d8a5456edd810a631c6f)

ceph-volume tests catch utf-8 encoded stdin errors

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 31f565e90bb346c2be982f7daba31431b5368677)

ceph-volume process use the as_bytes util for stdin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6a9368c1db468e05944607f9814f19053de80b6b)

ceph-volume util create a helper for converting to bytes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb518165445479cf08eefc8a2755ad78dcafd07f)

ceph-volume tests validate the new utils for prompts

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2d988337a96fe6cc8fffb37e4641c86cd47e1139)

ceph-volume tests verify str_to_int utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 56b1bf633d3d16cf335043450df95f191e2907a4)

ceph-volume util create prompt helpers

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0283fcc38b13c2d090b1d25be732e6f5f43b834c)

ceph-volume util move the str_to_int utility out of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cb5b30ab43f37adc45fb6114452b37722c65501e)

ceph-volume tests run to log_descriptors, verifying utf-8

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process decode to utf-8 when logging descriptors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.process ensure that stdin works on py3 and py2.7

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process encode stdin with utf-8 to prevent bytes TypeError

Signed-off-by: Alfredo Deza <adeza@redhat.com>

luminous: qa: drop upgrade/luminous-x

This suite would typically be run from the mimic or master branches, not
luminous.

This commit is intentionally not a cherry-pick, since it is relevant to
luminous only.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23274 from yuriw/wip-yuriw-25123-luminous

qa/upgrade/luminous-p2p: cleaned up "#branch: luminous" lines

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23179 from smithfarm/wip-25066-luminous

luminous: build/ops: install-deps.sh fails on newest openSUSE Leap

Reviewed-by: Tim Serong <tserong@suse.com>

qa/tests: cleaned up "#branch: luminous" lines

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23265 from yuriw/wip-yuriw-25123-luminous

qa/tests: added 12.2.7 to the mix

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23150 from alfredodeza/luminous-rm24952

luminous: ceph-volume lvm.listing only include devices if they exist

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa/tests: added 12.2.7 to the mix
also added 'overall HEALTH_' to the whitelist

Fixes: https://tracker.ceph.com/issues/25123
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23177 from smithfarm/wip-25063-luminous

luminous: doc: ceph-bluestore-tool manpage not getting rendered correctly

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

test: Add test for automatic repair during scrub when distrust set

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Even when we are distrust data_digest, go ahead and set it when we can

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix digest if osd_distrust_data_digest set and replicas match each other

Fixes: https://tracker.ceph.com/issues/24949
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: be_select_auth_object() gives chksum objectstore priority if osd_distrust_data_digest true

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add test for repair of bad object info data_digest on all copies

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 67d9e44de6b52673a3e23d7fd7f2d66566f2268a)

osd: Fix pg repair to handle object info data_digest mismatched, but replicas match each other.

Fixes: https://tracker.ceph.com/issues/25085
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 61e995b2c1b866768151a405641463de067c210c)

tools: Add testing feature "corrupt-info" to ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 8e9c08524da0cf465553f99b2e160e59abc8e84b)

Conflicts:
    src/tools/ceph_objectstore_tool.cc (adjust for Luminous transactions)

osd: Improve logic in be_compare_scrubmaps()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 135d5d0209fd6441a470fd2301c6c0e2ec803091)

ceph-volume lvm.listing only include devices if they exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 37565f81634f47c389926d300ed5f92b223bddea)

ceph-volume: do not pin the testinfra version for the simple tests

For testinfra to work with a 2.6.x version of ansible it must use the
latest version.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 70b86012b56fd4b66fc0c7ce88a6cfbe895a0702)

Merge pull request #23264 from ceph/backport-luminous-23263

luminous: ceph-volume: update version of ansible to 2.6.x for simple tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: update version of ansible to 2.6.x for simple tests

ceph-ansible now requires a 2.5.x or 2.6.x version of ansible if you're
using the master branch. This updates our functional tests for the
simple subcommand to use a 2.6.x version of ansible.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 654578928dd840801967c5409b5b19ad6cf90317)

Merge pull request #23245 from alfredodeza/luminous-ansible-26

luminous ceph-volume tests/functional use Ansible 2.6

Reviewed-by: Andrew Schoen <aschoen@redhat.com

ceph-volume tests/functional use testinfra latest for ansible 2.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f48ceab975539c9ce989aade7a1f3ae7d5b51d5)

ceph-volume tests/functional use Ansible 2.6

The ceph-ansible project now refuses to use 2.4 even though it works
with that version

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9570864dd614d2442278e2ba3501788fefd753e3)

Merge pull request #23235 from rhcs-dashboard/wip-25102-luminous

pybind/mgr/mgr_module: make rados handle available to all modules

pybind/mgr/mgr_module: make rados handle available to all modules

Fixes: http://tracker.ceph.com/issues/25102
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 366905fb5779ecb110ec4216afe7cde14fbbb2d8)
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #23227 from liewegas/wip-25057-luminous

luminous: osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

test: validate empty json output during recursive scrub

Signed-off-by: Venky Shankar <vshankar@redhat.com>
Fixes: http://tracker.ceph.com/issues/23958
(cherry picked from commit f253e3c3117aad183b05b863474c6d92fb923ed4)

test: subclass test_scrub.Workload from CephFSTestCase

This allows usage of assert*() calls instead of writing
our own version.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit ba23f5aa707b306f35939325b48de79859c93bcc)

mds: dump minimal cli output during recursive scrub trigger

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit a70d21b162e00e49813f25149fa93442f5434e0b)

osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

In 0ceb5c0f3694725c7043ea99b077816da2ca44a7 I screwed up the version that
CRUSH_TUANBLES5 (chooseleaf_stable) was added: it comes from
043a7378ec4e828f2568ea3906d9076848161ccd, v10.0.3, which leads up to
jewel.

This, in turn, means that when jewel tunables are set we do not actually
encode the jewel tunables.  At least, luminous doesn't; jewel uses a
different code path to decode how to encode the OSDMap.

This, in turn, leads to crc errors:

- luminous mons running
- require_osd_release is jewel
- jewel tunables are set.  mon in-memory crushmap gets chooseleaf_stable=1
- osds are luminous, but don't see the new field in the crushmap
- upgrade finishes, and require_osd_Release becomes luminous
- the full osdmap encodes the chooseleaf_stable field, finally, and it
  is 1
- the osds try to encode the same thing, but their crushmap has 0.
-> crc mismatch

This is a regression from 0ceb5c0f3694725c7043ea99b077816da2ca44a7,
backported to luminous in 686b054f48c50859bd5e8bad93007bf9d7d7899c in
12.2.6.

Fixes: 686b054f48c50859bd5e8bad93007bf9d7d7899c
Fixes: http://tracker.ceph.com/issues/25057
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7e9d24d0602abefa4cca245f5f86dcd4b412b97a)

Merge pull request #23219 from liewegas/wip-slow-requests-upgrade-luminous

luminous: qa/suites/upgrade/jewel-x: whitelist 'slow requests'

qa/suites/upgrade/jewel-x: whitelist 'slow requests'

This message is generated by jewel but not luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

Add new OpenSUSE Leap id for install-deps.sh

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
(cherry picked from commit bd0448c0b5a09a9314fddf1757502529801366e3)

Conflicts:
install-deps.sh (trivial resolution)

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Older versions of Sphinx, such as the one in CentOS 7, do not render "..
option::" lines correctly if the option contains a hyphen but does not start
with a hyphen. And ceph-bluestore-tool appears to be the only Ceph manpage
affected by this bug.

Fixes: http://tracker.ceph.com/issues/24800
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 7c9229e7876dfc1bf6e415ed7cce924c3b50925d)

Merge pull request #23147 from alfredodeza/luminous-rm24961

ceph-volume tests/functional run lvm list after OSD provisioning

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests/functional run lvm list after OSD provisioning

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e01fb206d67f26724f3b03e4b22f5ba6eea22207)

Merge pull request #23102 from ceph/backport-luminous-23093

luminous: ceph-volume: an OSD ID must exist and be destroyed before reuse

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23128 from ceph/backport-luminous-23117

luminous: ceph-volume: unmount lvs correctly before zapping

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/suites: set osd_pg_log_dups_tracked in cfuse_workunit_suites_fsync.yaml

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 94a2d810826d710c3c71403d6a78249cc66a39e7)

ceph-volume: unmount lvs correctly before zapping

If the lv found does not include the meatadata to construct an osd path,
then we should try to unmount it's path instead.

Fixes: http://tracker.ceph.com/issues/24796
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5b175a5bb68264b80c00cb4f04026a4d19ca20ad)

client: add inst to asok status output

Makes it easier for checking if a client becomes blacklisted.

Fixes: http://tracker.ceph.com/issues/24724
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 47c67d584e204d56c2a5f9a414082bb523098015)

ceph-volume: an osd ID must exist and be destroyed before reuse

Fixes: https://tracker.ceph.com/issues/24044
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 02a13f2de0783ff3c680a6042388a4692250edfb)