osd: reset complete_to when trimming the log past it

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 38170cdb1b8c3ea7e8b411fabe1fe99abd06cf52)

osd: allow trim() to proceed when there are missing items

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit de42fee0dff299f4d0377961d05e02fd8f49f21b)

osd: handle trim() during backfill and async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit e538c31f0f3133f811a8e478fcb25b575cad66bf)

osd: add flag to remember async recovery

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 22d17fb5aad6ab9d7525d9492c0e96a36d02879e)

osd: print pg log length and trim_to

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit f48584a5b468949c31bffba1b507fb13a8756284)

osd: make calc_trim_to() independent of min_last_complete_ondisk

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 1ae5fd32c92ea2f025344c663535d00f71f2cdda)

Merge pull request #23349 from ceph/backport-mimic-23348

mimic: ceph-volume: adds test for `ceph-volume lvm list /dev/sda`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: adds a unit test for `lvm list /dev/sda`

This test is to prove that the issue from
http://tracker.ceph.com/issues/24957 was fixed
by http://tracker.ceph.com/issues/24784

When running lvm list against a raw device it should handle
gracefully the situation where there are multiple PVs with the
name of the given device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit ef10886f1e061d4273d3c0763ab0827e7e58e9d7)

ceph-volume: move pvolumes fixture into conftest.py

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 37ed1be08b226a5b3a7a5bb70c4d5d1fd22db5d1)

Merge pull request #23303 from smithfarm/wip-qa-ceph-detect-init

mimic: tools/ceph-detect-init: support RHEL as a platform

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/ceph-detect-init: support RHEL as a platform

12d94cae2a06fd2ae39379e233588be99bff91de introduced a regression in that it
started returning '' as the codename for most supported distros, and also
(apparently) changed the distro string returned on Red Hat Enterprise Linux (to
"rhel"). As a result, ceph-detect-init began throwing an UnsupportedPlatform
exception on RHEL.

Can not be cherry-picked because ceph-disk and ceph-detect-init are no longer
present in master.

Fixes: 12d94cae2a06fd2ae39379e233588be99bff91de
Fixes: http://tracker.ceph.com/issues/18163
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23327 from ceph/backport-mimic-23234

mimic: ceph-volume: PVolumes.get() should return one PV when using name or uuid

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23333 from alfredodeza/wip-rm25169

mimic: ceph-volume add a __release__ string, to help version-conditional calls

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume add a __release__ string, to help version-conditional calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: PVolumes.get() should return one PV when using name or uuid

It is possible to get duplicated pv entires from the 'pvs' lvm command.
If we're using PVolumes.get() with either pv_name or pv_uuid we can
safely return a single PVolume object. However, if we use pv_tags with
PVolumes.get() we must still raise a MultiplePVsError if many pvs
are found that have the tags, because they are not guaranteed to be the
same pv, as would be the case with pv_name or pv_uuid.

Fixes: http://tracker.ceph.com/issues/24784
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit e5e866e8ecaf0522f3ebac83fa6fbc5a666acd58)

Merge pull request #23239 from alfredodeza/mimic-rm24993

mimic ceph-volume process.call with stdin in Python 3 fix

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23295 from b-ranto/wip-volume-selinux-mimic

[mimic] ceph-volume: Restore SELinux context

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23301 from yuriw/wip-yuriw-p2p-mimic

qa/tests:  added OBJECT_MISPLACED to the whitelist

qa/tests:  added OBJECT_MISPLACED to the whitelist

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

ceph-volume: Restore SELinux context

We need to reset the root context of the file system after mounting it.
Otherwise, the SELinux policy rules will not be preserved.

Fixes: https://tracker.ceph.com/issues/24785
Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 61dbf4ac2af7d1e9da927bbc4006c9fbccb55478)

ceph-volume tests.util verify as_bytes behavior

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1802c386da3bf1e77f20d8a5456edd810a631c6f)

ceph-volume tests catch utf-8 encoded stdin errors

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 31f565e90bb346c2be982f7daba31431b5368677)

ceph-volume process use the as_bytes util for stdin

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6a9368c1db468e05944607f9814f19053de80b6b)

ceph-volume util create a helper for converting to bytes

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb518165445479cf08eefc8a2755ad78dcafd07f)

ceph-volume tests validate the new utils for prompts

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2d988337a96fe6cc8fffb37e4641c86cd47e1139)

ceph-volume tests verify str_to_int utility

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 56b1bf633d3d16cf335043450df95f191e2907a4)

ceph-volume util create prompt helpers

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0283fcc38b13c2d090b1d25be732e6f5f43b834c)

ceph-volume util move the str_to_int utility out of lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cb5b30ab43f37adc45fb6114452b37722c65501e)

ceph-volume tests run to log_descriptors, verifying utf-8

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process decode to utf-8 when logging descriptors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.process ensure that stdin works on py3 and py2.7

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process encode stdin with utf-8 to prevent bytes TypeError

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23218 from yuriw/wip-yuriw-25086-mimic

qa/tests: added v13.2.1 to the mix

Merge branch 'mimic' of github.com:ceph/ceph into mimic

Merge pull request #22965 from ceph/backport-mimic-22764

mimic: ceph-volume: refuse to zap mapper devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

lvm zap: refuse to zap mapper devices

Fixes: http://tracker.ceph.com/issues/24504
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 8e602d52657d50888a83963148334deb1f8d4e1c)

adds the util.disk.is_mapper_device method

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 852a466fdfa29c91f266ec122dcd5b01b8d1fca5)

Merge pull request #23149 from alfredodeza/mimic-rm24952

mimic: ceph-volume lvm.listing only include devices if they exist

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23101 from ceph/backport-mimic-23093

mimic: ceph-volume: an OSD ID must exist and be destroyed before reuse

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.listing only include devices if they exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 37565f81634f47c389926d300ed5f92b223bddea)

ceph-volume: do not pin the testinfra version for the simple tests

For testinfra to work with a 2.6.x version of ansible it must use the
latest version.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 70b86012b56fd4b66fc0c7ce88a6cfbe895a0702)

Merge pull request #23269 from ceph/backport-mimic-23263

mimic: ceph-volume: update version of ansible to 2.6.x for simple tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: update version of ansible to 2.6.x for simple tests

ceph-ansible now requires a 2.5.x or 2.6.x version of ansible if you're
using the master branch. This updates our functional tests for the
simple subcommand to use a 2.6.x version of ansible.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 654578928dd840801967c5409b5b19ad6cf90317)

13.2.1

Merge pull request #23262 from liewegas/wip-mimic-p2p

mimic: qa/suites/upgrade/mimic-p2p: allow target version to apply

qa/suites/upgrade/mimic-p2p: allow target version to apply

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23246 into mimic

* refs/pull/23246/head:
rpm,deb: fix chown to use -h
rpm,deb: chown bluestore block* links to ceph.ceph on install/upgrade

Reviewed-by: Alfredo Deza <adeza@redhat.com>

githubmap: update contributors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #23250 into mimic

* refs/pull/23250/head:
osd/OSDMap: fix CEPHX_V2 osd requirement to nautilus, not mimic

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/OSDMap: fix CEPHX_V2 osd requirement to nautilus, not mimic

This was delayed and did not appear in 13.2.0.

Fixes: 1fdc85f224d52655d74eff9a29b6029b974661b3
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 37781061022a92e88dff91ff18fd0fc14b5436de)

rpm,deb: fix chown to use -h

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 92207c76b91c26c0558688a358ade7c0d94d1b59)

rpm,deb: chown bluestore block* links to ceph.ceph on install/upgrade

This works around https://tracker.ceph.com/issues/24903

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 58cde3fd793c037beca6645cc30b2ae1c30e2af1)

Merge pull request #23244 from alfredodeza/mimic-ansible-26

mimic ceph-volume tests/functional use Ansible 2.6

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests/functional use testinfra latest for ansible 2.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4f48ceab975539c9ce989aade7a1f3ae7d5b51d5)

ceph-volume tests/functional use Ansible 2.6

The ceph-ansible project now refuses to use 2.4 even though it works
with that version

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9570864dd614d2442278e2ba3501788fefd753e3)

Merge PR #23233 into mimic

* refs/pull/23233/head:
mon: don't require CEPHX_V2 from mons until nautilus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon: don't require CEPHX_V2 from mons until nautilus

The mimic 13.2.0 didn't have it.

Fixes: 3dc80e5f9b6ebf1bc1cecbd95b288005216bdbec
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7ba0c14ce75f6936b59a3c61ea34a6f0fb909280)

# Conflicts:
# src/mon/Monitor.cc

- dropped the nautilus cases in the original commit

Merge pull request #23209 from smithfarm/wip-25081-mimic

mimic: qa: krbd_exclusive_option.sh: bump lock_timeout to 60 seconds

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: added v13.2.1 to the mix

Fixes: https://tracker.ceph.com/issues/25086
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23136 from smithfarm/wip-25010-mimic

mimic: tests: osd sends op_reply out of order

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa: krbd_exclusive_option.sh: bump lock_timeout to 60 seconds

Avoid sporadic failures in combination with msgr-failures/many.yaml,
where assert_locked() might take over 10 seconds.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 69619b69663e026aa4e9c7c21c1fc879782c26e9)

Merge pull request #23178 from smithfarm/wip-25065-mimic

mimic: build/ops: install-deps.sh fails on newest openSUSE Leap

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23176 from smithfarm/wip-25062-mimic

mimic: doc: ceph-bluestore-tool manpage not getting rendered correctly

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #23154 from smithfarm/wip-25034-mimic

mimic: tests: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) in powercycle

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23156 from yuriw/wip-yuriw-25005-mimic

mimic: qa/tests -  added overrides stanza to allow runs on ovh on rhel OS

Merge PR #23184 into mimic

* refs/pull/23184/head:
common: add osd_distrust_data_digest option
osd: apply osd_skip_data_digest to read operations

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23164 from smithfarm/wip-25056-mimic

mimic: tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

common: add osd_distrust_data_digest option

If we have reason to distrust the stored full-object oi data digests,
this option will ignore them (and opportunistically clear them).  It
basically affects the same behaviors that osd_skip_data_digest except
the previous option only takes effect if bluestore is in use, while this
option is unconditional.

This serves a workaround for clusters that suffered the 12.2.6 bug that
produced bad full-object digests and *also* had PGs' primaries move from
bluestore to filestore nodes.  In those situations, this option can be
set to true.

After all OSDs are upgrade and all PGs come back with a clean deep
scrub this option can be disabled again.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit b8476ebc5b3c5b5d801bf875bbd6dce5b113325c)

# Conflicts:
# src/osd/PrimaryLogPG.cc

- osd->osd_skip_data_digest, not g_conf->osd_skip_data_digest

osd: apply osd_skip_data_digest to read operations

If we would skip updating data digests, apply that same decision to read
operations as well.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 49207a6cb388446361aafacfaa863d66dd452ded)

# Conflicts:
# src/osd/PrimaryLogPG.cc

- one of the do_read() cases moved
- osd->osd_skip_data_digest (observed), not g_conf->...

Merge pull request #23122 from yuriw/wip-yuriw-24926-mimic

mimic - qa/tests: Set ansible-version: 2.4

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Add new OpenSUSE Leap id for install-deps.sh

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
(cherry picked from commit bd0448c0b5a09a9314fddf1757502529801366e3)

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Older versions of Sphinx, such as the one in CentOS 7, do not render "..
option::" lines correctly if the option contains a hyphen but does not start
with a hyphen. And ceph-bluestore-tool appears to be the only Ceph manpage
affected by this bug.

Fixes: http://tracker.ceph.com/issues/24800
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 7c9229e7876dfc1bf6e415ed7cce924c3b50925d)

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Fixes: http://tracker.ceph.com/issues/25051
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 2555524f1fcf230f5468e1a5fd20f9fb741e03aa)

qa/tests -  added overrides stanza to allow runs on ovh on rhel OS

Fixes: https://tracker.ceph.com/issues/25005
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit ac87fd946c8806f6f84d6c6ea124b4731b7b8294)

qa/tests: Set ansible-version: 2.4

Fixes http://tracker.ceph.com/issues/24926
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #23148 from alfredodeza/mimic-rm24961

ceph-volume tests/functional run lvm list after OSD provisioning

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests/functional run lvm list after OSD provisioning

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e01fb206d67f26724f3b03e4b22f5ba6eea22207)

Merge pull request #23127 from ceph/backport-mimic-23117

mimic: ceph-volume: unmount lvs correctly before zapping

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/suites: set osd_pg_log_dups_tracked in cfuse_workunit_suites_fsync.yaml

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 94a2d810826d710c3c71403d6a78249cc66a39e7)

ceph-volume: unmount lvs correctly before zapping

If the lv found does not include the meatadata to construct an osd path,
then we should try to unmount it's path instead.

Fixes: http://tracker.ceph.com/issues/24796
Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5b175a5bb68264b80c00cb4f04026a4d19ca20ad)

Merge pull request #22845 from smithfarm/wip-24736-mimic

mimic: add unit test for cls bi list command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21523 from smithfarm/wip-18163

mimic: ceph-detect-init: stop using platform.linux_distribution

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22684 from pdvian/wip-24472-mimic

mimic: spdk: fix ceph-osd crash when activate SPDK

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22948 from tchaikov/mimic-pr-22871

mimic: deb/rpm: add python-six as build-time and run-time dependency

Reviewed-by:  Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22988 from pdvian/wip-24825-mimic

mimic: tests: make test_ceph_argparse.py pass on py3-only systems

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23017 from ceph/wip-cd-fix-supp-distro

mimic: qa/tests: fix supported distro lists for ceph-deploy

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23030 from smithfarm/wip-24904-mimic

mimic: qa: client socket inaccessible without sudo

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23049 from zeichenanonym/mimic

mimic: include: fix build with older clang (OSX target)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #23084 from dzafman/wip-disable-skip-dd-mimic

mimic: osd: Change osd_skip_data_digest default to false and make it LEVEL_DEV

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23094 from theanalyst/wip-mimic-24958

mimic: osd: fix deep scrub with osd_skip_data_digest=true (default) and blue…

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23096 from cbodley/wip-mimic-qa-rgw-no-ec-cache

mimic: qa/rgw: disable testing on ec-cache pools

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23024 from smithfarm/wip-24865-mimic

mimic: common: Abort in OSDMap::decode() during qa/standalone/erasure-code/test-erasure-eio.sh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #23065 from pdvian/wip-24887-mimic

mimic: os/bluestore: fix races on SharedBlob::coll in ~SharedBlob.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #23070 from smithfarm/wip-24942-mimic

mimic: build/ops: ceph.git has two different versions of dpdk in the source tree

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22812 from smithfarm/wip-24705-mimic

mimic: cephfs: allow prohibiting user snapshots in CephFS

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22835 from batrick/i24758

mimic: test gets ENOSPC from bluestore block device

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #22869 from pdvian/wip-24583-mimic

mimic: osdc: Fix the wrong BufferHead offset

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22956 from joscollin/wip-24829-mimic

mimic: qa: fix ffsb cd argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22981 from joscollin/wip-24861-mimic

mimic: cephfs-journal-tool: Fix purging when importing an zero-length journal.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>