ceph-volume: filter by osd-id or osd-fsid when zapping

2f5c10c12c37e6865ce54bb4940d3779353cba4f introduced a bug:

`ceph-volume lvm zap` command fails under certain conditions.

when passing `--osd-id` or `--osd-fsid` to `ceph-volume lvm zap` command
it tries to zap additionnal devices that have nothing to do with the osd
being zapped.

When calling `api.get_lvs()` in `ensure_associated_lvs()` we have to
pass the osd-id/osd-fsid information so only related devices are
returned by `get_lvs()` method

Closes: https://tracker.ceph.com/issues/46627
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit e94aef6f96189648d0f204d8118501577dde5f52)

ceph-volume: delete devices.lvm.prepare.Prepare.get_lv

Since it calls get_lv_from_argument() internally and use get_first_lv()
instead and update tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit ca1d3e694c6390632c86e6dfcf01a2b3c6a67f0d)

ceph-volume: don't use api.lvm.PVolumes

Update api.lvm.get_first_pv() and api.lvm.get_pvs() instead and update
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit a6e3c69c6cd561ac6680fceb3017b9a2afe3d9de)

ceph-volume: don't use api.lvm.VolumeGroups

Use api.lvm.get_vgs() and api.lvm.get_first_vg() instead and update
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit dfe6b1b90384482e8ec38e3be7024971fb252968)

ceph-volume: don't use api.lvm.get_lv

Use api.lvm.get_lvs() and api.lvm.get_first_lv() instead and update the
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit b2254050c01d872ebc906708a3b4c8d4fa2105d7)

ceph-volume: don't use api.lvm.get_lv_from_argument

Use api.lvm.get_lvs() and api.lvm.get_first_lv() instead and update
tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 5301c4959e5169b10cc92716de6222f7516770e6)

ceph-volume: don't use class api.lvm.Volumes

Replace code using class Volumes by methods get_lvs() and
get_first_lv(). Also, update the tests accordingly.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 2f5c10c12c37e6865ce54bb4940d3779353cba4f)

Merge branch 'wip-46095-octopus' into octopus - from  https://github.com/ceph/ceph/pull/35685

octopus: mon: Warn when too many reads are repaired on an OSD

Reviewed-by: Neha Ojha <nojha@redhat.com>

test: flush_pg_stats() ignore OSDs that don't respond to getting sequence

This eliminates bogus errors in the logs and returned from flush_pg_stats()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 41322eaa62be0de20fdedb44d5817a3f1916ab5e)

mgr: Warn when too many reads are repaired on an OSD

Include test case
Configurable by setting mon_osd_warn_num_repaired (default 10)
Ignore new health warning with random eio injection test

Fixes: https://tracker.ceph.com/issues/41564
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 661996d4342c427209b1eae4b0247f8210a00fc3)

Merge pull request #35715 from dzafman/wip-46115

octopus: tools: Add statfs operation to ceph-objecstore-tool

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35550 from neha-ojha/wip-reduce-noise-octopus

octopus: qa/tasks/ceph_manager.py: dump more useful info before failing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35136 from shyukri/wip-45209-octopus

octopus: monitoring: alert for pool fill up broken

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #35830 from guits/wip-46251-octopus

octopus: ceph-volume: add dmcrypt support in raw mode

Merge pull request #35753 from smithfarm/wip-wipe-prn-octopus

octopus: doc: PendingReleaseNotes: clean slate for 15.2.5

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #35809 into octopus

* refs/pull/35809/head:
qa/tasks/vstart_runner.py: be python3 compatible
doc/dev/developer_guide: use python3 to launch vstart_runner.py
pybind/mgr/dashboard/run-backend-api-tests.sh: use python3 by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

15.2.4

ceph-volume: remove unused function

This function is never called in raw context, let's remove it.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 20e8286f5e1b79678b17659272daa1e28e763618)

ceph-volume: add raw testing coverage

This commit adds testing against `ceph-volume raw` subcommand.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7bb811c3b6ec25ca34734131115cee531439f0ff)

ceph-volume: add dmcrypt support in raw mode

This commit adds the dmcrypt support in `ceph-volume raw` mode.

Note about `ceph-volume raw list` change:
Given `lsblk -J` (json output) option isn't available on all OS, I came up with
adding '--inverse' option to the existing command which allows us to get the
mapper devices list in that command output. Not listing root devices containing
partitions shouldn't have side effect since we are in `ceph-volume raw`
context.

example:
running `lsblk --paths --nodeps --output=NAME --noheadings` doesn't allow to
get the mapper list because the output is like following :

$ lsblk --paths --nodeps --output=NAME --noheadings
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd

the dmcrypt mappers are hidden because of the `--nodeps` given they are
displayed as a dependency.

$ lsblk --paths --output=NAME --noheadings
/dev/sda
|-/dev/mapper/ceph-3b52c90d-6548-407d-bde1-efd31809702f-sda-block-dmcrypt
`-/dev/mapper/ceph-3b52c90d-6548-407d-bde1-efd31809702f-sda-db-dmcrypt
/dev/sdb
/dev/sdc
/dev/sdd

adding `--inverse` is a trick to get around this issue, the counterpart is that
we can't list root devices if they contain at least one partition but this
shouldn't be an issue in `ceph-volume raw` context given we only deal with
raw devices.

Fixes: https://tracker.ceph.com/issues/45959
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 159c30d29f009c07e51a4bd7cd5ecea8cf6c8739)

qa/tasks/vstart_runner.py: be python3 compatible

differentiate `str` and `bytes` instances, and drop python2 support from
vstart_runner.py, as we've moved to python3 already

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit bedea85019076999a205c421bbf593ab8ab0641b)

doc/dev/developer_guide: use python3 to launch vstart_runner.py

as vstart_runner.py is compatible with python3, and so is the tests
under `qa/` directory.

and use `pip` to install teuthology, simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c6e48ef0f024b4a3f4b457367544d773197933d4)

pybind/mgr/dashboard/run-backend-api-tests.sh: use python3 by default

since vstart_runner.py is now py3, let's use python3 by default

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d54f3ea43e05c2823c813e9687194b0b1fe15ffe)

Merge pull request #35807 from cbodley/wip-qa-rgw-ceph-octopus

octopus: qa/rgw: run external tests against ceph-octopus branch

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

octopus: qa/rgw: run external tests against ceph-octopus branch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35801 from yuriw/wip-yuriw-rgw-octopus

qa/test: fixed s3tests, changed to `force-branch: ceph-octopus`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/test: fixed s3tests, changed to `force-branch: ceph-octopus`

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35780 from sebastian-philipp/octopus-qa-cephad-upgrade-no-rgw

octopus: qa/cephadm/upgrade: Exclude RGW for now

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35774 from theanalyst/octopus-rgw-cors-fixes

octopus: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader

Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/cephadm/upgrade: use quay.ceph.io

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit ab1d42c991037741064f35a9e881c7a45ef8b6c4)

qa/cephadm/upgrade: Exclude RGW for now

tasks/cephadm.py gained RGW support very recently and
I'm now facing a dilemman:

* Either we set the upgrade start to 15.2.4 and thus
  no longer upgrade from an old version, or
* Disable RGW upgrade for now.

I think doing both would be optinal, but for now, let's
disable RGW, in order to keep the coverage for everything
else.

Fixes: https://tracker.ceph.com/issues/46157
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 4c5b20f9168d2d7ea40f48702f1bc9fb999bbce3)

rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader

the values in the <ExposeHeader> element are sent back to clients in a
Access-Control-Expose-Headers response header. if the values are allowed
to have newlines in them, they can be used to inject arbitrary response
headers

this issue only affects s3, which gets these values from an xml document

in swift, they're given in the request header
X-Container-Meta-Access-Control-Expose-Headers, so the value itself
cannot contain newlines

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Reported-by: Adam Mohammed <amohammed@linode.com>

doc: PendingReleaseNotes: clean slate for 15.2.5

All of these Pending Release Notes have been included in the official
15.2.4 Release Notes, so keeping them in this file any longer would be
counterproductive.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35625 from kshtsk/wip-46061-octopus

octopus: qa/tasks/cephadm: setup site based container registry

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35517 from votdev/wip-45952-octopus

octopus: mgr/dashboard: Show labels in hosts page

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35498 from ifed01/wip-ifed-hybrid-alloc-octopus

octopus: os/bluestore: introduce hybrid allocator

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35463 from bk201/wip-45836-octopus

octopus: monitoring: fixing some issues in RBD detail dashboard

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35446 from smithfarm/wip-45682-octopus

octopus: os/bluestore: fix large (>2GB) writes when bluefs_buffered_io = true

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35236 from smithfarm/wip-45676-octopus

octopus: qa/workunits: Remove redundant Xenial cmake3 requirements

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34849 from rhcs-dashboard/wip-45346-octopus

octopus: mgr/dashboard: Asynchronous unique username validation for User Component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #35718 from callithea/wip-46136-octopus

octopus: mgr/dashboard: fix autocomplete input backgrounds in chrome and firefox

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ishan Rai <ishanrai05@gmail.com>

Merge pull request #35135 from shyukri/wip-45557-octopus

octopus: mgr/dashboard: Prevent dashboard breakdown on bad pool selection

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35706 from tspmelo/wip-45541-octopus

octopus: mgr/dashboard: Fix random E2E error in mgr-modules

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #35693 from rhcs-dashboard/wip-45727-octopus

octopus: mgr/dashboard: grafana panels for rgw multisite sync performance

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: fix autocomplete input backgrounds in chrome and firefox

Fixes: https://tracker.ceph.com/issues/46109
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>
(cherry picked from commit 01df08a49d7d06c7688cc1bcb5eac1fac402c5da)

tools: Add statfs operation to ceph-objecstore-tool

Fixes: https://tracker.ceph.com/issues/46064
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 19054ceb43bb2f56dadfe5462d546f2ee1de9263)

Merge pull request #35670 from kotreshhr/wip-46001-octopus

octopus: mgr/volumes: Add snapshot info command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35671 from kotreshhr/wip-45849-octopus

octopus: mgr/volumes: Create subvolume with isolated rados namespace

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35600 from smithfarm/wip-46013-octopus

octopus: qa/vstart_runner: update vstart_runner.LocalRemote.sh

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35555 from Vicente-Cheng/wip-45851-octopus

octopus: mds: flag backtrace scrub failures for new files as okay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35554 from Vicente-Cheng/wip-45848-octopus

octopus: qa/tasks/vstart_runner.py: add def mountpoint parameter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #35451 from smithfarm/wip-45846-octopus

octopus: tests: cephfs: run modprobe beforehand so that /sys/fs/fuse/connections is never absen

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>t

Merge pull request #35450 from smithfarm/wip-45845-octopus

octopus: src/client/fuse_ll: compatible with libfuse3.5 or higher

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35449 from smithfarm/wip-45842-octopus

octopus: cephfs: fuse: add the '-d' option back for libfuse

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #34998 from smithfarm/wip-45476-octopus

octopus: cephfs-shell: Change tox testenv name to py3

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35616 from Vicente-Cheng/wip-45888-octopus

octopus: qa/tasks/cephfs/test_scrub.py: use umount_wait to avoid possible ceph-fuse daemon stuck

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35448 from smithfarm/wip-45838-octopus

octopus: mds: cleanup uncommitted fragments before mds goes to active

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Fix random E2E error in mgr-modules

This test failed at random times when it tried to find the new value of pool_ids
in the balancer module.
This happened because the value of pool_ids is automatically reverted by ceph,
so in some situations when we tried to read the new value,
it was already reverted and failed.

Enhanced the tests to be able to use any text input, not only the ones with
empty default values.

Fixes: https://tracker.ceph.com/issues/45445
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 08bb5a851f729b01c7cf69f662588903fd7c14bc)

Merge pull request #35268 from callithea/wip-45710-octopus

octopus: mgr/dashboard: redesign the login screen

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #35269 from callithea/wip-45368-octopus

octopus: mgr/dashboard: displaying Service detail inside table

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: grafana panels for rgw multisite sync performance

* RGW sync perf. counters are now exposed through grafana panels.
* Sync Performance tab is only shown if rgw realm is detected.
* Prometheus module: added metrics suitable for prometheus consumption (from existing ones, not replacing for backward compatibility).

Fixes: https://tracker.ceph.com/issues/45310
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit cf4ff7d2f03bc285a3fae3f27577333f11dab58a)

 Conflicts:
   src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form/rgw-bucket-form.component.spec.ts
   src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-form/rgw-bucket-form.component.ts

   - Backport conflicts resolved in these files.

Merge pull request #35679 from tspmelo/wip-46111-octopus

octopus: mgr/dashboard: Fix e2e chromium binary validation

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix e2e chrome and chromium binaries verification

Fixes: https://tracker.ceph.com/issues/46110
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit e4b8d7eac352eb31fd17e420521aa2c1ce44453f)

Conflicts:
src/pybind/mgr/dashboard/run-frontend-e2e-tests.sh
There was an extra square bracket in octopus that caused the conflict.
This was manually fixed, since the commit that removed it will not be
backported.

Merge pull request #35243 from smithfarm/wip-45707-octopus

octopus: mgr/dashboard: Fix redirect after changing password

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/volumes: Create subvolume with isolated rados namespace

1. Add --namespace-isolated option to 'subvolume create' command
   to create subvolume in a separate RADOS namespace
2. Add "pool_namespace" field to 'subvolume info' command
   which displays the rados namespace if set else empty string

Fixes: https://tracker.ceph.com/issues/45289
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 7396880d4096b3a2a813b8ebcdcdf3082ee98b23)

mgr/volumes: Add snapshot info command

The following command is added

"ceph fs subvolume snapshot info <vol_name> <sub_name> <snap_name> [<group_name>]"

The output is in json format with following fields

    created_at: time of creation of snapshot in the format "YYYY-MM-DD HH:MM:SS:ffffff"
    data_pool: data pool the snapshot belongs to
    has_pending_clones: "yes" if snapshot clone is in progress otherwise "no"
    protected: "yes" if snapshot is protected otherwise "no"
    size: snapshot size in bytes

Fixes: https://tracker.ceph.com/issues/45237
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 171930ef773ee6630c76eb9581c725594fb4e884)

Merge pull request #35270 from callithea/wip-45232-octopus

octopus: mgr/dashboard: Show table details inside the datatable

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35452 from shyukri/wip-45783-octopus

octopus: ceph-volume: add and delete lvm tags in a single lvchange call.

mgr/dashboard: Asynchronous unique username validation for User Component

Implements an asynchronous validation for the username field in the Create User form which immediately display an error message if the username already exists.

Fixes: https://tracker.ceph.com/issues/36375
Signed-off-by: Nizamudeen <nia@redhat.com>
(cherry picked from commit c4dfe20c08ecb9e50b7cb7bef6eb603e04d56387)

mgr/dashboard: update background image in about modal box

Signed-off-by: Ishan Rai <ishanrai05@gmail.com>
(cherry picked from commit 80d5ad81fa740bd3b9f597632f546949700dc0ec)

mgr/dashboard: redesign the login screen

fixes: https://tracker.ceph.com/issues/35689
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>
(cherry picked from commit 6b35508ef87cf5c94f6290e5752e36af3acab9bf)

Merge pull request #35444 from smithfarm/wip-45882-octopus

octopus: Objecter: don't attempt to read from non-primary on EC pools

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #35442 from smithfarm/wip-45775-octopus

octopus: osd: make "missing incremental map" a debug log message

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35382 from jschmid1/wip-45880-octopus

octopus: osd: add --osdspec-affinity flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35291 from callithea/wip-45738-octopus

octopus: mgr/dashboard: Proposed About Modal box

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35152 from callithea/wip-45370-octopus

octopus: mgr/dashboard: ECP modal enhancement

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35151 from callithea/wip-45468-octopus

octopus: monitoring: Fix "10% OSDs down" alert description

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #35447 from smithfarm/wip-45773-octopus

octopus: cephfs: vstart_runner: set mounted to True at the end of mount()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35256 from Vicente-Cheng/wip-45680-octopus

octopus: mgr/volumes: Fix subvolume create idempotency

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

fuse: update to newer FUSE_USE_VERSION

The build was failing for me against fuse-devel v3.9.1. The prototype
for fuse_ll_ioctl was wrong, as it was expecting the old-style one with
signed int args.

In newer libfuse versions, the prototype varies based on
FUSE_USE_VERSION. Update to a newer FUSE_USE_VERSION value to ensure
that we use the newer ioctl prototype. This also means that we need to
handle a new prototype for fuse_session_loop_mt as well.

While we're in here, move the definition of FUSE_USE_VERSION to
ceph_fuse.h so we have the definition in one place. This does mean we
need to reorganize the includes in a few places.

Fixes: https://tracker.ceph.com/issues/45866
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit cfbc0fb48a9880dabe8f332e9bf67d81867ae198)

Merge pull request #35531 from bk201/wip-45921-octopus

octopus: mgr/dashboard: remove space after service name in the Hosts List table

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa/tasks/cephadm: setup site based container registry

Add containers registry mirror for 'docker.io' to the config
file /etc/containers/registries.conf

Since we need site based config, so each teuthology instance
could have own containers' mirror, store corresponding data
in /etc/teuthology.yaml under 'overrides/cephadm' task specific
section.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit d8edde0c53b80d01c3797c521c5adfc2ac0a99bb)

qa/tasks/vstart_runner: make remote.sh return str as default

To address the issue in https://github.com/ceph/teuthology/pull/1459.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 624944503e7c6aa246331f4d9f291cf714296380)

qa/tasks/cephfs/test_scrub.py: use umount_wait to avoid ceph-fuse stuck

If the ceph-fuse client need to flush the caps and does sync wait,
the umount() will just return successfully, then the netns container
will be destroyed and the network will not be reachable, but the
ceph-fuse daemon is still stucked and waiting for the flush caps ack.

This will cause the ceph-fuse daemon get stuck forever and if the
mds daemons get restarted, it will try to reconnect the clients,
but the stucked ceph-fuse daemnon won't reply to it, because it is
not reachable any more.

Fixes: https://tracker.ceph.com/issues/45665
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit a7423b2286154a4403e586b5148984649f988a03)

qa/cephfs: add FUSE module before running mount -t fusectl

Also, change timeout from 15 minutes to 30 seconds for the command
"mount -t fusectl xxx xxx" since 15 minutes is too much as per Zheng.

Fixes: https://tracker.ceph.com/issues/45304
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 2d7c32997b9e5db5a13f5ed6e117ac9f71dac370)

qa/tasks/cephfs/fuse_mount.py: by python3 compatible

pass `StringIO()` to capture stdout whose value will be interpreted as a
string later on

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c5bd318de65866b4992314c2fb0e82090b4d2979)

qa/vstart_runner: update vstart_runner.LocalRemote.sh

Commit 9f6c764f10f replaces remote.run calls by remote.sh without
updating the definition of vstart_runner.LocalRemote.sh which breaks the
cephfs tests when executed locally.

Fixes: https://tracker.ceph.com/issues/44579
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 2cc0ee709c36eabe03311a00b72295da468bccf4)

Conflicts:
qa/tasks/vstart_runner.py
- trivial

Merge pull request #35364 from tchaikov/octopus-py3

octopus: migrate qa/ to Python3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa: add .teuthology_branch with `master` contents

This is follow up change for:

    teuthology-suite: automate -t argument default value
    https://github.com/ceph/teuthology/pull/1490

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 07cc36dcd264b66a7f3a19687b336737a772c2f5)

qa/tasks/rbd_fio: unbreak after the conversion from StringIO

Fix a bad typo in commit db7ae8eff60a ("qa/tasks/rbd_fio: get rid of
StringIO for py3").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit c3f4f1d6604749a51c552b868ef373f321139005)

qa/tasks/cephfs: add mount_wait() support to simplify the code

Mostly we should wait the mountpoint to get ready, especially for
the fuse mountpoint, sometimes it may take a few seconds to get
ready.

Fixes: https://tracker.ceph.com/issues/44044
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 5c24d91327f598ea9fee59ea87ddc6582d48bd96)

qa/tasks/cephfs: use StringIO for capturing str

if we don't expect non-utf8 in stdout, use StringIO instead of BytesIO.

see also d8d44ed1566b19eec055e07da2a0fed88fed4152

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f10f86579684a0d1b8d99787b8e19709a64df7d6)

qa/rgw: partial revert of 'import with full path'

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit aa60edcec89b1538070b316c496e55bd0e15f966)

qa/cephfs: use StringIO instead of BytesIO

Fixes: https://tracker.ceph.com/issues/45425
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 71b823daee05dd96cb3cced8ff34943d7afc4210)

Conflicts:
qa/tasks/cephfs/mount.py: trivial resolution

test/rgw: should import parse not urlparse

urlparse is a function while parse is a submodule where parse_qsl() is
provided

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0db88f775b204b44d5c7b3c50e5ad3fffad029f1)

qa/tasks/barbican.py: convert to str before json.loads()

in Python3, json.loads() expects a string, while
HTTPConnection.getresponse() returns a byte-like object, so we need to
coerce it to str first.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c79e1e1a525e08d55fd60c2333e8f5d4df603350)

qa/tasks/tempest: use --black-regex for blacklisting tests

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b51139fce72e5ed12eb2696c416b68af6daac8cd)

qa/keystone: unpin python-openstackclient

this reverts 2dac5c5012cffb5f41d7e7457569c6bfdc53ec26,
as we don't use `--os-url` anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 60269764b432d16421b5431afbd0d05de4ded991)

qa/tasks/tempest: use user/pass to authenticate

instead of using admin token use "admin" user to authenticate,
as admin token is not suggested anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8f5c832915cd5f7e7e9c4f2a0517edc9573fa349)

qa/suites/rgw/tempest: update unsupported tests of tempest

after rerunning tempest with lastest radosgw, remove the supported
tests from the blacklist, and add the ones which are not supported
yet. now we can pass 123 tests in total.

also enable discoverity for better testing coverage, since it's
supported now.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 739751ef7a23b6e6a85cee522d255936b0912780)