qa/tasks: use spaces intead of tabs for indent

to address the errors of
```
TabError: inconsistent use of tabs and spaces in indentation
```

somehow in master, we are using spaces instead of tabs, so this change
is not cherry-picked from master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: use python3 compatible print

Fixes: https://tracker.ceph.com/issues/42210
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 7e87f80a8754c93c10ff937844ad456d4a236f57)

Conflicts:
qa/tasks/cephfs/mount.py
qa/tasks/cephfs/test_journal_migration.py
qa/workunits/mon/caps.py
src/test/rgw/rgw_multi/tests_ps.py
src/test/rgw/rgw_multi/zone_ps.py
src/test/rgw/test_multi.py: trivial resolutions

qa/tasks/cephfs: get rid of StringIO for py3

Use io.BytesIO and six.ensure_str for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 9f6c764f10f99cbf5ca164c2681b001d29505d9d)

Conflicts:
qa/tasks/cephfs/fuse_mount.py
qa/tasks/cephfs/mount.py
qa/tasks/cephfs/test_acls.py
qa/tasks/cephfs/test_cephfs_shell.py
qa/tasks/cephfs/test_journal_migration.py
qa/tasks/cephfs/xfstests_dev.py: trivial resolution

qa/tasks/mgr: partial revert of 'import with full path'

this change partially reverts #34139

using relative import helps with readability and ease the pain to write
down the full parent module name

in #34139, all relative imports were replaced with full path, because we
were using following code to verify if the code is python3 compatible:

```
  mod_spec = importlib.util.spec_from_file_location(mod_name, path)
  mod = importlib.util.module_from_spec(mod_spec)
  mod_spec.loader.exec_module(mod)
```

but this does not work with submodule which can import using relative
import without specifying the name of the package and its parent module.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b0bdbc34147c7e795e1a01cce1197b990c48effb)

Conflicts:
qa/tasks/mgr/dashboard/test_auth.py
qa/tasks/mgr/dashboard/test_cephfs.py
qa/tasks/mgr/dashboard/test_crush_rule.py
qa/tasks/mgr/dashboard/test_host.py
qa/tasks/mgr/dashboard/test_orchestrator.py
qa/tasks/mgr/dashboard/test_user.py: trivial resolutions

qa/rgw: partial revert of 'import with full path'

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit aa60edcec89b1538070b316c496e55bd0e15f966)

Conflicts:
src/test/rgw/rgw_multi/tests_az.py
src/test/rgw/rgw_multi/tests_ps.py
src/test/rgw/rgw_multi/zone_az.py: trivial resolution

test/rgw: should import parse not urlparse

urlparse is a function while parse is a submodule where parse_qsl() is
provided

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0db88f775b204b44d5c7b3c50e5ad3fffad029f1)

qa: import py3 compatible modules using six

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 80b71ef461043bfa274b2c73fb0697a0e75759cd)

Conflicts:
qa/tasks/barbican.py
qa/tasks/vault.py: trivial resolution

qa: import with full path

to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 947a74349d2ce1266cb0f3c6a7030605999e21b3)

Conflicts:
qa/tasks/cephadm.py
qa/tasks/cephfs/test_acls.py
qa/tasks/cephfs/test_openfiletable.py
qa/tasks/cram.py
qa/tasks/mgr/dashboard/test_auth.py
qa/tasks/mgr/dashboard/test_cephfs.py
qa/tasks/mgr/dashboard/test_crush_rule.py
qa/tasks/mgr/dashboard/test_host.py
qa/tasks/mgr/dashboard/test_orchestrator.py
qa/tasks/mgr/dashboard/test_rbd_mirroring.py
qa/tasks/mgr/dashboard/test_settings.py
qa/tasks/mgr/dashboard/test_user.py
src/test/rgw/rgw_multi/tests_az.py
src/test/rgw/rgw_multi/tests_ps.py
src/test/rgw/rgw_multi/zone_az.py: trivial resolution

qa/tasks/workunit: py3 compat

Use six.ensure_str for `get_file()` output in order to
make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit ae6befb3be46d84da67092aff8bff95d9f646aea)

qa/tasks: fix import module path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 1c81d3873d904e42f7a534db0e419c23e5220aed)

Conflicts:
qa/tasks/mon_thrash.py
qa/tasks/workunit.py: trivial resolutions

qa/tasks/thrashosds: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit c26ea687d48b7742eb527a77385321b0138bd634)

qa/tasks/omapbench: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 1140b5f027be472ff7f1c949305d5d9400cba191)

qa/tasks/mgr: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit eb10276c1c062a196bd0391691f6e8ef5378f5b3)

qa/tasks/watch_notify_stress: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 6e328edcbcab28b6d6a3abbb46b48a6a18e660c2)

qa/tasks/osd_max_pg_per_osd: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit b2d52220e3b7e31b16bb6442848c5ce2f9d4fc76)

qa/tasks/radosbench.py: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 3b21067bbbf5e375700554e8b985bc4762d03a46)

qa/tasks/ceph: get rid of cStringIO for py3 compat

Use io.BytesIO instead of cStringIO.StringIO
Use six.ensure_str whenever it needs to convert binary to str.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 35cf5131e7152ce20d916aa99c124751d6a97f5c)

qa/tasks/admin_socket: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 3eb341db27290d01417d7809b198bd17a7f5fa3a)

qa/tasks/systemd: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit a28d3473055ca1440949f3fb99161246f8bb71b1)

qa/tasks/scrub_test: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 1d6c4228cd83ad930119f77a22feae2d3df57589)

qa/tasks/netem: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit fa3db2529fb1b7f9f5145d3d8752d38b439a32f8)

qa: Enable flake8 tox and fix failures

There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
(cherry picked from commit 0127cd1e8817b05b1c3150540b021f9a24b47089)

Conflicts:
qa/tasks/barbican.py
qa/tasks/cephadm.py
qa/tasks/cephfs/test_cephfs_shell.py
qa/tasks/cephfs/xfstests_dev.py
qa/tasks/daemonwatchdog.py
qa/tasks/mgr/dashboard/test_cephfs.py
qa/tasks/mgr/dashboard/test_orchestrator.py
qa/tasks/mgr/dashboard/test_rbd.py
qa/tasks/mgr/mgr_test_case.py
qa/tasks/mgr/test_orchestrator_cli.py
qa/tasks/s3tests.py
qa/tasks/s3tests_java.py
qa/tasks/vstart_runner.py
qa/workunits/mon/caps.py: trivial resolutions,
          and drop the the change to qa/CMakeLists.txt, as we don't have
          add_tox_test() back in nautilus

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 99e9e82d646a9db1773ca97da7460dbfb5a0b415)

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 71ada20a0e362db3cb5ce2a4afa753082f62f9cd)

qa/tasks/reg11184: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit fa16954df94a763d2604f58c119b8ee943fa2cfe)

qa/tasks/radosbenchsweep: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 7ffcd00b652b955f52df7a27ce2130cbc40ebd72)

qa/tasks/divergent_priors2: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit b6eefbb02896043333f264023274e83ec36b8362)

qa/tasks/ceph_objectstore_tool: get rid of cStringIO for py3

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit b502bc17d17c784b83f652b79f04eccfd1048805)

qa/tasks/ceph_manager: ensure str for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 4c992baf251d272800e76bb40584f169163f6c1b)

qa/tasks/vstart_runner: use io.BytesIO for py3 compat

Use io.BytesIO instead of StringIO for py3 compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 49592af4005808db4ed5301e9072525bd709f44a)

Conflicts:
qa/tasks/vstart_runner.py: trivial resolution

vstart_runnner: add sh method to LocalRemote

The method is a wrapper of teuthology.misc.sh

Fixes: https://tracker.ceph.com/issues/43496
Signed-off-by: Rishabh Dave <ridave@gmail.com>
(cherry picked from commit b7d740ede7ff5839fdce32a5bfc78ca935990208)

qa/tasks/manypools: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 5f876961ef584ab07c8f9ef50f10f010a5e7510f)

Conflicts:
qa/tasks/manypools.py: trivial resolution

qa/tasks/ceph_deploy: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 982ba399b34765ca888dafbbf4221bab53e7964a)

tasks/ceph: get rid of iteritems for python3

For python3 compatibility use items() instead of iteritems().

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 9b5bd6c7155579d3b903ef0eb0ac8eddfe8f29e1)

Merge pull request #35305 from tchaikov/nautilus-qa-rgw-s3a-hadoop

nautilus: qa/rgw: s3a-hadoop task defaults to maven-version 3.6.3

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35181 from smithfarm/wip-45644-nautilus

nautilus: rgw/notifications: version id was not sent in versioned buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35162 from tchaikov/nautilus-qa-keystone

nautilus: qa/keystone: use bindep to discover keystone's dependencies

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35319 from neha-ojha/wip-45772-nautilus

nautilus: qa/suites/upgrade/nautilus-p2: disable more min pg per osd warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35248 from rhcs-dashboard/wip-45582-nautilus

nautilus: mgr/dashboard/grafana: Add rbd-image details dashboard

Merge pull request #35199 from smithfarm/wip-45600-nautilus

nautilus: mds: don't shallow copy when decoding xattr map

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35185 from smithfarm/wip-45497-nautilus

nautilus: cephfs/tools: add accounted_rstat/rstat when building file dentry

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #35163 from tchaikov/nautilus-upgrade-from-luminous-fs-disable-too-few-pgs

nautilus: qa/suites/fs/upgrade: disable mon_pg_warn_min_per_osd

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35149 from Vicente-Cheng/wip-45602-nautilus

nautilus: mds: Handle blacklisted error in purge queue

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35118 from Vicente-Cheng/wip-45478-nautilus

nautilus: client: only set MClientCaps::FLAG_SYNC when flushing dirty auth caps

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35117 from Vicente-Cheng/wip-45474-nautilus

nautilus: bash_completion: Do not auto complete obsolete and hidden cmds

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/suites/upgrade/nautilus-p2p: disable more min pg per osd warnings

This follows 58eb3edc8478c993c5446475df58d659d3f6d356 and
b75907e7575dbdc5888c198f354cb5f71bf21ab3.

This change is not cherry-picked from master since it already has
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4.

Fixes: https://tracker.ceph.com/issues/45772
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/rgw: s3a-hadoop task defaults to maven-version 3.6.3

binaries for v3.6.2 are no longer available under
http://www-us.apache.org/dist/maven/maven-3/

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 82351b9874a9bb5e2e07436303be34e9f5db0ae5)

Conflicts:
qa/tasks/s3a_hadoop.py: trivial resolution

Merge pull request #35214 from shyukri/wip-43990-nautilus

nautilus: mgr/insights: fix prune-health-history

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35213 from shyukri/wip-43464-nautilus

nautilus: mgr: close restful socket after exec

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35212 from shyukri/wip-43098-nautilus

nautilus: mgr/dashboard: Display the aggregated number of request

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35211 from shyukri/wip-45469-nautilus

nautilus: monitoring: Fix "10% OSDs down" alert description

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #35174 from smithfarm/wip-45637-nautilus

nautilus: mgr/dashboard: install teuthology using pip

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Remove additional unneeded steps for the metrics calculation

Signed-off-by: Enno Gotthold <egotthold@suse.de>
(cherry picked from commit dfb1e0020ed8130a2276be1ca94d8e7d7f931273)

doc/mgr/prometheus: Change images to rbd-enabled pools

Signed-off-by: Enno Gotthold <egotthold@suse.de>
(cherry picked from commit db76547c8d829e529dcebccde100fca438d64446)

mgr/prometheus: Add config value to honor ceph config in prometheus

Signed-off-by: Enno Gotthold <egotthold@suse.de>
Author: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 6d5f88450e61122016fcf7b6cf9431dc67128d3d)

Merge pull request #35183 from smithfarm/wip-45579-nautilus

nautilus: pybind/rbd: RBD.create() method's 'old_format' parameter now defaults to False

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #35182 from smithfarm/wip-45577-nautilus

nautilus: librbd: copy API should not inherit v1 image format by default

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35086 from tspmelo/wip-45540-nautilus

nautilus: mgr/dashboard: Fix HomeTest setup

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35137 from shyukri/wip-45208-nautilus

nautilus: monitoring: alert for pool fill up broken

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc: Add information on how to enable the rbd-images dashboard

Fixes: https://tracker.ceph.com/issues/44623
Signed-off-by: Enno Gotthold <egotthold@suse.de>
In the previous commit (1c52ee2b) the added dashboard will stay empty
until it is manually enabled to display information.

(cherry picked from commit 6386095e51a1d58fa0093d37b26df25c524b72bc)

Conflicts:
doc/cephadm/monitoring.rst
        - Delete (file doesn't exist in nautilus)

mgr/dashboard: Add grafana chart for rbd image details

Fixes: https://tracker.ceph.com/issues/44623
Signed-off-by: Enno Gotthold <egotthold@suse.de>
This dashboard will per default be empty as the already existing
dashboard with the summary for all rbd images.

(cherry picked from commit 9707cb30cbbfc0f7c669e19fb9d2482ea422993c)

mgr/insights: make 'insights prune-health-history 0' zap current state

The self._health_slot value needs to be updated too if the current key
is removed.

Fixes: https://tracker.ceph.com/issues/43886
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 04f25b26c5aea5b4cf5cb94e939d6fba74228b27)

mgr: close restful socket after exec

Signed-off-by: liushi <liu.shi@navercorp.com>
(cherry picked from commit ec3a696a5d328936b611b2a19fc9ab12ae183b48)

mgr/dashboard: Display the aggregated number of request

convertTimeSeries will now calculate the aggregated total number of client
request made in the last seconds, instead of the number of request per second.

Fixes: https://tracker.ceph.com/issues/42622
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 1dc483e8d9699a57ef61ea77d4a54ce93b341b17)

monitoring: Fix "10% OSDs down" alert description

The alert was triggered when less than 90% of OSDs were _up_, but then the
description took that value and described it as the percentage of OSDs being
_down_. So with 12% of OSDs down, the alert description would read:

```
88% or 88 of 100 OSDs are down (>=10%).
```

which can be panic-inducing.

This commit changes the alert expression to actually compute the ratio of OSDs
being down, which makes the correct value appear in the description.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
(cherry picked from commit 653c3f66823179fc5b9cbb74ff932d61a6c4178c)

Merge pull request #35180 from smithfarm/wip-45642-nautilus

nautilus: tests: src/test/compressor: Add missing gtest

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #35047 from smithfarm/wip-44686-nautilus

nautilus: qa/standalone/osd/osd-backfill-stats.sh: get_latest_osdmap to propagate map change

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #35049 from smithfarm/wip-45224-nautilus

nautilus: ceph_test_rados_api_{watch_notify,misc}: tolerate some timeouts

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #35048 from smithfarm/wip-44689-nautilus

nautilus: test: Add flush_pg_stats to avoid race with getting num_shards_repaired

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #35028 from shyukri/wip-44163-nautilus

nautilus:test/fio: bump to fio 3.15

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35024 from smithfarm/wip-43919-nautilus

nautilus: osd: dispatch_context and queue split finish on early bail-out

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34957 from smithfarm/wip-44841-nautilus

nautilus: osd/PeeringState: do not trim pg log past last_update_ondisk

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34823 from votdev/wip-45323-nautilus

nautilus: mgr/dashboard: monitoring menu entry should indicate firing alerts

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35050 from smithfarm/wip-43773-nautilus

nautilus: qa: log warning on scrub error

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35000 from smithfarm/wip-45496-nautilus

nautilus: cephfs: Client: fix Finisher assert failure

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34784 from Vicente-Cheng/wip-45221-nautilus

nautilus: cephfs-journal-tool: correctly parse --dry_run argument

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34783 from Vicente-Cheng/wip-45217-nautilus

nautilus: qa/cephfs: update TestClusterFull:test_barrier

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34435 from Vicente-Cheng/wip-44483-nautilus

nautilus: mds: handle ceph_assert on blacklisting

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34338 from Vicente-Cheng/wip-44478-nautilus

nautilus: mds: Using begin() and empty() to iterate the xlist

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

mds: don't shallow copy when decoding xattr map

Otherwise inodes' xattr maps may reference a large shared buffer
(from omap fetch or journal read). If mds trims and loads inode
repeatly, each inode can reference different large buffer in the
worst case.

Fixes: https://tracker.ceph.com/issues/45090
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit deff94c8f43bb3734b688ccb828d942b8f150638)

Conflicts:
src/mds/CInode.cc
- CInode::decode_lock_ixattr is missing in nautilus
src/mds/mdstypes.h
- nautilus has a code comment that is not in master

qa/cephfs: scrub the subdir/ after data scan

This will make sure the rstat between subdir's inode and dirfrag
are matched.

Fixes: https://tracker.ceph.com/issues/44380
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 62e9820c3eb7311b886bed1a8c06da59bd7f301a)

qa/cephfs: remove the whole subdir/ after test done

Fixes: https://tracker.ceph.com/issues/44380
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit b24edff02e6e668046b0cab6709d0ccfaf5edbc5)

tools/cephfs: Disable dirfrag auto merging and splitting

Make sure that the dirfrag auto merging and splitting are disabled
explicitly.

Fixes: https://tracker.ceph.com/issues/44380
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 39df09e52b84b688f99f4e18190683c957d40783)

tools/cephfs: scan the links to set the proper dnfirst

The ceph code will set the dnfirst=2 as default instead of CEPH_NOSNAP.
And this will avoid the MDS daemon assert crash:

ceph_assert(in->first <= straydn->first)

Fixes: https://tracker.ceph.com/issues/44380
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit b8d34cdbdc02488c9c62193a507882ec6991362a)

mds: fix debug log typo

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 6969d41e3d4cb2b460508cee044161de2a196ebe)

pybind/rbd: RBD.create() method's 'old_format' parameter now defaults to False

The RBD v1 format has been deprecated for numerous releases and creation of
v1 format images has been disabled since the Mimic release. This fixes
the Python API's image create() method to ensure v2 images are created by
default (and no longer throw an exception that creation of v1 images are
disabled).

Fixes: https://tracker.ceph.com/issues/45504
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 45349355f4b41c6f9de594ef34a8657230113e6b)

librbd: copy API should not inherit v1 image format by default

When copying from a v1 image, by default the new destination image
would be created using the v1 format. Since the creation of v1 images
is disallowed, this has been updated to default to using the v2
image format.

Fixes: https://tracker.ceph.com/issues/45518
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 69b6d5997e8c6a11739e4d5a32564e61eb2f470f)

Conflicts:
src/librbd/internal.cc
- in nautilus, the call immediately preceding the change is
  "src->snap_lock.put_read()" whereas in master it is
  "src->image_lock.unlock_shared()"

rgw/notifications: version id was not sent in versioned buckets

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
Fixes: https://tracker.ceph.com/issues/45440
(cherry picked from commit 973b0b1ed8767767f0c4446d97248c1b8002f1d3)

src/test/compressor: Add missing gtest

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit a11c72d6d6c72b6801cf58c202ea92f77af14ebf)

mgr/dashboard: require enum34 only for python_version<'3.4'

This is a partial backport of 1e07237d92da3bb5eba82b49f6276d17696e6985

Backporting the entire thing would be too intrusive.

Fixes: https://tracker.ceph.com/issues/45636
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35093 from neha-ojha/wip-too-few-pgs-nautilus

nautilus: qa/suites/upgrade: disable more min pg per osd warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

vstart_runner: be python3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a9906b8cd21ef2ffea7ba9298951542843433287)

mgr/dashboard: install teuthology using pip

* install teuthology using pip
* do not install the dependencies manually, it's error-prone
  to maintain two copies of requirements.txt

Fixes: https://tracker.ceph.com/issues/42969
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 68527ab83a77d20b779759581ce0b8844991231b)

Conflicts:
src/pybind/mgr/dashboard/run-backend-api-tests.sh
- 7567948ba05a78a5a61010994e43e8a78641970c is not being backported to
  nautilus

qa/suites/fs/upgrade: disable mon_pg_warn_min_per_osd

if we upgrade from luminous, then a luminous monitor won't have the
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 or its backport, but after
upgrading to nautilus, mgr detects this issue and translates it to
a warning like

2020-05-21T05:33:26.934 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN
too few PGs per OSD (12 < min 30)

if the number of pg per osd is too low. so we need to disable this
warning as we did in later releases and in master.

this change is not cherry-picked from master, as we don't do
upgrade test from luminous to master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/keystone: pip install bindep

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2144fd8cc7f156264aff958c54c5e35d0e65757c)

qa/keystone: use bindep to discover keystone's dependencies

downloads the keystone repository first, because we need
keystone/bindep.txt to discover the binary dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 86930f3e0f7aa7cd8b847bf01a156437e64be0b4)

Conflicts:
qa/tasks/keystone.py: trivial resolution

qa/suites/upgrade: disable more min pg per osd warnings

This follows 58eb3edc8478c993c5446475df58d659d3f6d356.

This change is not cherry-picked from master since it already has
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35004 from smithfarm/wip-44490-nautilus

nautilus: compressor/lz4: work around bug in liblz4 versions <1.8.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34983 from jecluis/wip-45402-nautilus

nautilus: mon/OSDMonitor: allow trimming maps even if osds are down

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34916 from sseshasa/wip-45391-nautilus

nautilus: mon/OSDMonitor: Always tune priority cache manager memory on all mons

Reviewed-by: Neha Ojha <nojha@redhat.com>