qa/tasks/radosgw_admin.py: do not use dict.has_keys()

in python3, `dict.has_key()` was removed. let's use __contains__
instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 590e9b35aa86ab3e70561a8ecde21da2729dafb3)

qa/tasks/tempest.py: always write str is value of options

in Python2, ConfigParser is almost the same as RawConfigParser, which
allows set non-string values, but in Python3, ConfigParser.set() only
accepts strings as value of option.

since we do not use "cpar" as an internal storage for options, it does
not matter what type of options we set using ConfigParser as long as it
can be consumed by tempest. boolean settings are translated to "true" or
"false". see also
https://docs.openstack.org/tempest/latest/sampleconf.html

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5ef3a5fe3282146b56bd2a589fef17c28d57e8e2)

qa/tasks/tempest.py: use configparser from six.moves

as ConfigParser is offerd by `configparser` module in Python3, so use
six.moves before the migration.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b1726bfb08f7a408b37f6c6d1d7dc80eae0d6423)

qa/tasks/ragweed.py: use str methods of helper from string module

in Python3, some methods offered by `string` module are now member
method of `str` class, and `string.uppercase` is renamed to
`string.ascii_uppercase` in Python2 and Python3. so let's update
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6ad2ca9767fde8d6c8ce68035f34f93a4594fd4a)

test/rgw/rgw_multi: do not decode a str

`s` comes from `rgwadmin()`, which passes `StringIO` as stdout, so the
the output should an instance of `str` in both Python2 and Python3.
hence there is no need to decode it using UTF-8 codecs again.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7202d8a72c749a152a648dcbbcd86387dc988672)

Conflicts:
src/test/rgw/rgw_multi/tests.py: trivial resolution

qa/tasks/radosgw_admin: use HTTPMessage as a dict in PY3

in Python3, HTTPMessage is a dict-like class by itself, and it does not
offer `dict` attribute anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4a9c43313acbd26d33364a43b2835e561ceb259b)

qa/tasks/keystone.py: use list(a_dict) for concatenating lists

`dict.items()` does not return a list in python3, so we need to convert
it to a list first.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 35a3d5f7f8a53e667d3196aa72be5b495b70e91f)

Conflicts:
qa/tasks/keystone.py: trivial resolution

qa/tasks/util/rgw: use StringIO for capturing strings

this change should address the failure of
```
2020-04-06T03:07:59.152 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/teuthology/contextutil.py", line 32, in nested
    vars.append(enter())
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw.py", line 266, in configure_compression
    rgwadmin(ctx, client, cmd=['user', 'list'], check_status=True)
  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/util/rgw.py", line 43, in rgwadmin
    j = json.loads(out)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 5d0dc8346c7c025a868fd32b112cdb32fa5dc9ea)

qa/tasks/cephfs: return ascii decoded xattr

as callers of `TestForwardScrub._read_str_xattr()` expects str.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7f48f61bdc8f88e1025796aec5158bbd69b1eb65)

qa/tasks/cephfs: cast mds_recall_warning_decay_rate to float

this change should address the failure of
```
2020-04-05T15:14:23.088 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T15:14:23.088 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_client_limits.py", line 110, in test_client_pin_mincaps
2020-04-05T15:14:23.089 INFO:tasks.cephfs_test_runner:    self._test_client_pin(True, 200)
2020-04-05T15:14:23.089 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_client_limits.py", line 71, in _test_client_pin
2020-04-05T15:14:23.090 INFO:tasks.cephfs_test_runner:    self.wait_for_health("MDS_CLIENT_RECALL", mds_recall_warning_decay_rate*2)
2020-04-05T15:14:23.091 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/ceph_test_case.py", line 152, in wait_for_health
2020-04-05T15:14:23.091 INFO:tasks.cephfs_test_runner:    self.wait_until_true(seen_health_warning, timeout)
2020-04-05T15:14:23.092 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/ceph_test_case.py", line 193, in wait_until_true
2020-04-05T15:14:23.093 INFO:tasks.cephfs_test_runner:    if elapsed >= timeout:
2020-04-05T15:14:23.093 INFO:tasks.cephfs_test_runner:TypeError: unorderable types: int() >= str()
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 162be9210613a4b4a921fa0ff285c869548d7d49)

Conflicts:
qa/tasks/cephfs/test_client_limits.py: trivial resolution

qa/tasks/cephfs: allow caller to use BytesIO when calling rados()

when the caller expects binary data, it should pass BytesIO as stdout.

this change shold address the failure of
```
2020-04-05T12:47:25.335 INFO:tasks.cephfs_test_runner:======================================================================
2020-04-05T12:47:25.336 INFO:tasks.cephfs_test_runner:ERROR: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2020-04-05T12:47:25.336 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-04-05T12:47:25.336 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:47:25.337 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_forward_scrub.py", line 75, in test_apply_tag
2020-04-05T12:47:25.337 INFO:tasks.cephfs_test_runner:    self.assertTagged(inos[dirpath], tag, self.fs.get_metadata_pool_name())
2020-04-05T12:47:25.337 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_forward_scrub.py", line 98, in assertTagged
2020-04-05T12:47:25.338 INFO:tasks.cephfs_test_runner:    "scrub_tag"
2020-04-05T12:47:25.338 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_forward_scrub.py", line 35, in _read_str_xattr
2020-04-05T12:47:25.339 INFO:tasks.cephfs_test_runner:    strlen = struct.unpack('i', output[0:4])[0]
2020-04-05T12:47:25.339 INFO:tasks.cephfs_test_runner:TypeError: a bytes-like object is required, not 'str'
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 20dafc6d52acc4af235dc061fa67ecd617a90159)

qa/tasks/cephfs: use BytesIO for capturing binary

this change partially reverts e46eb8348e0639ea162b7e224bca40e0257ad6ef.

xattrs could contain non-utf8 encoded data, and should be captured using
BytesIO. moreover, it will be fed to `ceph-dencoder`, which expects
binary when performing "import".

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit eb2c6b3740cee7c8e28c163f73525cbabf2e7faf)

qa/tasks/fs.py: use six.viewitems()

in python3, dict.viewitems() is replaced with dict.items().

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 51fc7a992204fb14aba2eb29d96c1785a79ca27f)

qa/tasks/cephfs: use assertCountEqual if PY3

assertItemsEqual is renamed to assertCountEqual in Python3.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 21d6c5a95b4a6873bc8f35a74e3f48436bd96264)

qa/tasks/cephfs: compare number with number

to address following failure:
```
2020-04-05T12:25:30.997 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:25:30.997 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_misc.py", line 50, in test_root_rctime
2020-04-05T12:25:30.998 INFO:tasks.cephfs_test_runner:    self.assertGreaterEqual(rctime, t-10)
2020-04-05T12:25:30.998 INFO:tasks.cephfs_test_runner:  File "/usr/lib/python3.5/unittest/case.py", line 1215, in assertGreaterEqual
2020-04-05T12:25:30.998 INFO:tasks.cephfs_test_runner:    if not a >= b:
2020-04-05T12:25:30.999 INFO:tasks.cephfs_test_runner:TypeError: unorderable types: str() >= float()
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 83e41daec4d7353361d3897770ad31f79bc80681)

qa/tasks/cephfs: do no radom.sample(a_dict)

collect the keys instead of filtering a dict,
to address following failure:
```
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 29, in test_files_throttle
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:    self._test_throttling(self.FILES_THROTTLE)
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 96, in _test_throttling
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner:    return self._do_test_throttling(throttle_type)
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 176, in _do_test_throttling
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner:    mds_max_purge_ops = int(self.fs.get_config("mds_max_purge_ops", 'mds'))
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/filesystem.py", line 285, in get_config
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner:    service_id = random.sample(filter(lambda i: self.mds_daemons[i].running(), self.mds_daemons), 1)[0]
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/virtualenv/lib/python3.5/random.py", line 311, in sample
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:    raise TypeError("Population must be a sequence or set.  For dicts, use list(d).")
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:TypeError: Population must be a sequence or set.  For dicts, use list(d).
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 40d0356249193bf4aaa2d38923eef19caa63eb59)

qa/tasks: use StringIO for capturing string output

see d8d44ed1566b19eec055e07da2a0fed88fed4152

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8bfe977854686149c74de3ebaed7b571d43404dd)

Conflicts:
qa/tasks/ceph_manager.py
qa/tasks/cephadm.py: trivial resolutions

test/rgw: use "a // b" instead of "a / b"

for expressions where the value is expected to be integer. as in
python3, `a / b` returns a float.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 253fc931800a0fac8f86c9228f58ba096f58dfeb)

qa/tasks: use "a // b" instead of "a / b"

for expressions where the value is expected to be integer. as in
python3, `a / b` returns a float.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2089bf04b9c3b4bb065bf4c0bafa97419f2d87ff)

Conflicts:
qa/tasks/cephfs/test_full.py
qa/tasks/cephfs/test_strays.py: trivial resolutions

qa/tasks: do not random.choice(a_view)

use `random.sample()` instead of `random.choice(list(a_view))` for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 9ca45bd942ef1ed0d7ed61d87caffdfc97e54e5e)

Conflicts:
qa/tasks/ceph_manager.py: trivial resolution

pybind/mgr/dashboard/.pylintrc: drop xrange from the whitelist

since dashboard is now xrange-free, there is no need to have
xrange-builtin in whitelist, moreover, in python3 the error message
cannot be emitted at seeing xrange.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0930652318f0f2bbbefbe7dd829bc914405c9013)

pybind: s/xrange/range/

fortunately, cython does not complain at seeing xrange, but let's drop
the last bit of python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ad2bbbc1edb29d3623299fd25f72458623415a96)

qa,tes: s/xrange/range/

use six.moves.range for python3 copatibility, we can drop six after
migrating to python3.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit dd8d929c9ccd9864e4a5005e44eac2739e5fa762)

Conflicts:
qa/tasks/cephfs/test_failover.py: trivial resolution

qa/tasks: use next(iter(..)) for accessing first element in a view

in python2, dict.values() and dict.keys() return lists. but in python3,
they return views, which cannot be indexed directly using an integer index.

there are three use cases when we access these views in python3:

1. get the first element
2. get all the elements and then *might* want to access them by index
3. get the first element assuming there is only a single element in
   the view
4. iterate thru the view

in the 1st case, we cannot assume the number of elements, so to be
python3 compatible, we should use `next(iter(a_dict))` instead.

in the 2nd case, in this change, the view is materialized using
`list(a_dict)`.

in the 3rd case, we can just continue using the short hand of
```py
(first_element,) = a_dict.keys()
```
to unpack the view. this works in both python2 and python3.

in the 4th case, the existing code works in both python2 and python3, as
both list and view can be iterated using `iter`, and `len` works as
well.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d7258ea7fdcd81d4a88028a25b32ed5b278d0752)

Conflicts:
qa/tasks/barbican.py
qa/tasks/cbt.py
qa/tasks/vault.py: trivial resolutions

qa/tasks/cephfs: use StringIO for capturing str

if we don't expect non-utf8 in stdout, use StringIO instead of BytesIO.

see also d8d44ed1566b19eec055e07da2a0fed88fed4152

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f10f86579684a0d1b8d99787b8e19709a64df7d6)

Conflicts:
qa/tasks/cephfs/mount.py: trivial resolution

qa: get rid of iteritems for python3 compatibility

Fixes: https://tracker.ceph.com/issues/42267
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 50b1823faca0e73fe81a3c2aca12dae3e250abb6)

Conflicts:
qa/tasks/barbican.py
qa/tasks/ragweed.py
qa/tasks/s3roundtrip.py
qa/tasks/s3tests.py
qa/tasks/s3tests_java.py
qa/tasks/vault.py: trivial resolution

qa/tasks/ceph.py: quote "<kind>" in command line

otherwise bash will intepret "kind" as a file when handling command like
```
sudo zgrep <kind> /var/log/ceph/valgrind/* /dev/null | sort | uniq
```
and try to feed its content to zgrep, and write the output of zgrep
to /var/log/ceph/valgrind/*. this is not the intended behavior. what we
what to do is to pass "<kind>" as an argument to zgrep, along with
the globbed files names which matches "/var/log/ceph/valgrind/*".

in this change, "<kind>" is quoted as in the command line. it's also
what `pipes.quote()` does before the change of
35cf5131e7152ce20d916aa99c124751d6a97f5c.

this addresses the regression introduced by
35cf5131e7152ce20d916aa99c124751d6a97f5c.

Fixes: https://tracker.ceph.com/issues/44454
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit da736c22c5ac39f3b45018f6a9a52cc469e504d5)

tasks/ceph: drop testdir replacement in skeleton_config

The str.format is not used anymore for ceph.conf.template

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit f1a96b55dc9b383810cbae6b2c4e953806ed9313)

qa/tasks/cephfs: fix syntax error

it's a regression introduced by 9f6c764f10

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 496d7e21325e8efa6bbac57a519db2f07d8d7ee5)

qa/tasks/mgr: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit e7473b631d862110160c3ef95aa6edce11fe2092)

qa/tasks/ceph_fuse: fix import for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit ac2fa95b21a3d4b3719af39c6416606111458cf8)

qa/tasks/qemu: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 914c0816e639b16780cfb991f5a89ffbb496d001)

qa/tasks/ragweed: get rid of cString for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit f0c5f56ef8ba869ccacf799a9ed6df9dedf0caaa)

qa/tasks/s3tests: py3 compat

- use string.ascii_uppercase instead string.uppercase
- use six.ensure_str for bytes when required
- use six.ensure_binary if needed
- get rid of dict.itervalues in favor of dict.values
- get rid of cStringIO.StringIO in favor io.BytesIO

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 478083e7319b0fc0eba5fc4f7fa9193ece784b15)

qa/tasks/rgw_logsocket: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 4cf205f2856b66140f798484bff4951c95ad01a1)

qa/tasks: fix import module path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 1c81d3873d904e42f7a534db0e419c23e5220aed)

qa/tasks/cephfs/kernel_mount: get rid of StringIO.StringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 875b45f3c903781ba99dbd2fcf4d2fc89001c25b)

Conflicts:
qa/tasks/cephfs/kernel_mount.py: trivial resolution

qa/tasks/kclient: fix import for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 06f22a3097c66a3f04e2b00a388aa468fa5dbd30)

qa/tasks/rbd_fio: get rid of StringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit db7ae8eff60a9e3cb26e4e5c736a9691cd3224e3)

qa/tasks/s3readwrite: avoid using cStringIO

cStringIO is not offered by python3, so avoid using using this module.

this file does not exist in master anymore, so this change is not
cherry-picked from master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/rbd: get rid of cStringIO for py3

Use io.BytesIO and six.ensure_str for py3 compatibility
instead of cStringIO.StringIO

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 7b72097cd1cf93f5445483802344327f1e3da0bc)

qa: Run flake8 on python2 and python3

To be able to catch problems with python2 *and* python3, run flake8
with both versions. From the flake8 homepage:

It is very important to install Flake8 on the correct version of
Python for your needs. If you want Flake8 to properly parse new
language features in Python 3.5 (for example), you need it to be
installed on 3.5 for Flake8 to understand those features. In many
ways, Flake8 is tied to the version of Python on which it runs.

Also fix the problems with python3 on the way.
Note: This requires now the six module for teuthology. But this is
already an install_require in teuthology itself.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
(cherry picked from commit bdcc94a1d1f659b1524fdc6bb8bd1da6d38a30d2)

Conflicts:
qa/CMakeLists.txt
qa/tasks/ceph_manager.py
qa/tasks/cephfs/xfstests_dev.py
qa/tasks/cram.py
qa/tasks/manypools.py
qa/tasks/ragweed.py
qa/tasks/s3tests.py
qa/tasks/vstart_runner.py
qa/tasks/workunit.py
qa/tox.ini: trivial resolutions

qa/tasks/devstack: use six.StringIO for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 09ae86aa9ecc276d75a14395aaabc12de1115c04)

qa/tasks/swift.py: s/StringIO/BytesIO/

cStringIO is not offered by python3, so avoid using using this module.

this file does not exist in master anymore, so this change is not
cherry-picked from master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/radosgw_admin: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 0a224991cfb4cb9683e27c9afe6b922415a6f8ed)

tasks/s3roundtrip.py: do not use cStringIO

cStringIO is not offered by python3, so avoid using using this module.

this file does not exist in master anymore, so this change is not
cherry-picked from master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/util/rgw: get rid of cString for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 683421ea24bbed4ba72ea96a9055134bed554696)

qa/tasks/rgw: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit eec9479f6d9ee0ed8dfa705491159d49a89bec19)

qa/tasks: fix imports for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit e46eb8348e0639ea162b7e224bca40e0257ad6ef)

qa/tasks/ceph_manager: get rid of CStringIO for py3

Use io.BytesIO instead cStringIO.StringIO for py3 compatibility

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 204ceee156cbb8a20bdf56efb0cd0610ee4c107e)

Conflicts:
qa/tasks/ceph_manager.py: trivial resolution

qa/tasks: use spaces intead of tabs for indent

to address the errors of
```
TabError: inconsistent use of tabs and spaces in indentation
```

somehow in master, we are using spaces instead of tabs, so this change
is not cherry-picked from master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: use python3 compatible print

Fixes: https://tracker.ceph.com/issues/42210
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 7e87f80a8754c93c10ff937844ad456d4a236f57)

Conflicts:
qa/tasks/cephfs/mount.py
qa/tasks/cephfs/test_journal_migration.py
qa/workunits/mon/caps.py
src/test/rgw/rgw_multi/tests_ps.py
src/test/rgw/rgw_multi/zone_ps.py
src/test/rgw/test_multi.py: trivial resolutions

qa/tasks/cephfs: get rid of StringIO for py3

Use io.BytesIO and six.ensure_str for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 9f6c764f10f99cbf5ca164c2681b001d29505d9d)

Conflicts:
qa/tasks/cephfs/fuse_mount.py
qa/tasks/cephfs/mount.py
qa/tasks/cephfs/test_acls.py
qa/tasks/cephfs/test_cephfs_shell.py
qa/tasks/cephfs/test_journal_migration.py
qa/tasks/cephfs/xfstests_dev.py: trivial resolution

qa/tasks/mgr: partial revert of 'import with full path'

this change partially reverts #34139

using relative import helps with readability and ease the pain to write
down the full parent module name

in #34139, all relative imports were replaced with full path, because we
were using following code to verify if the code is python3 compatible:

```
  mod_spec = importlib.util.spec_from_file_location(mod_name, path)
  mod = importlib.util.module_from_spec(mod_spec)
  mod_spec.loader.exec_module(mod)
```

but this does not work with submodule which can import using relative
import without specifying the name of the package and its parent module.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b0bdbc34147c7e795e1a01cce1197b990c48effb)

Conflicts:
qa/tasks/mgr/dashboard/test_auth.py
qa/tasks/mgr/dashboard/test_cephfs.py
qa/tasks/mgr/dashboard/test_crush_rule.py
qa/tasks/mgr/dashboard/test_host.py
qa/tasks/mgr/dashboard/test_orchestrator.py
qa/tasks/mgr/dashboard/test_user.py: trivial resolutions

qa/rgw: partial revert of 'import with full path'

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit aa60edcec89b1538070b316c496e55bd0e15f966)

Conflicts:
src/test/rgw/rgw_multi/tests_az.py
src/test/rgw/rgw_multi/tests_ps.py
src/test/rgw/rgw_multi/zone_az.py: trivial resolution

test/rgw: should import parse not urlparse

urlparse is a function while parse is a submodule where parse_qsl() is
provided

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0db88f775b204b44d5c7b3c50e5ad3fffad029f1)

qa: import py3 compatible modules using six

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 80b71ef461043bfa274b2c73fb0697a0e75759cd)

Conflicts:
qa/tasks/barbican.py
qa/tasks/vault.py: trivial resolution

qa: import with full path

to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 947a74349d2ce1266cb0f3c6a7030605999e21b3)

Conflicts:
qa/tasks/cephadm.py
qa/tasks/cephfs/test_acls.py
qa/tasks/cephfs/test_openfiletable.py
qa/tasks/cram.py
qa/tasks/mgr/dashboard/test_auth.py
qa/tasks/mgr/dashboard/test_cephfs.py
qa/tasks/mgr/dashboard/test_crush_rule.py
qa/tasks/mgr/dashboard/test_host.py
qa/tasks/mgr/dashboard/test_orchestrator.py
qa/tasks/mgr/dashboard/test_rbd_mirroring.py
qa/tasks/mgr/dashboard/test_settings.py
qa/tasks/mgr/dashboard/test_user.py
src/test/rgw/rgw_multi/tests_az.py
src/test/rgw/rgw_multi/tests_ps.py
src/test/rgw/rgw_multi/zone_az.py: trivial resolution

qa/tasks/workunit: py3 compat

Use six.ensure_str for `get_file()` output in order to
make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit ae6befb3be46d84da67092aff8bff95d9f646aea)

qa/tasks: fix import module path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 1c81d3873d904e42f7a534db0e419c23e5220aed)

Conflicts:
qa/tasks/mon_thrash.py
qa/tasks/workunit.py: trivial resolutions

qa/tasks/thrashosds: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit c26ea687d48b7742eb527a77385321b0138bd634)

qa/tasks/omapbench: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 1140b5f027be472ff7f1c949305d5d9400cba191)

qa/tasks/mgr: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit eb10276c1c062a196bd0391691f6e8ef5378f5b3)

qa/tasks/watch_notify_stress: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 6e328edcbcab28b6d6a3abbb46b48a6a18e660c2)

qa/tasks/osd_max_pg_per_osd: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit b2d52220e3b7e31b16bb6442848c5ce2f9d4fc76)

qa/tasks/radosbench.py: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 3b21067bbbf5e375700554e8b985bc4762d03a46)

qa/tasks/ceph: get rid of cStringIO for py3 compat

Use io.BytesIO instead of cStringIO.StringIO
Use six.ensure_str whenever it needs to convert binary to str.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 35cf5131e7152ce20d916aa99c124751d6a97f5c)

qa/tasks/admin_socket: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 3eb341db27290d01417d7809b198bd17a7f5fa3a)

qa/tasks/systemd: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit a28d3473055ca1440949f3fb99161246f8bb71b1)

qa/tasks/scrub_test: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 1d6c4228cd83ad930119f77a22feae2d3df57589)

qa/tasks/netem: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit fa3db2529fb1b7f9f5145d3d8752d38b439a32f8)

qa: Enable flake8 tox and fix failures

There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
(cherry picked from commit 0127cd1e8817b05b1c3150540b021f9a24b47089)

Conflicts:
qa/tasks/barbican.py
qa/tasks/cephadm.py
qa/tasks/cephfs/test_cephfs_shell.py
qa/tasks/cephfs/xfstests_dev.py
qa/tasks/daemonwatchdog.py
qa/tasks/mgr/dashboard/test_cephfs.py
qa/tasks/mgr/dashboard/test_orchestrator.py
qa/tasks/mgr/dashboard/test_rbd.py
qa/tasks/mgr/mgr_test_case.py
qa/tasks/mgr/test_orchestrator_cli.py
qa/tasks/s3tests.py
qa/tasks/s3tests_java.py
qa/tasks/vstart_runner.py
qa/workunits/mon/caps.py: trivial resolutions,
          and drop the the change to qa/CMakeLists.txt, as we don't have
          add_tox_test() back in nautilus

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 99e9e82d646a9db1773ca97da7460dbfb5a0b415)

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 71ada20a0e362db3cb5ce2a4afa753082f62f9cd)

qa/tasks/reg11184: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit fa16954df94a763d2604f58c119b8ee943fa2cfe)

qa/tasks/radosbenchsweep: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 7ffcd00b652b955f52df7a27ce2130cbc40ebd72)

qa/tasks/divergent_priors2: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit b6eefbb02896043333f264023274e83ec36b8362)

qa/tasks/ceph_objectstore_tool: get rid of cStringIO for py3

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit b502bc17d17c784b83f652b79f04eccfd1048805)

qa/tasks/ceph_manager: ensure str for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 4c992baf251d272800e76bb40584f169163f6c1b)

qa/tasks/vstart_runner: use io.BytesIO for py3 compat

Use io.BytesIO instead of StringIO for py3 compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 49592af4005808db4ed5301e9072525bd709f44a)

Conflicts:
qa/tasks/vstart_runner.py: trivial resolution

vstart_runnner: add sh method to LocalRemote

The method is a wrapper of teuthology.misc.sh

Fixes: https://tracker.ceph.com/issues/43496
Signed-off-by: Rishabh Dave <ridave@gmail.com>
(cherry picked from commit b7d740ede7ff5839fdce32a5bfc78ca935990208)

qa/tasks/manypools: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 5f876961ef584ab07c8f9ef50f10f010a5e7510f)

Conflicts:
qa/tasks/manypools.py: trivial resolution

qa/tasks/ceph_deploy: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 982ba399b34765ca888dafbbf4221bab53e7964a)

tasks/ceph: get rid of iteritems for python3

For python3 compatibility use items() instead of iteritems().

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 9b5bd6c7155579d3b903ef0eb0ac8eddfe8f29e1)

Merge pull request #35305 from tchaikov/nautilus-qa-rgw-s3a-hadoop

nautilus: qa/rgw: s3a-hadoop task defaults to maven-version 3.6.3

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35181 from smithfarm/wip-45644-nautilus

nautilus: rgw/notifications: version id was not sent in versioned buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35162 from tchaikov/nautilus-qa-keystone

nautilus: qa/keystone: use bindep to discover keystone's dependencies

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35319 from neha-ojha/wip-45772-nautilus

nautilus: qa/suites/upgrade/nautilus-p2: disable more min pg per osd warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35248 from rhcs-dashboard/wip-45582-nautilus

nautilus: mgr/dashboard/grafana: Add rbd-image details dashboard

Merge pull request #35199 from smithfarm/wip-45600-nautilus

nautilus: mds: don't shallow copy when decoding xattr map

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35185 from smithfarm/wip-45497-nautilus

nautilus: cephfs/tools: add accounted_rstat/rstat when building file dentry

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #35163 from tchaikov/nautilus-upgrade-from-luminous-fs-disable-too-few-pgs

nautilus: qa/suites/fs/upgrade: disable mon_pg_warn_min_per_osd

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35149 from Vicente-Cheng/wip-45602-nautilus

nautilus: mds: Handle blacklisted error in purge queue

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35118 from Vicente-Cheng/wip-45478-nautilus

nautilus: client: only set MClientCaps::FLAG_SYNC when flushing dirty auth caps

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35117 from Vicente-Cheng/wip-45474-nautilus

nautilus: bash_completion: Do not auto complete obsolete and hidden cmds

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/suites/upgrade/nautilus-p2p: disable more min pg per osd warnings

This follows 58eb3edc8478c993c5446475df58d659d3f6d356 and
b75907e7575dbdc5888c198f354cb5f71bf21ab3.

This change is not cherry-picked from master since it already has
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4.

Fixes: https://tracker.ceph.com/issues/45772
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/rgw: s3a-hadoop task defaults to maven-version 3.6.3

binaries for v3.6.2 are no longer available under
http://www-us.apache.org/dist/maven/maven-3/

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 82351b9874a9bb5e2e07436303be34e9f5db0ae5)

Conflicts:
qa/tasks/s3a_hadoop.py: trivial resolution

Merge pull request #35214 from shyukri/wip-43990-nautilus

nautilus: mgr/insights: fix prune-health-history

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35213 from shyukri/wip-43464-nautilus

nautilus: mgr: close restful socket after exec

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35212 from shyukri/wip-43098-nautilus

nautilus: mgr/dashboard: Display the aggregated number of request

Reviewed-by: Stephan Müller <smueller@suse.com>