qa/tasks/keystone.py: use list(a_dict) for concatenating lists

`dict.items()` does not return a list in python3, so we need to convert
it to a list first.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 35a3d5f7f8a53e667d3196aa72be5b495b70e91f)

qa/tasks/cephfs: cast mds_recall_warning_decay_rate to float

this change should address the failure of
```
2020-04-05T15:14:23.088 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T15:14:23.088 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_client_limits.py", line 110, in test_client_pin_mincaps
2020-04-05T15:14:23.089 INFO:tasks.cephfs_test_runner:    self._test_client_pin(True, 200)
2020-04-05T15:14:23.089 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_client_limits.py", line 71, in _test_client_pin
2020-04-05T15:14:23.090 INFO:tasks.cephfs_test_runner:    self.wait_for_health("MDS_CLIENT_RECALL", mds_recall_warning_decay_rate*2)
2020-04-05T15:14:23.091 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/ceph_test_case.py", line 152, in wait_for_health
2020-04-05T15:14:23.091 INFO:tasks.cephfs_test_runner:    self.wait_until_true(seen_health_warning, timeout)
2020-04-05T15:14:23.092 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/ceph_test_case.py", line 193, in wait_until_true
2020-04-05T15:14:23.093 INFO:tasks.cephfs_test_runner:    if elapsed >= timeout:
2020-04-05T15:14:23.093 INFO:tasks.cephfs_test_runner:TypeError: unorderable types: int() >= str()
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 162be9210613a4b4a921fa0ff285c869548d7d49)

qa/tasks/cephfs: allow caller to use BytesIO when calling rados()

when the caller expects binary data, it should pass BytesIO as stdout.

this change shold address the failure of
```
2020-04-05T12:47:25.335 INFO:tasks.cephfs_test_runner:======================================================================
2020-04-05T12:47:25.336 INFO:tasks.cephfs_test_runner:ERROR: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2020-04-05T12:47:25.336 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-04-05T12:47:25.336 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:47:25.337 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_forward_scrub.py", line 75, in test_apply_tag
2020-04-05T12:47:25.337 INFO:tasks.cephfs_test_runner:    self.assertTagged(inos[dirpath], tag, self.fs.get_metadata_pool_name())
2020-04-05T12:47:25.337 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_forward_scrub.py", line 98, in assertTagged
2020-04-05T12:47:25.338 INFO:tasks.cephfs_test_runner:    "scrub_tag"
2020-04-05T12:47:25.338 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_forward_scrub.py", line 35, in _read_str_xattr
2020-04-05T12:47:25.339 INFO:tasks.cephfs_test_runner:    strlen = struct.unpack('i', output[0:4])[0]
2020-04-05T12:47:25.339 INFO:tasks.cephfs_test_runner:TypeError: a bytes-like object is required, not 'str'
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 20dafc6d52acc4af235dc061fa67ecd617a90159)

qa/tasks/cephfs: use BytesIO for capturing binary

this change partially reverts e46eb8348e0639ea162b7e224bca40e0257ad6ef.

xattrs could contain non-utf8 encoded data, and should be captured using
BytesIO. moreover, it will be fed to `ceph-dencoder`, which expects
binary when performing "import".

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit eb2c6b3740cee7c8e28c163f73525cbabf2e7faf)

qa/tasks/fs.py: use six.viewitems()

in python3, dict.viewitems() is replaced with dict.items().

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 51fc7a992204fb14aba2eb29d96c1785a79ca27f)

qa/tasks/cephfs: use assertCountEqual if PY3

assertItemsEqual is renamed to assertCountEqual in Python3.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 21d6c5a95b4a6873bc8f35a74e3f48436bd96264)

qa/tasks/cephfs: compare number with number

to address following failure:
```
2020-04-05T12:25:30.997 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:25:30.997 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_misc.py", line 50, in test_root_rctime
2020-04-05T12:25:30.998 INFO:tasks.cephfs_test_runner:    self.assertGreaterEqual(rctime, t-10)
2020-04-05T12:25:30.998 INFO:tasks.cephfs_test_runner:  File "/usr/lib/python3.5/unittest/case.py", line 1215, in assertGreaterEqual
2020-04-05T12:25:30.998 INFO:tasks.cephfs_test_runner:    if not a >= b:
2020-04-05T12:25:30.999 INFO:tasks.cephfs_test_runner:TypeError: unorderable types: str() >= float()
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 83e41daec4d7353361d3897770ad31f79bc80681)

qa/tasks/cephfs: do no radom.sample(a_dict)

collect the keys instead of filtering a dict,
to address following failure:
```
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 29, in test_files_throttle
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:    self._test_throttling(self.FILES_THROTTLE)
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 96, in _test_throttling
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner:    return self._do_test_throttling(throttle_type)
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 176, in _do_test_throttling
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner:    mds_max_purge_ops = int(self.fs.get_config("mds_max_purge_ops", 'mds'))
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/filesystem.py", line 285, in get_config
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner:    service_id = random.sample(filter(lambda i: self.mds_daemons[i].running(), self.mds_daemons), 1)[0]
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/virtualenv/lib/python3.5/random.py", line 311, in sample
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:    raise TypeError("Population must be a sequence or set.  For dicts, use list(d).")
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:TypeError: Population must be a sequence or set.  For dicts, use list(d).
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 40d0356249193bf4aaa2d38923eef19caa63eb59)

pybind: s/xrange/range/

fortunately, cython does not complain at seeing xrange, but let's drop
the last bit of python2.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ad2bbbc1edb29d3623299fd25f72458623415a96)

qa/tasks: use next(iter(..)) for accessing first element in a view

in python2, dict.values() and dict.keys() return lists. but in python3,
they return views, which cannot be indexed directly using an integer index.

there are three use cases when we access these views in python3:

1. get the first element
2. get all the elements and then *might* want to access them by index
3. get the first element assuming there is only a single element in
   the view
4. iterate thru the view

in the 1st case, we cannot assume the number of elements, so to be
python3 compatible, we should use `next(iter(a_dict))` instead.

in the 2nd case, in this change, the view is materialized using
`list(a_dict)`.

in the 3rd case, we can just continue using the short hand of
```py
(first_element,) = a_dict.keys()
```
to unpack the view. this works in both python2 and python3.

in the 4th case, the existing code works in both python2 and python3, as
both list and view can be iterated using `iter`, and `len` works as
well.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d7258ea7fdcd81d4a88028a25b32ed5b278d0752)

qa/tasks/cephfs/mount: to make sure that the count/seek are int type

To fix "stderr:dd: invalid number: '5.0'".

Fixes: https://tracker.ceph.com/issues/45247
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 65f97690af8ca49e7a8b508621bf47d4fc004853)

qa/tasks/cephfs/test_volumes: to make sure the size is int type

Fix "Error EINVAL: invalid size option '4915200.0'".

In pytho2 int/2 will get a int type result, but in python3 it maybe
a float type.

Fixes: https://tracker.ceph.com/issues/45247
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit afe19f24baf3983a42f6213c04efc20307c286d2)

qa/tasks/cephfs/cephfs_test_case: switch 'dict_keys' object to list

Fix the "TypeError: 'dict_keys' object does not support indexing" under
py3.

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 666974e0a663aba6aff37cf78728c44b0e3a08c1)

qa/tasks/cephfs: switch to StringIO to fix byte-like errors

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit dd4fc808a8a26ebbfae5b5e4bc3b6c99bf0f5d37)

qa/cephfs: fix AssertionError: CommandFailedError not raised under py3

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 7f57576123b81051a5e64f7c7234153ee11f4062)

qa/tasks/cephfs/test_scrub: fix self.assertEqual no attribute '_type_equality_funcs'

Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 89d06490886fb5cf840cadde49913dd497c14cf8)

qa: return a string via getfattr

Fixes: https://tracker.ceph.com/issues/45666
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 3c1330795add81126424e9e0a49cc07881fbdf8f)

qa/tasks/cephfs/test_cephfs_shell: assertRegex(text, regex)

per Python3 doc of unittest,

> assertRegex(text, regex, msg=None)

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 981ad349871f6b9fb71fe6b2a9d0fd65e3720deb)

qa/tasks/cephfs: get rid of StringIO for py3

Use io.BytesIO and six.ensure_str for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 9f6c764f10f99cbf5ca164c2681b001d29505d9d)

qa/tasks/cephfs: return ascii decoded xattr

as callers of `TestForwardScrub._read_str_xattr()` expects str.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7f48f61bdc8f88e1025796aec5158bbd69b1eb65)

qa/tasks/ragweed.py: open file in text mode

this file is a template for `boto.cfg` which only contains text, so we
can just open it in text mode, and the file-like object will be passed
to run() as stdin, which is able to consume strings.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3809737fb859ebebe72d635708e0d69676cc5aa6)

qa/tasks/s3tests_java.py: do not change a dict while iterating it

in Python3, dict.items() returns a view instead of an instance of list,
so we have to materialize the view for changing the dict being iterated.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit dcecf1dbe4e456c060259b8591a887abc1f228bd)

qa/tasks: pass encoding to Key.get_content_as_string()

we assume that boto.Key.get_content_as_string() returns str instead of
bytes, and compare the return value with a string, so, to ensure that
lhs and rhs are both strings, we need to decode the returned content.

since we always store strings composed with ASCII, it's safe to use
'ascii' to decode them.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 9ccf88d95fa5d184c56ac638bed37c3e3101356d)

test/rgw: pass encoding to Key.get_content_as_string()

we assume that boto.Key.get_content_as_string() returns str instead of
bytes, and compare the return value with a string, so, to ensure that
lhs and rhs are both strings, we need to decode the returned content.

since we always store strings composed with ASCII, it's safe to use
'ascii' to decode them.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit fcf4de3855f01e80b90d246529916a5915fe413d)

test/rgw/rgw_multi/tests.py: decode retval of bucket.get_policy()

return value of bucket.get_policy() is a bytes, so we need to decode it
before comparing it with a string, since the content of policy is ASCII,
it'd be suffice to decode it with 'ascii'.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ea63b52b1705e80977ce1fe6d2dc276ea9477344)

qa/tasks/cephadm.py: decode data returned by teuthology.get_file()

it's used as part of the command line sent to shell, so we need to
decode it first.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6eb57a2a4fc63771096f8fd08c02b2bda49c9301)

test/rgw/rgw_multi: return (str, int) from make_request()

so its consumer can feed the returned str to json.loads()

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 23ddf03d26c2c8550b8c9a0c5334fe7b8f4e1682)

test/rgw/rgw_multi: pass bytes to hmac.new()

in Python3, `key` passed to `hmac.new()` should be bytes or bytearray

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit adfcf08588e6031be837602dc4d1f8d6d14d795b)

qa/tasks/mgr: s/urllib.quote_plus/parse.quote_plus/

in Python3, urllib.quote_plus is moved into urllib.parse, let's use
six.moves at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6c2f40a5281a1cccf0cab82c620ef1c6a40dd6ef)

qa/tasks/ceph_objectstore_tool.py: use str.startswit

in Python3, string module does not offer `string.find()` anymore, let's
use `str.find()` method instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 77ec9ce88df54b7758e56eeef1b3cf3f14952513)

Conflicts:
qa/tasks/ceph_objectstore_tool.py: trivial resolution

qa/tasks/ceph_manager.py: do not return a filter

as the caller might want to `len(manager.get_osd_status()['raw'])`, and
`len()` does not accept a `filter` object.

also, the filtered osd statuses are printed out using `self.log()`, so
we should materialize the `filter` object before sending it to logging
facility. otherwise we will have something like:
```
2020-04-08T02:58:37.001 INFO:tasks.ceph.ceph_manager.ceph:<filter object at 0x7f5a080e1518>
```
in the logging message.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6bc09c5041d025c7ed3638ccfbf6496e1626abfb)

qa/tasks/openssl_keys.py: sort cert configs before creating certs

we cannot rely on the order in which items are arranged in a dict, the
order varies from version to another. in Python2, it happens to work,
and we can always have the self-signed cert added first. but in Python3,
it does not. and an exception is thrown
```
teuthology.exceptions.ConfigError: ssl: ca root not found for
certificate rgw.client.0
```

in this change, before creating certs, the settings are reordered so
that the self-signed ones are created first.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f28a5fef3b8ddb97962f91cc78174fd6e1431fed)

pybind/mgr/dashboard/.pylintrc: drop xrange from the whitelist

since dashboard is now xrange-free, there is no need to have
xrange-builtin in whitelist, moreover, in python3 the error message
cannot be emitted at seeing xrange.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0930652318f0f2bbbefbe7dd829bc914405c9013)

qa/tasks: do not random.choice(a_view)

use `random.sample()` instead of `random.choice(list(a_view))` for better performance.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 9ca45bd942ef1ed0d7ed61d87caffdfc97e54e5e)

test/rgw: use "a // b" instead of "a / b"

for expressions where the value is expected to be integer. as in
python3, `a / b` returns a float.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 253fc931800a0fac8f86c9228f58ba096f58dfeb)

qa,tes: s/xrange/range/

use six.moves.range for python3 copatibility, we can drop six after
migrating to python3.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit dd8d929c9ccd9864e4a5005e44eac2739e5fa762)

qa/tasks: use "a // b" instead of "a / b"

for expressions where the value is expected to be integer. as in
python3, `a / b` returns a float.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2089bf04b9c3b4bb065bf4c0bafa97419f2d87ff)

qa/tasks/tox: use the latest stable tox

in hope to avoid any ancient bugs

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b7152e8e943b713e779aec0c2414986bd556a423)

qa/valgrind.supp: less specific when suppressing issues/22052

Fixes: https://tracker.ceph.com/issues/22052
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a8e9fd71d79f7696a47773acdcc738f47a88d8d8)

qa/tasks/rbd_fio: update NamedTemporaryFile usage for python3

Open the file in text mode to avoid 'bytes' objects.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 4e3160a019b5af51e498793aaa24a2a7b82b4b54)

qa/tasks/rbd: update NamedTemporaryFile usage for python3

- open the file in text mode to avoid 'bytes' objects

- 'bufsize' is now spelled 'buffering' and switching buffering off
  with 'buffering=0' is only allowed in binary mode.  As we need the
  file to be either in the page cache or on disk before we copy it
  to the remote, call flush().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit aba9910388cce7c03aa90726b5de1263298ac8ec)

qa/test_exports: fix TestExports failure under new python3 compability changes

self.mount_a.client_remote.sh() returns an 'str' object rather than a StringIO object. Hence the p.stdout.getvalue() produces an error. This commit fixes this and also fix str and byte mismatch as byte and string were the same object in Python2 but this is not the case in Python3.

Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
(cherry picked from commit b98f3e388b6a38e91c2ee064466ae8f130a5bcda)

qa/tasks: use list comprehension for checking the length

instead of using filter(), use `sum()` for counting its
length, as in Python3, `filter()` actually returns a `filter` object
instead of a list.

in this change, `filter()` calls are replaced with `sum()`
for Python3 compatibility.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit 726c59be58473cfe53fa3ea63cf35b58ed797886)

qa/tasks/qemu: open file in text mode

in python3, if a file is opened in 'rb' mode, we will have bytes when
reading from it, as a result, we have following failure like:
```
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 32, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_tchaikov_ceph_nautilus-py3/qa/tasks/qemu.py", line 124, in generate_iso
    test_setup = ''.join(f.readlines())
TypeError: sequence item 0: expected str instance, bytes found
```

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0b21c597d438fcba43629d0795b2d4ff6112c9cf)

mgr/insights: fix python3 DeprecationWarning

insights/module.py:167: DeprecationWarning: invalid escape sequence \d
    r = "ceph version (?P<release>\d+)\.(?P<major>\d+)\.(?P<minor>\d+)"

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 41ffb701eca6280df00cd526edfce3a9c851b53b)

vstart: remove commented-out vault config options

we don't have any vstart automation for rgw encryption with vault. if
anyone wants to do local testing, they can follow instructions in
the documentation

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9f0ccb5f17e8723eb62974b87c4d5ed4a621f33a)

test/rgw: Cluster.admin() returns output as utf-8 str

fixes the TypeError:

  File "rgw_multi/tests.py", line 1107, in test_bucket_sync_disable
    check_buckets_sync_status_obj_not_exist(zone, buckets)
  File "rgw_multi/tests.py", line 438, in check_buckets_sync_status_obj_not_exist
    if log_list.find(':'+bucket+":") >= 0:
TypeError: argument should be integer or bytes-like object, not 'str'

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ff5ce2cf1457e1945aea0fe4a023bc48bffe1174)

radosgw-admin: period commit writes messages to stderr

the period's json output is written to stdout. any other messages should
be written to stderr

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e48a20076274174705c5e6efde3730e993a0538d)

qa/tasks/cephfs: remove sys.exc_clear() calls

sys.exc_clear() was removed in Python3, see
https://docs.python.org/3/whatsnew/3.0.html#index-22. so we should not
call it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 702de9590cdfe6c95ef8e87cb74cd05944e380c9)

vstart_runner.py: use tuple instead of set

Using python3 leads to an error that says
"TypeError: unhashable type: 'Raw'".

Fixes: https://tracker.ceph.com/issues/45446
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit cb0bfc8fab4372b0caf5102823883b0b913d4fcb)

mgr: logger.warn() -> logger.warning()

logger.warn() is the same as logger.warning(), but was removed in py3.3

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit ef25650721e07f97da4a71cd0b1f9d3bfc862769)

pybind/ceph_volume_client: logger.warn() -> logger.warning()

logger.warn() is the same as logger.warning(), but was removed in py3.3

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit fe4c7fd7471bf724c4a1f027580d9bae3ddc24eb)

qa/tasks: logger.warn() -> logger.warning()

logger.warn() is the same as logger.warning(), but was removed in py3.3

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 44bcee8d1a637b1429a0c9576397f70f0a660ec2)

Merge pull request #35475 from sebastian-philipp/octopus-backport-34879-34860-35069-35262-35320-35322-35342-35359-35271-35375-35365-3541

octopus: cephadm batch backport June (2)

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35487 from jdurgin/wip-radosbench-qa-octopus

octopus: qa/tasks/radosbench: use long form of option for compatibility

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35445 from smithfarm/wip-45884-octopus

octopus: qa/*/osd-scrub-repair.sh: Convert to python3 print syntax

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35443 from smithfarm/wip-45779-octopus

octopus: qa/workunits/rados/test_envlibrados_rocksdb: Build correct rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #35441 from smithfarm/wip-45673-octopus

octopus: qa: run install task only once

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephadm: minor py2 fix

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35440 from smithfarm/wip-45885-octopus

octopus: rbd-mirror: stop local journal replayer first during shut down

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #35439 from smithfarm/wip-45801-octopus

octopus: rbd: librbd: Watcher should not attempt to re-watch after detecting blacklisting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35438 from smithfarm/wip-45799-octopus

octopus: rbd: librbd: make rbd_read_from_replica_policy actually work

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35437 from smithfarm/wip-45763-octopus

octopus: rbd-mirror: wait for in-flight start/stop/restart

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #35436 from smithfarm/wip-45597-octopus

octopus: mgr/rbd_support: rename "rbd_trash_trash_purge_schedule" oid

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35435 from smithfarm/wip-45580-octopus

octopus: rbd: pybind: RBD.create() method's 'old_format' parameter now defaults to False

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #35430 from smithfarm/wip-45881-octopus

octopus: rbd-mirror: don't hold (stale) copy of local image journal pointer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35433 from smithfarm/wip-45782-octopus

octopus: rgw: amqp: fix the "routable" delivery mode

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #35432 from smithfarm/wip-45777-octopus

octopus: rgw: url: fix amqp urls with vhosts

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #35431 from smithfarm/wip-45645-octopus

octopus: rgw: lc: enable thread-parallelism in RGWLC

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35434 from smithfarm/wip-45844-octopus

octopus: rgw: gc: Clearing off urgent data in bufferlist, before

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

qa/tasks/radosbench: use long form of option for compatibility

Since the short version of --object-size changed from -o to -O, it
does not work with upgrade tests.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 9673d37f1b72681f7583bf36537575d876da9712)

Revert "qa/tasks/radosbench: only set object size if != block size"

This reverts commit fb991fa5f62c4aaa6c564a6d15aeee4bfaa33b39.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
qa/tasks/radosbench.py - extra context from -t, trivial resolution

(cherry picked from commit a78781d82e55bb01a8a1fb42eb072b944e336973)

Revert "qa/tasks/radosbench: fix usage of -O"

This reverts commit da19fd14dbe8bdb62ce376d9df7bf5578f169b9c.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
qa/tasks/radosbench.py - concurrency arg added in context, trival resolution

(cherry picked from commit 5a8c7f7d7f28a697611bab4fb816771b45cecd3e)

Merge pull request #34979 from rhcs-dashboard/wip-45460-octopus

octopus: mgr/test_orchestrator: fix _get_ceph_daemons()

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.

Merge pull request #35085 from tspmelo/wip-45539-octopus

octopus: mgr/dashboard: Fix HomeTest setup

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #35241 from smithfarm/wip-45704-octopus

octopus: cls/queue: fix empty markers when listing entries

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #35036 from badone/wip-octopus-fix-ftbfs-rocksdb_env_librados_test

octopus: rocksdb: Update to ceph-octopus-v5.8-1436

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35105 from smithfarm/wip-45585-octopus

octopus: doc: drop obsolete cache tier options

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: More robust way to deduce the systemd unit

Generate a better error message when using wrong args.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 182d926a4b8a2d1c5c66809dbc89e3521b24d941)

doc/cephadm: Adopt single-site RGW

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 10b4f7a6d822f175bf0aab6d8b26dd09cb737de0)

qa/cephadm: update adopt corpus

to 9cd9ad020d93b0b420924fec55da307aff8bd422

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit e45e9bf84e84a6e6c57c45b1f30283682ea40b07)

doc/cephadm: Add alternative error for missing CIDR network

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit bd43551304575a9eb22e0c01ba36ef535a65fa50)

doc/mgr: add docs for unmanaged=True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit a46d582422bf9e24af920420f73e4b447dafbc61)

mgr/cephadm: refactor templating in services

Using Jinja2 to render configs for services:
- Grafana
- Alertmanager
- Prometheus
- NFS
- iSCSI

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 544629833949a4104e236f19054fc614e0077f96)

mgr/cephadm: add template engine Jinja2

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 64bb27f2f62010de44d50bc1a876436159770b56)

doc/cephadm: align to all-available-devices flag

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit e9600666e0ab85203172b3d033322635197cd5c3)

mgr/orch: use keyword rather than positional arg

use keyword `service_id` from the orch cli

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 248888dab4bc3b173210c48fbdba031091fd3f19)

mgr/cephadm: use keyword rather than positional arg

use keyword `service_id` during the ServiceSpec tests

cephadm/tests/test_cephadm.py:51: in <module>
    class TestCephadm(object):
cephadm/tests/test_cephadm.py:508: in TestCephadm
    ???
../../python-common/ceph/deployment/service_spec.py:606: in __init__
    assert service_type == 'iscsi'
E   AssertionError

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit a4a523d5fb9753945f677d9bd8eabb2e970691af)

mgr/cephadm: fix RGWSpec mypy errors

cephadm/module.py:34: note: In module imported here:
cephadm/services/cephadmservice.py: note: In member "config" of class "RgwService":
cephadm/services/cephadmservice.py:167: error: Incompatible types in assignment (expression has type "List[str]", variable has type "str")

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit bd6ea9c6a92ffbfa8676212b85d590e12decdbf5)

python-common: fix pytest failure

523: error: Unsupported operand types for + ("str" and "None")
523: note: Right operand is of type "Optional[str]"

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 30a9c2f51747deb4861d1b6eaba667585e6957d0)

python-common: add ServiceSpec type checking

- add missing type checking to NFSServiceSpec and IscsiServiceSpec
- clean-up type checking for ServiceSpec and RGWSpec

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 5ca009c86857b9af301024db6a9c2d63ebb7cb3a)

cephadm: fix package is not running when enabler is not none

Signed-off-by: shinhwagk <shanghai_sl@hotmail.com>
(cherry picked from commit 76919e79bf7f9c462bf6df8bc00225e56ed9ba97)

mgr/cephadm: drop py2 compatible bits

since we've dropped Python3 support in octopus, there is no need to keep
the workaround for python2 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2c71d733d6327a29617ba05b5d56891d45ad28d8)

cephadm: Add a bunch of type annotations

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 888754ba2c74fba42bc26c9006361150ed7ddc16)

cephadm: Make easy manager modules development

A volume replaces content  in </usr/share/mgr> folder in manager containers.
The new argument <shared_mgr_folder> sets the source folder for the volume

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit f615591808a9ba2fc7b2f088016d33b72068e3e5)

qa/cephadm: Add RGW

Fixes: https://tracker.ceph.com/issues/43911
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit ffef64d046cb175732b4b97f898dabac5d83df10)

python-common: improve host matching funcs

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit d2aa0543d924d659fa0c5321b3fa9c607a44b5e3)

mgr/test_orchestrator: placementspec->host matching adaption

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 646ff0f749a2c7bd7ffb1356bb6cec674f7c2c57)

mgr/rook: placementspec->host matching adaption

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit d488de90ad7fbcf4ea838d8ca33511fdae7823d8)

mgr/cephadm: placementspec->host matching adaption

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 02231859860d096689c4f2e3f5ea4d502d8dc761)