Tiago Melo [Sat, 9 May 2020 02:28:39 +0000 (02:28 +0000)]
mgr/dashboard: Fix random E2E error in mgr-modules
This test failed at random times when it tried to find the new value of pool_ids
in the balancer module.
This happened because the value of pool_ids is automatically reverted by ceph,
so in some situations when we tried to read the new value,
it was already reverted and failed.
Enhanced the tests to be able to use any text input, not only the ones with
empty default values.
Conflicts:
src/pybind/mgr/dashboard/run-frontend-e2e-tests.sh
There was an extra square bracket in octopus that caused the conflict.
This was manually fixed, since the commit that removed it will not be
backported.
Kefu Chai [Sun, 31 May 2020 00:47:34 +0000 (08:47 +0800)]
qa/suites/rgw/tempest: update unsupported tests of tempest
after rerunning tempest with lastest radosgw, remove the supported
tests from the blacklist, and add the ones which are not supported
yet. now we can pass 123 tests in total.
also enable discoverity for better testing coverage, since it's
supported now.
Kefu Chai [Sun, 31 May 2020 00:38:00 +0000 (08:38 +0800)]
qa/tasks/keystone: use "keystone-manage bootstrap"
* qa/tasks/keystone.py:
instead of prefilling keystone manually, use "keystone-manage bootstrap"
instead. it helps to setup the admin user, a "Default" domain with
"default" id, and wire them up with the expected role and a "admin" project,
etc. as id of the admin domain is known to be "default", we can just use it
in our tests without querying openstack for the id of "Default"
domain. this is very handy.
* qa/suites/rgw/tempest/tasks/rgw_tempest.yaml:
use "Default" for domain name. as "Default" is the name of the domain
created by bootstrap, while "default" is its id.
* qa/suites/rgw/crypt/2-kms/barbican.yaml:
remove settings to bootstrap keystone
Kefu Chai [Thu, 28 May 2020 16:51:39 +0000 (00:51 +0800)]
qa/suites/rgw/tempest: use the latest tempest supporting py3.5
in case we need to use ubuntu xenial for testing, xenial only had python
3.5 packaged. and tempest 23.0 was the last version which supports
python3.5 and python2.7.
also do not replace link in tox.ini, as it is reachable.
to address the issues of
- pallets/markupsafe#116
- pypa/setuptools#2017
MarkupSafe is installed by
https://opendev.org/openstack/requirements/raw/branch/stable/pike/upper-constraints.txt
Kefu Chai [Thu, 28 May 2020 15:14:35 +0000 (23:14 +0800)]
qa/tasks/keystone.py: support multiple positional args
it's required when creating endpoint, see
https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/endpoint.html,
where we need to pass <service>, <interface>, and <url>
Kefu Chai [Mon, 25 May 2020 07:52:04 +0000 (15:52 +0800)]
qa/suites/rgw/tempest: bump up keystone to 17.0.0
* also generate a sample conf file following the document at
https://github.com/openstack/keystone/tree/17.0.0.0rc2/etc
* use "projects" instead of "tenants" to match the terminology used by
openstack identify API 3.0.
* test API 3.0 instead of API 2.0, by changing
`rgw_keystone_api_version` from "2" to "3"
* explicitly specify a domain "default" for project to be created,
otherwise a POST request will fail with:
```
{"error":{"code":400,"message":"You have tried to create a resource using the admin token. As this token is not within a domain you must explicitly include a domain for this resource to belong
to.","title":"Bad Request"}}
````
* create "default" domain, and use it, othewise a GET request fails
like:
```
2020-05-28T11:17:28.751 INFO:teuthology.orchestra.run.smithi092.stderr:http://smithi092.front.sepia.ceph.com:35357 "GET /v3/domains/default HTTP/1.1" 404 87
2020-05-28T11:17:28.752 INFO:teuthology.orchestra.run.smithi092.stderr:RESP: [404] Content-Length: 87 Content-Type: application/json Date: Thu, 28 May 2020 11:17:28 GMT Server: WSGIServer/0.2
CPython/3.6.9 Vary: X-Auth-Token x-openstack-request-id: req-bc33796f-2bc3-411c-a7fb-1208918e0dbd
2020-05-28T11:17:28.752 INFO:teuthology.orchestra.run.smithi092.stderr:RESP BODY: {"error":{"code":404,"message":"Could not find domain: default.","title":"Not Found"}}
```
* add user to "default" domain when creating it.
* use "type" as the positional argument, per
https://docs.openstack.org/keystone/pike/admin/cli-keystone-manage-services.html
otherwise we will have failures like:
```
2020-05-28T13:38:24.867 INFO:teuthology.orchestra.run.smithi198.stderr:openstack service create: error: unrecognized arguments: --type keystone
```
* update `create_endpoint()` to use the V3 API,
see
https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/endpoint.html
`s` comes from `rgwadmin()`, which passes `StringIO` as stdout, so the
the output should an instance of `str` in both Python2 and Python3.
hence there is no need to decode it using UTF-8 codecs again.
qa/tasks/util/rgw: use StringIO for capturing strings
this change should address the failure of
```
2020-04-06T03:07:59.152 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/teuthology/contextutil.py", line 32, in nested
vars.append(enter())
File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw.py", line 266, in configure_compression
rgwadmin(ctx, client, cmd=['user', 'list'], check_status=True)
File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/util/rgw.py", line 43, in rgwadmin
j = json.loads(out)
File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
```
Kyr Shatskyy [Sat, 15 Feb 2020 10:12:40 +0000 (11:12 +0100)]
qa/tasks/s3tests: py3 compat
- use string.ascii_uppercase instead string.uppercase
- use six.ensure_str for bytes when required
- use six.ensure_binary if needed
- get rid of dict.itervalues in favor of dict.values
- get rid of cStringIO.StringIO in favor io.BytesIO
in boto, it tries to figure out the MIME type of a file by its name, if
the file-like objects has an attribute of "name". in Python2, the "name"
is always "<fdopen>", fortunately. while in Python3, `TemporaryFile` also
have a "name" which is its fd, and it is an integer now. so we have following
error when sending a `TemporaryFile` using
`upload_part_from_file()`:
```
2020-04-08T02:25:34.660 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw_multi/tests_ps.py", line 2567, in test_ps_creation_triggers
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: uploader.upload_part_from_file(fp, 1)
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/multipart.py", line 260, in upload_part_from_file
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: query_args=query_args, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 750, in send_file
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 920, in
_send_file_internal
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: self.content_type = mimetypes.guess_type(self.path)[0]
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 289, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: return _db.guess_type(url, strict)
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 114, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: scheme, url = urllib.parse.splittype(url)
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/urllib/parse.py", line 881, in splittype
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: match = _typeprog.match(url)
2020-04-08T02:25:34.669 INFO:tasks.rgw_multisite_tests:TypeError: expected string or bytes-like object
```
to address this issue, in this change, a `NamedTemporaryFile` is used
instead of `TemporaryFile`. the former does have a "name" which is a
`str`.
qa/tasks/radosgw_admin.py: coerce key.name and key.acl to str
if `key.name` is not set, boto fills it with md5, in that case, it comes
from `base64.b64encode()`. so we need to make sure it's str before
passing it to shell.
the same applies to `key.get_xml_acl()`, as its return value comes
directly from something like
```
response = self.connection.make_request('GET', self.name, key_name,
query_args=query_args,
headers=headers)
body = response.read()
# ...
return body
```
base64.b64encode() returns bytes in Python3, and we will pass it as a
commandline parameter later on, which will be quoted. so the command
line can be consumed by shell. so before sending it to shell, we should
convert it to string.
qa/tasks/tempest.py: always write str is value of options
in Python2, ConfigParser is almost the same as RawConfigParser, which
allows set non-string values, but in Python3, ConfigParser.set() only
accepts strings as value of option.
since we do not use "cpar" as an internal storage for options, it does
not matter what type of options we set using ConfigParser as long as it
can be consumed by tempest. boolean settings are translated to "true" or
"false". see also
https://docs.openstack.org/tempest/latest/sampleconf.html
qa/tasks/ragweed.py: use str methods of helper from string module
in Python3, some methods offered by `string` module are now member
method of `str` class, and `string.uppercase` is renamed to
`string.ascii_uppercase` in Python2 and Python3. so let's update
accordingly.
xattrs could contain non-utf8 encoded data, and should be captured using
BytesIO. moreover, it will be fed to `ceph-dencoder`, which expects
binary when performing "import".
collect the keys instead of filtering a dict,
to address following failure:
```
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 29, in test_files_throttle
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner: self._test_throttling(self.FILES_THROTTLE)
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 96, in _test_throttling
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner: return self._do_test_throttling(throttle_type)
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 176, in _do_test_throttling
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner: mds_max_purge_ops = int(self.fs.get_config("mds_max_purge_ops", 'mds'))
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/filesystem.py", line 285, in get_config
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner: service_id = random.sample(filter(lambda i: self.mds_daemons[i].running(), self.mds_daemons), 1)[0]
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/virtualenv/lib/python3.5/random.py", line 311, in sample
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner: raise TypeError("Population must be a sequence or set. For dicts, use list(d).")
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:TypeError: Population must be a sequence or set. For dicts, use list(d).
```
Kefu Chai [Tue, 31 Mar 2020 02:16:40 +0000 (10:16 +0800)]
qa/tasks: use next(iter(..)) for accessing first element in a view
in python2, dict.values() and dict.keys() return lists. but in python3,
they return views, which cannot be indexed directly using an integer index.
there are three use cases when we access these views in python3:
1. get the first element
2. get all the elements and then *might* want to access them by index
3. get the first element assuming there is only a single element in
the view
4. iterate thru the view
in the 1st case, we cannot assume the number of elements, so to be
python3 compatible, we should use `next(iter(a_dict))` instead.
in the 2nd case, in this change, the view is materialized using
`list(a_dict)`.
in the 3rd case, we can just continue using the short hand of
```py
(first_element,) = a_dict.keys()
```
to unpack the view. this works in both python2 and python3.
in the 4th case, the existing code works in both python2 and python3, as
both list and view can be iterated using `iter`, and `len` works as
well.
this file is a template for `boto.cfg` which only contains text, so we
can just open it in text mode, and the file-like object will be passed
to run() as stdin, which is able to consume strings.
qa/tasks: pass encoding to Key.get_content_as_string()
we assume that boto.Key.get_content_as_string() returns str instead of
bytes, and compare the return value with a string, so, to ensure that
lhs and rhs are both strings, we need to decode the returned content.
since we always store strings composed with ASCII, it's safe to use
'ascii' to decode them.
test/rgw: pass encoding to Key.get_content_as_string()
we assume that boto.Key.get_content_as_string() returns str instead of
bytes, and compare the return value with a string, so, to ensure that
lhs and rhs are both strings, we need to decode the returned content.
since we always store strings composed with ASCII, it's safe to use
'ascii' to decode them.
test/rgw/rgw_multi/tests.py: decode retval of bucket.get_policy()
return value of bucket.get_policy() is a bytes, so we need to decode it
before comparing it with a string, since the content of policy is ASCII,
it'd be suffice to decode it with 'ascii'.
projects / ceph.git / log