Introduced in 4d42b4c5a0ed ("common/TextTable: default to 2 spaces
separating columns") and 41f003518a07 ("common/TextTable: only pad
between columns").
Jan Fajerski [Wed, 8 Apr 2020 09:55:57 +0000 (11:55 +0200)]
ceph-volume/batch: return success when all devices are filtered
batch should only return an error if some (but not all) devices are
filtered. When only some devices are filtered the resulting osd layout
could look very different from what a user expects. If all devies are
filtered just return success.
Fixes: https://tracker.ceph.com/issues/44994 Signed-off-by: Jan Fajerski <jfajerski@suse.com>
in tasks/module_selftest.yaml, `TestModuleSelftest.test_telegraf()` is
called. but we fail to prepare a unix domain socket to which the telegraf
module can send stats. and telegraf module does not catch
FileNotFoundError exception, so the exception is populated to ceph-mgr
and is found by the test, hence the test is marked a failure whenever
telegraf is tested.
in this change,
* catch this exception, so it won't be caught by ceph-mgr
* whitelist the error message, so the test can pass
qa/tasks/openssl_keys.py: sort cert configs before creating certs
we cannot rely on the order in which items are arranged in a dict, the
order varies from version to another. in Python2, it happens to work,
and we can always have the self-signed cert added first. but in Python3,
it does not. and an exception is thrown
```
teuthology.exceptions.ConfigError: ssl: ca root not found for
certificate rgw.client.0
```
in this change, before creating certs, the settings are reordered so
that the self-signed ones are created first.
in this change, instead of trying to be smart and to walk all the way
up to the root directory or $top_srcdir, we just check $top_srcdir/.git
directly, as we just know it's there or it does not exist at all.
in boto, it tries to figure out the MIME type of a file by its name, if
the file-like objects has an attribute of "name". in Python2, the "name"
is always "<fdopen>", fortunately. while in Python3, `TemporaryFile` also
have a "name" which is its fd, and it is an integer now. so we have following
error when sending a `TemporaryFile` using
`upload_part_from_file()`:
```
2020-04-08T02:25:34.660 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw_multi/tests_ps.py", line 2567, in test_ps_creation_triggers
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: uploader.upload_part_from_file(fp, 1)
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/multipart.py", line 260, in upload_part_from_file
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: query_args=query_args, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 750, in send_file
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 920, in
_send_file_internal
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: self.content_type = mimetypes.guess_type(self.path)[0]
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 289, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: return _db.guess_type(url, strict)
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 114, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: scheme, url = urllib.parse.splittype(url)
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/urllib/parse.py", line 881, in splittype
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: match = _typeprog.match(url)
2020-04-08T02:25:34.669 INFO:tasks.rgw_multisite_tests:TypeError: expected string or bytes-like object
```
to address this issue, in this change, a `NamedTemporaryFile` is used
instead of `TemporaryFile`. the former does have a "name" which is a
`str`.
to address the test failures like
```
2020-04-07T15:44:58.693 INFO:tasks.workunit.client.0.smithi049.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed: ceph pg dump
pgs
2020-04-07T15:44:58.694 INFO:tasks.workunit.client.0.smithi049.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed: pgid
2020-04-07T15:44:58.694 INFO:tasks.workunit.client.0.smithi049.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh: line 498: pgid: command not found
```
as the caller might want to `len(manager.get_osd_status()['raw'])`, and
`len()` does not accept a `filter` object.
also, the filtered osd statuses are printed out using `self.log()`, so
we should materialize the `filter` object before sending it to logging
facility. otherwise we will have something like:
```
2020-04-08T02:58:37.001 INFO:tasks.ceph.ceph_manager.ceph:<filter object at 0x7f5a080e1518>
```
in the logging message.
to address the test failures like
```
2020-04-07T15:44:58.693 INFO:tasks.workunit.client.0.smithi049.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed: ceph pg dump
pgs
2020-04-07T15:44:58.694 INFO:tasks.workunit.client.0.smithi049.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_failed: pgid
2020-04-07T15:44:58.694 INFO:tasks.workunit.client.0.smithi049.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh: line 498: pgid: command not found
```
mgr/dashboard: fix errors related to frontend service subscriptions.
- Created TimerService: for getting a scheduled observable that checks if there are any observers subscribed before sending a request.
Now FeatureTogglesService uses it internally (solving a hidden bug in it).
- SummaryService & RbdMirroringService: instead of having window.setInterval/setTimeout tasks launched in constructor, now they have a method for polling (and subscribing to it). These methods are called in the appropriate component so we unsubscribe properly.
- Fixed: some subscriptions were not being unsubscribed properly.
- RbdFormComponent: little refactoring and fixed (also improved) unit tests.
Fixes: https://tracker.ceph.com/issues/44228 Signed-off-by: Alfonso Martínez <almartin@redhat.com>
qa/tasks/radosgw_admin.py: coerce key.name and key.acl to str
if `key.name` is not set, boto fills it with md5, in that case, it comes
from `base64.b64encode()`. so we need to make sure it's str before
passing it to shell.
the same applies to `key.get_xml_acl()`, as its return value comes
directly from something like
```
response = self.connection.make_request('GET', self.name, key_name,
query_args=query_args,
headers=headers)
body = response.read()
# ...
return body
```
test/rgw/rgw_multi/tests.py: decode retval of bucket.get_policy()
return value of bucket.get_policy() is a bytes, so we need to decode it
before comparing it with a string, since the content of policy is ASCII,
it'd be suffice to decode it with 'ascii'.
test/rgw: pass encoding to Key.get_content_as_string()
we assume that boto.Key.get_content_as_string() returns str instead of
bytes, and compare the return value with a string, so, to ensure that
lhs and rhs are both strings, we need to decode the returned content.
since we always store strings composed with ASCII, it's safe to use
'ascii' to decode them.
qa/tasks: pass encoding to Key.get_content_as_string()
we assume that boto.Key.get_content_as_string() returns str instead of
bytes, and compare the return value with a string, so, to ensure that
lhs and rhs are both strings, we need to decode the returned content.
since we always store strings composed with ASCII, it's safe to use
'ascii' to decode them.
this file is a template for `boto.cfg` which only contains text, so we
can just open it in text mode, and the file-like object will be passed
to run() as stdin, which is able to consume strings.
base64.b64encode() returns bytes in Python3, and we will pass it as a
commandline parameter later on, which will be quoted. so the command
line can be consumed by shell. so before sending it to shell, we should
convert it to string.
qa/tasks/tempest.py: always write str is value of options
in Python2, ConfigParser is almost the same as RawConfigParser, which
allows set non-string values, but in Python3, ConfigParser.set() only
accepts strings as value of option.
since we do not use "cpar" as an internal storage for options, it does
not matter what type of options we set using ConfigParser as long as it
can be consumed by tempest. boolean settings are translated to "true" or
"false". see also
https://docs.openstack.org/tempest/latest/sampleconf.html
qa/tasks/ragweed.py: use str methods of helper from string module
in Python3, some methods offered by `string` module are now member
method of `str` class, and `string.uppercase` is renamed to
`string.ascii_uppercase` in Python2 and Python3. so let's update
accordingly.
`s` comes from `rgwadmin()`, which passes `StringIO` as stdout, so the
the output should an instance of `str` in both Python2 and Python3.
hence there is no need to decode it using UTF-8 codecs again.
qa/tasks/util/rgw: use StringIO for capturing strings
this change should address the failure of
```
2020-04-06T03:07:59.152 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/teuthology/contextutil.py", line 32, in nested
vars.append(enter())
File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw.py", line 266, in configure_compression
rgwadmin(ctx, client, cmd=['user', 'list'], check_status=True)
File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/util/rgw.py", line 43, in rgwadmin
j = json.loads(out)
File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
```
xattrs could contain non-utf8 encoded data, and should be captured using
BytesIO. moreover, it will be fed to `ceph-dencoder`, which expects
binary when performing "import".
collect the keys instead of filtering a dict,
to address following failure:
```
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 29, in test_files_throttle
2020-04-05T12:15:36.275 INFO:tasks.cephfs_test_runner: self._test_throttling(self.FILES_THROTTLE)
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 96, in _test_throttling
2020-04-05T12:15:36.276 INFO:tasks.cephfs_test_runner: return self._do_test_throttling(throttle_type)
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/test_strays.py", line 176, in _do_test_throttling
2020-04-05T12:15:36.278 INFO:tasks.cephfs_test_runner: mds_max_purge_ops = int(self.fs.get_config("mds_max_purge_ops", 'mds'))
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/cephfs/filesystem.py", line 285, in get_config
2020-04-05T12:15:36.279 INFO:tasks.cephfs_test_runner: service_id = random.sample(filter(lambda i: self.mds_daemons[i].running(), self.mds_daemons), 1)[0]
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/virtualenv/lib/python3.5/random.py", line 311, in sample
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner: raise TypeError("Population must be a sequence or set. For dicts, use list(d).")
2020-04-05T12:15:36.280 INFO:tasks.cephfs_test_runner:TypeError: Population must be a sequence or set. For dicts, use list(d).
```
pybind/mgr/dashboard/.pylintrc: drop xrange from the whitelist
since dashboard is now xrange-free, there is no need to have
xrange-builtin in whitelist, moreover, in python3 the error message
cannot be emitted at seeing xrange.
Kefu Chai [Tue, 31 Mar 2020 02:16:40 +0000 (10:16 +0800)]
qa/tasks: use next(iter(..)) for accessing first element in a view
in python2, dict.values() and dict.keys() return lists. but in python3,
they return views, which cannot be indexed directly using an integer index.
there are three use cases when we access these views in python3:
1. get the first element
2. get all the elements and then *might* want to access them by index
3. get the first element assuming there is only a single element in
the view
4. iterate thru the view
in the 1st case, we cannot assume the number of elements, so to be
python3 compatible, we should use `next(iter(a_dict))` instead.
in the 2nd case, in this change, the view is materialized using
`list(a_dict)`.
in the 3rd case, we can just continue using the short hand of
```py
(first_element,) = a_dict.keys()
```
to unpack the view. this works in both python2 and python3.
in the 4th case, the existing code works in both python2 and python3, as
both list and view can be iterated using `iter`, and `len` works as
well.
projects / ceph.git / log