Tiago Melo [Sat, 9 May 2020 02:28:39 +0000 (02:28 +0000)]
mgr/dashboard: Fix random E2E error in mgr-modules
This test failed at random times when it tried to find the new value of pool_ids
in the balancer module.
This happened because the value of pool_ids is automatically reverted by ceph,
so in some situations when we tried to read the new value,
it was already reverted and failed.
Enhanced the tests to be able to use any text input, not only the ones with
empty default values.
Alfonso Martínez [Fri, 22 May 2020 11:36:10 +0000 (13:36 +0200)]
mgr/dashboard: grafana panels for rgw multisite sync performance
* RGW sync perf. counters are now exposed through grafana panels.
* Sync Performance tab is only shown if rgw realm is detected.
* Prometheus module: added metrics suitable for prometheus consumption (from existing ones, not replacing for backward compatibility).
Fixes: https://tracker.ceph.com/issues/45310 Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit cf4ff7d2f03bc285a3fae3f27577333f11dab58a)
Conflicts:
src/pybind/mgr/dashboard/run-frontend-e2e-tests.sh
There was an extra square bracket in octopus that caused the conflict.
This was manually fixed, since the commit that removed it will not be
backported.
mgr/volumes: Create subvolume with isolated rados namespace
1. Add --namespace-isolated option to 'subvolume create' command
to create subvolume in a separate RADOS namespace
2. Add "pool_namespace" field to 'subvolume info' command
which displays the rados namespace if set else empty string
"ceph fs subvolume snapshot info <vol_name> <sub_name> <snap_name> [<group_name>]"
The output is in json format with following fields
created_at: time of creation of snapshot in the format "YYYY-MM-DD HH:MM:SS:ffffff"
data_pool: data pool the snapshot belongs to
has_pending_clones: "yes" if snapshot clone is in progress otherwise "no"
protected: "yes" if snapshot is protected otherwise "no"
size: snapshot size in bytes
Nizamudeen A [Wed, 22 Apr 2020 11:23:41 +0000 (16:53 +0530)]
mgr/dashboard: Asynchronous unique username validation for User Component
Implements an asynchronous validation for the username field in the Create User form which immediately display an error message if the username already exists.
Jeff Layton [Wed, 3 Jun 2020 15:29:07 +0000 (11:29 -0400)]
fuse: update to newer FUSE_USE_VERSION
The build was failing for me against fuse-devel v3.9.1. The prototype
for fuse_ll_ioctl was wrong, as it was expecting the old-style one with
signed int args.
In newer libfuse versions, the prototype varies based on
FUSE_USE_VERSION. Update to a newer FUSE_USE_VERSION value to ensure
that we use the newer ioctl prototype. This also means that we need to
handle a new prototype for fuse_session_loop_mt as well.
While we're in here, move the definition of FUSE_USE_VERSION to
ceph_fuse.h so we have the definition in one place. This does mean we
need to reorganize the includes in a few places.
Fixes: https://tracker.ceph.com/issues/45866 Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit cfbc0fb48a9880dabe8f332e9bf67d81867ae198)
Xiubo Li [Mon, 1 Jun 2020 01:57:24 +0000 (21:57 -0400)]
qa/tasks/cephfs/test_scrub.py: use umount_wait to avoid ceph-fuse stuck
If the ceph-fuse client need to flush the caps and does sync wait,
the umount() will just return successfully, then the netns container
will be destroyed and the network will not be reachable, but the
ceph-fuse daemon is still stucked and waiting for the flush caps ack.
This will cause the ceph-fuse daemon get stuck forever and if the
mds daemons get restarted, it will try to reconnect the clients,
but the stucked ceph-fuse daemnon won't reply to it, because it is
not reachable any more.
Commit 9f6c764f10f replaces remote.run calls by remote.sh without
updating the definition of vstart_runner.LocalRemote.sh which breaks the
cephfs tests when executed locally.
Fixes: https://tracker.ceph.com/issues/44579 Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 2cc0ee709c36eabe03311a00b72295da468bccf4)
Kefu Chai [Sun, 31 May 2020 00:47:34 +0000 (08:47 +0800)]
qa/suites/rgw/tempest: update unsupported tests of tempest
after rerunning tempest with lastest radosgw, remove the supported
tests from the blacklist, and add the ones which are not supported
yet. now we can pass 123 tests in total.
also enable discoverity for better testing coverage, since it's
supported now.
Kefu Chai [Sun, 31 May 2020 00:38:00 +0000 (08:38 +0800)]
qa/tasks/keystone: use "keystone-manage bootstrap"
* qa/tasks/keystone.py:
instead of prefilling keystone manually, use "keystone-manage bootstrap"
instead. it helps to setup the admin user, a "Default" domain with
"default" id, and wire them up with the expected role and a "admin" project,
etc. as id of the admin domain is known to be "default", we can just use it
in our tests without querying openstack for the id of "Default"
domain. this is very handy.
* qa/suites/rgw/tempest/tasks/rgw_tempest.yaml:
use "Default" for domain name. as "Default" is the name of the domain
created by bootstrap, while "default" is its id.
* qa/suites/rgw/crypt/2-kms/barbican.yaml:
remove settings to bootstrap keystone
Kefu Chai [Thu, 28 May 2020 16:51:39 +0000 (00:51 +0800)]
qa/suites/rgw/tempest: use the latest tempest supporting py3.5
in case we need to use ubuntu xenial for testing, xenial only had python
3.5 packaged. and tempest 23.0 was the last version which supports
python3.5 and python2.7.
also do not replace link in tox.ini, as it is reachable.
to address the issues of
- pallets/markupsafe#116
- pypa/setuptools#2017
MarkupSafe is installed by
https://opendev.org/openstack/requirements/raw/branch/stable/pike/upper-constraints.txt
Kefu Chai [Thu, 28 May 2020 15:14:35 +0000 (23:14 +0800)]
qa/tasks/keystone.py: support multiple positional args
it's required when creating endpoint, see
https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/endpoint.html,
where we need to pass <service>, <interface>, and <url>
Kefu Chai [Mon, 25 May 2020 07:52:04 +0000 (15:52 +0800)]
qa/suites/rgw/tempest: bump up keystone to 17.0.0
* also generate a sample conf file following the document at
https://github.com/openstack/keystone/tree/17.0.0.0rc2/etc
* use "projects" instead of "tenants" to match the terminology used by
openstack identify API 3.0.
* test API 3.0 instead of API 2.0, by changing
`rgw_keystone_api_version` from "2" to "3"
* explicitly specify a domain "default" for project to be created,
otherwise a POST request will fail with:
```
{"error":{"code":400,"message":"You have tried to create a resource using the admin token. As this token is not within a domain you must explicitly include a domain for this resource to belong
to.","title":"Bad Request"}}
````
* create "default" domain, and use it, othewise a GET request fails
like:
```
2020-05-28T11:17:28.751 INFO:teuthology.orchestra.run.smithi092.stderr:http://smithi092.front.sepia.ceph.com:35357 "GET /v3/domains/default HTTP/1.1" 404 87
2020-05-28T11:17:28.752 INFO:teuthology.orchestra.run.smithi092.stderr:RESP: [404] Content-Length: 87 Content-Type: application/json Date: Thu, 28 May 2020 11:17:28 GMT Server: WSGIServer/0.2
CPython/3.6.9 Vary: X-Auth-Token x-openstack-request-id: req-bc33796f-2bc3-411c-a7fb-1208918e0dbd
2020-05-28T11:17:28.752 INFO:teuthology.orchestra.run.smithi092.stderr:RESP BODY: {"error":{"code":404,"message":"Could not find domain: default.","title":"Not Found"}}
```
* add user to "default" domain when creating it.
* use "type" as the positional argument, per
https://docs.openstack.org/keystone/pike/admin/cli-keystone-manage-services.html
otherwise we will have failures like:
```
2020-05-28T13:38:24.867 INFO:teuthology.orchestra.run.smithi198.stderr:openstack service create: error: unrecognized arguments: --type keystone
```
* update `create_endpoint()` to use the V3 API,
see
https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/endpoint.html
`s` comes from `rgwadmin()`, which passes `StringIO` as stdout, so the
the output should an instance of `str` in both Python2 and Python3.
hence there is no need to decode it using UTF-8 codecs again.
qa/tasks/util/rgw: use StringIO for capturing strings
this change should address the failure of
```
2020-04-06T03:07:59.152 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/teuthology/contextutil.py", line 32, in nested
vars.append(enter())
File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw.py", line 266, in configure_compression
rgwadmin(ctx, client, cmd=['user', 'list'], check_status=True)
File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/util/rgw.py", line 43, in rgwadmin
j = json.loads(out)
File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
```
Kyr Shatskyy [Sat, 15 Feb 2020 10:12:40 +0000 (11:12 +0100)]
qa/tasks/s3tests: py3 compat
- use string.ascii_uppercase instead string.uppercase
- use six.ensure_str for bytes when required
- use six.ensure_binary if needed
- get rid of dict.itervalues in favor of dict.values
- get rid of cStringIO.StringIO in favor io.BytesIO
in boto, it tries to figure out the MIME type of a file by its name, if
the file-like objects has an attribute of "name". in Python2, the "name"
is always "<fdopen>", fortunately. while in Python3, `TemporaryFile` also
have a "name" which is its fd, and it is an integer now. so we have following
error when sending a `TemporaryFile` using
`upload_part_from_file()`:
```
2020-04-08T02:25:34.660 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw_multi/tests_ps.py", line 2567, in test_ps_creation_triggers
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: uploader.upload_part_from_file(fp, 1)
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/multipart.py", line 260, in upload_part_from_file
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: query_args=query_args, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 750, in send_file
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 920, in
_send_file_internal
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: self.content_type = mimetypes.guess_type(self.path)[0]
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 289, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: return _db.guess_type(url, strict)
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 114, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: scheme, url = urllib.parse.splittype(url)
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/urllib/parse.py", line 881, in splittype
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: match = _typeprog.match(url)
2020-04-08T02:25:34.669 INFO:tasks.rgw_multisite_tests:TypeError: expected string or bytes-like object
```
to address this issue, in this change, a `NamedTemporaryFile` is used
instead of `TemporaryFile`. the former does have a "name" which is a
`str`.
qa/tasks/radosgw_admin.py: coerce key.name and key.acl to str
if `key.name` is not set, boto fills it with md5, in that case, it comes
from `base64.b64encode()`. so we need to make sure it's str before
passing it to shell.
the same applies to `key.get_xml_acl()`, as its return value comes
directly from something like
```
response = self.connection.make_request('GET', self.name, key_name,
query_args=query_args,
headers=headers)
body = response.read()
# ...
return body
```
base64.b64encode() returns bytes in Python3, and we will pass it as a
commandline parameter later on, which will be quoted. so the command
line can be consumed by shell. so before sending it to shell, we should
convert it to string.
qa/tasks/tempest.py: always write str is value of options
in Python2, ConfigParser is almost the same as RawConfigParser, which
allows set non-string values, but in Python3, ConfigParser.set() only
accepts strings as value of option.
since we do not use "cpar" as an internal storage for options, it does
not matter what type of options we set using ConfigParser as long as it
can be consumed by tempest. boolean settings are translated to "true" or
"false". see also
https://docs.openstack.org/tempest/latest/sampleconf.html
projects / ceph.git / log