David Galloway [Wed, 27 May 2020 21:42:02 +0000 (17:42 -0400)]
console.py: Wait longer for IPMI timeouts
The smithi have really crappy/flaky BMCs. I occasionally see `'wait for power on' reached maximum tries (5) after waiting for 20.0 seconds` but when I go to check the BMC, it just needed some more time. So let's give it more time.
Signed-off-by: David Galloway <dgallowa@redhat.com>
Kefu Chai [Wed, 27 May 2020 12:10:44 +0000 (20:10 +0800)]
bootstrap: s/python-pip/python3-pip/ for python3 on debian derivatives
python-pip was removed in debian/bullseye and ubuntu/focal, so let's
replace it with python3-pip. and more importantly, for python3 support
we should install python3-pip instead.
Kyr Shatskyy [Fri, 15 May 2020 20:48:47 +0000 (22:48 +0200)]
suite/build_matrix: get rid of '.yaml' in descriptions
Test descriptions have structured representation
fragments the test config is composed from.
It's worth noticing that no other files except yaml
are used for description composition.
Although having the .yaml extentions in the description
looks redundant and has no use in the code.
This change removes many '.yaml' sufixes from
descriptions which make it more compact and easy
to read.
Zack Cerza [Thu, 14 May 2020 21:55:27 +0000 (15:55 -0600)]
provision.fog: Add optional sentinel file
Similar to how provision.cloud.openstack works, wait for a particular
file to be created before considering provisioning to be finished.
Unlike the cloud.openstack module, teuthology doesn't control the entire
process, so to avoid breaking any labs that aren't using a sentinel
file, make the feature optional and default to being inoperative.
Shraddha Agrawal [Wed, 13 May 2020 22:20:33 +0000 (03:50 +0530)]
docs: acknowledge py2 and py3 where requirements.txt is mentioned
This commit removes mention of deleted file requirements.txt and
acknowledges the two separate requirements files, requirements2.txt
and requirements3.txt, for python2 and python3 respectively, in the
following files:
Kyr Shatskyy [Sat, 23 Nov 2019 19:34:54 +0000 (20:34 +0100)]
suite: add --filter-all parameter
Add --filter-all parameter to 'teuthology-suite' to allow
precise test case selection. The --filter-in makes use of filter list
with logical 'or' operation to include test cases.
The --filter-out takes filter list and excludes any test case which
match any from the filter list.
This patch provides with 'and' logical operation for a given filter list.
Kefu Chai [Thu, 7 May 2020 04:54:31 +0000 (12:54 +0800)]
teuthology/orchestra/remote.py: write remote file without sudo
the remote file is created using "ubuntu" user. in ubuntu xenial,
superuser is able to write to that file using tar, it seems that "tar"
starts another process when writing to the dest file specified by "-f",
and that process does not have the privilege for writing that file.
so when we are trying to archive a directory on ubuntu/focal test node,
we have following error:
```
tar (child): /tmp/tmp.vkl0kAtc06: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
tar: /tmp/tmp.vkl0kAtc06: Cannot write: Broken pipe
tar: Child returned status 2
tar: Error is not recoverable: exiting now
```
and this is reproduciable with tar 1.30:
```
$ touch /tmp/helloworld
$ mkdir /tmp/foobar
$ sudo tar czf /tmp/helloworld -C /tmp/foobar -- .
tar (child): /tmp/helloworld: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
tar: /tmp/helloworld: Cannot write: Broken pipe
tar: Child returned status 2
tar: Error is not recoverable: exiting now
```
but we can workaround this by writing the temp file using "ubuntu" user, like
```
sudo tar czf - -C /tmp/foobar -- . > /tmp/helloworld
```
Thomas Bechtold [Mon, 20 Apr 2020 13:10:39 +0000 (15:10 +0200)]
docs: Link commands to generated help
This is useful to quickly show the available CLI help for the
different commands.
For that, auto generate labels based on the headline via a Sphinx
extension.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
Thomas Bechtold [Sun, 19 Apr 2020 06:25:49 +0000 (08:25 +0200)]
requirements: Bump cffi version to 1.14.0
python 3.8 needs a newer cffi version. So bump it. Otherwise, calling
"./bootstrap" with python3.8 will fail.
This is also done for the python2 requirements to be in sync.
Fixes: https://tracker.ceph.com/issues/45132 Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
teuthology: pass integer as "tries" to safe_while()
in Python3, `a / b` could return a float, and `safe_while()` uses
`itertools.islice()` under the hood, where `stop` should be None or an
integer. so let's use `a // b` instead.
```
2020-04-10T10:35:13.508 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
File "/home/worker/src/teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/worker/src/teuthology_master/teuthology/task/internal/__init__.py", line 363, in archive
fetch_binaries_for_coredumps(path, rem)
File "/home/worker/src/teuthology_master/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps
dump_program))
File "/home/worker/src/teuthology_master/teuthology/orchestra/remote.py", line 325, in _sftp_get_file
sftp.get(remote_path, local_path)
File "/home/worker/src/teuthology_master/virtualenv/lib/python2.7/site-packages/paramiko/sftp_client.py", line 801, in get
with open(localpath, "wb") as fl:
IOError: [Errno 13] Permission denied: '/usr/bin/python3'
```
The issue happens because of os.path.join ignores all prior arguments
if the next has '/' prefix (i.e. absolute), for example:
```
>>> os.path.join('a/b/', '/x/y')
'/x/y'
```
when a user expect to get 'a/b/x/y', she gets '/x/y'.
to be py3 compatible. this addresses the failure of
```
2020-04-05T12:35:02.036 INFO:teuthology.run_tasks:Running task ssh_keys...
2020-04-05T12:35:02.050 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 64, in run_one_task
task = get_task(taskname)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 28, in get_task
module = _import('tasks', module_name, task_name, fail_on_import_error=True)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 52, in _import
0,
ImportError: No module named 'tasks.ssh_keys'
```
teuthology: use next(iter(..)) for accessing first element in a view
in python2, dict.values() and dict.keys() return lists. but in python3,
they return views, which cannot be indexed directly using an integer
index.
there are three use cases when we access these views in python3:
1. get the first element
2. get all the elements and then *might* want to access them by index
3. get the first element assuming there is only a single element in
the view
4. iterate thru the view
in the 1st case, we cannot assume the number of elements, so to be
python3 compatible, we should use `next(iter(a_dict))` instead.
in the 2nd case, in this change, the view is materialized using
`list(a_dict)`.
in the 3rd case, we can just continue using the short hand of
```py
(first_element,) = a_dict.keys()
```
to unpack the view. this works in both python2 and python3.
in the 4th case, the existing code works in both python2 and python3, as
both list and view can be iterated using `iter`, and `len` works as
well.
Xiubo Li [Mon, 30 Mar 2020 06:36:20 +0000 (02:36 -0400)]
requirements,setup: add ipy module for cephfs mount.py
The IPy module is a tool for handling of IPv4 and IPv6 addresses and
networks, the cephfs mount.py will use it to manage the IP addresses
to isolate the netns for each mount, which could let us get ride of
using ipmi command to hard shutdown the client node for some test cases.
projects / teuthology.git / log