Zack Cerza [Thu, 14 May 2020 21:55:27 +0000 (15:55 -0600)]
provision.fog: Add optional sentinel file
Similar to how provision.cloud.openstack works, wait for a particular
file to be created before considering provisioning to be finished.
Unlike the cloud.openstack module, teuthology doesn't control the entire
process, so to avoid breaking any labs that aren't using a sentinel
file, make the feature optional and default to being inoperative.
Shraddha Agrawal [Wed, 13 May 2020 22:20:33 +0000 (03:50 +0530)]
docs: acknowledge py2 and py3 where requirements.txt is mentioned
This commit removes mention of deleted file requirements.txt and
acknowledges the two separate requirements files, requirements2.txt
and requirements3.txt, for python2 and python3 respectively, in the
following files:
Kyr Shatskyy [Sat, 23 Nov 2019 19:34:54 +0000 (20:34 +0100)]
suite: add --filter-all parameter
Add --filter-all parameter to 'teuthology-suite' to allow
precise test case selection. The --filter-in makes use of filter list
with logical 'or' operation to include test cases.
The --filter-out takes filter list and excludes any test case which
match any from the filter list.
This patch provides with 'and' logical operation for a given filter list.
Kefu Chai [Thu, 7 May 2020 04:54:31 +0000 (12:54 +0800)]
teuthology/orchestra/remote.py: write remote file without sudo
the remote file is created using "ubuntu" user. in ubuntu xenial,
superuser is able to write to that file using tar, it seems that "tar"
starts another process when writing to the dest file specified by "-f",
and that process does not have the privilege for writing that file.
so when we are trying to archive a directory on ubuntu/focal test node,
we have following error:
```
tar (child): /tmp/tmp.vkl0kAtc06: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
tar: /tmp/tmp.vkl0kAtc06: Cannot write: Broken pipe
tar: Child returned status 2
tar: Error is not recoverable: exiting now
```
and this is reproduciable with tar 1.30:
```
$ touch /tmp/helloworld
$ mkdir /tmp/foobar
$ sudo tar czf /tmp/helloworld -C /tmp/foobar -- .
tar (child): /tmp/helloworld: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
tar: /tmp/helloworld: Cannot write: Broken pipe
tar: Child returned status 2
tar: Error is not recoverable: exiting now
```
but we can workaround this by writing the temp file using "ubuntu" user, like
```
sudo tar czf - -C /tmp/foobar -- . > /tmp/helloworld
```
Thomas Bechtold [Mon, 20 Apr 2020 13:10:39 +0000 (15:10 +0200)]
docs: Link commands to generated help
This is useful to quickly show the available CLI help for the
different commands.
For that, auto generate labels based on the headline via a Sphinx
extension.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
Thomas Bechtold [Sun, 19 Apr 2020 06:25:49 +0000 (08:25 +0200)]
requirements: Bump cffi version to 1.14.0
python 3.8 needs a newer cffi version. So bump it. Otherwise, calling
"./bootstrap" with python3.8 will fail.
This is also done for the python2 requirements to be in sync.
Fixes: https://tracker.ceph.com/issues/45132 Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
teuthology: pass integer as "tries" to safe_while()
in Python3, `a / b` could return a float, and `safe_while()` uses
`itertools.islice()` under the hood, where `stop` should be None or an
integer. so let's use `a // b` instead.
```
2020-04-10T10:35:13.508 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
File "/home/worker/src/teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/worker/src/teuthology_master/teuthology/task/internal/__init__.py", line 363, in archive
fetch_binaries_for_coredumps(path, rem)
File "/home/worker/src/teuthology_master/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps
dump_program))
File "/home/worker/src/teuthology_master/teuthology/orchestra/remote.py", line 325, in _sftp_get_file
sftp.get(remote_path, local_path)
File "/home/worker/src/teuthology_master/virtualenv/lib/python2.7/site-packages/paramiko/sftp_client.py", line 801, in get
with open(localpath, "wb") as fl:
IOError: [Errno 13] Permission denied: '/usr/bin/python3'
```
The issue happens because of os.path.join ignores all prior arguments
if the next has '/' prefix (i.e. absolute), for example:
```
>>> os.path.join('a/b/', '/x/y')
'/x/y'
```
when a user expect to get 'a/b/x/y', she gets '/x/y'.
to be py3 compatible. this addresses the failure of
```
2020-04-05T12:35:02.036 INFO:teuthology.run_tasks:Running task ssh_keys...
2020-04-05T12:35:02.050 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 64, in run_one_task
task = get_task(taskname)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 28, in get_task
module = _import('tasks', module_name, task_name, fail_on_import_error=True)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_py3/teuthology/run_tasks.py", line 52, in _import
0,
ImportError: No module named 'tasks.ssh_keys'
```
teuthology: use next(iter(..)) for accessing first element in a view
in python2, dict.values() and dict.keys() return lists. but in python3,
they return views, which cannot be indexed directly using an integer
index.
there are three use cases when we access these views in python3:
1. get the first element
2. get all the elements and then *might* want to access them by index
3. get the first element assuming there is only a single element in
the view
4. iterate thru the view
in the 1st case, we cannot assume the number of elements, so to be
python3 compatible, we should use `next(iter(a_dict))` instead.
in the 2nd case, in this change, the view is materialized using
`list(a_dict)`.
in the 3rd case, we can just continue using the short hand of
```py
(first_element,) = a_dict.keys()
```
to unpack the view. this works in both python2 and python3.
in the 4th case, the existing code works in both python2 and python3, as
both list and view can be iterated using `iter`, and `len` works as
well.
Xiubo Li [Mon, 30 Mar 2020 06:36:20 +0000 (02:36 -0400)]
requirements,setup: add ipy module for cephfs mount.py
The IPy module is a tool for handling of IPv4 and IPv6 addresses and
networks, the cephfs mount.py will use it to manage the IP addresses
to isolate the netns for each mount, which could let us get ride of
using ipmi command to hard shutdown the client node for some test cases.
Kefu Chai [Wed, 25 Mar 2020 10:26:24 +0000 (18:26 +0800)]
orchestra/remote.py: use ensure_online() helper in run()
for better readability, and to have better errors when we fail to
connect to remote. before this change, we could have following
backtrace:
2020-03-25T09:00:47.022 INFO:teuthology.lock.ops:Checking smithi088.front.sepia.ceph.com
2020-03-25T09:00:47.023 INFO:teuthology.lock.ops:New key found. Updating...
2020-03-25T09:00:47.044 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: set os type and version on server
2020-03-25T09:00:47.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-03-25T09:00:47.045 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi088.front.sepia.ceph.com', 'timeout': 60}
2020-03-25T09:00:47.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88
2020-03-25T09:00:47.046 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/lock_machines.py", line 78, in lock_machines
os_version, arch)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/ops.py", line 147, in lock_many
update_nodes(reimaged)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/lock/ops.py", line 33, in update_nodes
inventory_info = remote.inventory_info
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 475, in inventory_info
node['arch'] = self.arch
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 459, in arch
self._arch = self.sh('uname -m').strip()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 242, in sh
proc=self.run(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 412, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
projects / teuthology.git / log