valgrind.supp: move the whitelist file to /qa dir of ceph.git
This change drops valgrind.supp whitelist file from teuthology
repo and alters the install task to acquire it from /qa dir of
ceph-qa-suite repository.
It's expected the commit will have sibiling in ceph.git moving
the proper valgrind.supp there.
The rationale behind the entire change is duplication of these
files between two repos, their inconsistencies and confusion.
Kyr Shatskyy [Mon, 4 Mar 2019 18:18:10 +0000 (19:18 +0100)]
teuthology-openstack: strip out unicode tags for interim test repo yaml
When json file loads from file for --test-repo it is converted
to u'' instead of bare str. Later yaml.dump includes unicode
tags in resulting yaml which makes teuthology-suite fail and
throw an exception.
Kyr Shatskyy [Mon, 15 Oct 2018 20:00:15 +0000 (22:00 +0200)]
Add optional priority for --test-repo
Priority can be set adding optional '!' symbol with a number between
name and url. Internally there will be a 'priority' field added
next to 'name' and 'url' fields to the yaml definition.
Kyr Shatskyy [Thu, 28 Feb 2019 15:37:10 +0000 (16:37 +0100)]
openstack: take defaults from environment
This patch introduces TEUTH_ based environment variables
which can be used to avoid bulky commands while overiding
default values for some of the teuthology arguments.
Removes mandatory of --ceph option for teuthology-suite,
and defaults to 'master'.
Kyr Shatskyy [Wed, 6 Feb 2019 23:48:04 +0000 (00:48 +0100)]
openstack: fix all volumes nuke
Do not allow teuthology-nuke to delete all volumes on openstack.
There is a bug on some openstack clients that --name option
does not work correctly for 'openstack volume list' in contrast
to 'openstack server list' and lists all available volumes
ignoring '--name' option at all.
Kyr Shatskyy [Thu, 3 Jan 2019 07:48:07 +0000 (08:48 +0100)]
orchestra: allow runtime logging for sh
When run.run is used with 'stdout' argument, the output of run command
is read to the given stream and logged with corresponding level only after
completion, which is different from the default behaviour. This 'hides'
useful information when, for example, the command is hung or stuck.
Aso it is usually handy when there are some data can be found
in the log for a prolonged command execution during runtime.
Kefu Chai [Fri, 21 Dec 2018 07:17:09 +0000 (15:17 +0800)]
task/install: populate "downgrade_packages" to nested_config
- populate "downgrade_packages" to `nested_config` in `task()`.
otherwise, the "downgrade_packages" is invisible to the `install()`
function.
- also print logging message before downgrading packages.
- cleanup rpm._downgrade_packages(), no need to format `downgrade_cmd`.
it's already formated.
Kefu Chai [Tue, 11 Dec 2018 12:19:08 +0000 (20:19 +0800)]
task/install: add "downgrade_packages" option for "install" task
in rados/thrash-old-clients, hammer or jewel packages is installed. but
yum does not allow downgrade a package by default, if a newer version is
already installed. in this case, librbd1 and librados2 are installed as
dependencies of qemu-kvm. their version is 1:10.2.5-4.el7 at the time of
writing in CentOS/RHEL 7.5. so if we want to install librbd1 or
librados2 from jewel, yum will simply consider the requirement is
already fulfilled and hence do nothing. if we want to install
ceph-radosgw from jewel, yum will fail, as ceph-radosgw depends on
librados2 and other Ceph packages of the same version from jewel. but
librbd1 and librados2 have been already installed. the error message
looks like:
so we need to downgrade librbd1 and librados2 first.
in this change, "downgrade_package" option is added for "install" task,
so we can specify packages to be downgraded to given version, these
packages won't be installed twice if they are also specified by
"install" task elsewhere to be installed.
Kefu Chai [Tue, 11 Dec 2018 09:31:08 +0000 (17:31 +0800)]
task/install: define "project" before using it
no need to define `project` at the beginning of install(), we can
just use `config.get('project', 'ceph')`. better readability this way,
as we will reference `ceph` in the same expression.
Kefu Chai [Mon, 10 Dec 2018 14:50:38 +0000 (22:50 +0800)]
task/install: keep the order of pkgs to be installed
before this change, the package list to be installed are ordered by
the hash function used by set().
after this change, the order of packages are not changed. so we can
ensure the order of package installation is the identical to that
of package list in yaml.
Nathan Cutler [Thu, 22 Nov 2018 11:59:48 +0000 (12:59 +0100)]
run_tasks.py: allow _import to raise the right ImportError
It turns out it's possible for a file qa/tasks/foo.py to exist,
yet importing it still raises an ImportError because it references a
non-existent symbol.
In this case, teuthology was clobbering the real ImportError with its
own bogus text.
Allow users to skip validation of SHA1s passed to -S when submitting
jobs via teuthology-suite. This is useful when testing against
non-GitHub repos, since teuthology can't validate refs for a plain git
repository.
Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
Sage Weil [Fri, 9 Nov 2018 14:53:03 +0000 (08:53 -0600)]
valgind: ignore all leaks relating to CPython code
Yes, this is a big hammer, and we are ignoring a lot. However, it is a
HUGE step forward to what we do now, which is not check for ceph-mgr
leaks at all.
By adding this suppress I found and fixed 3 separate ceph-mgr leaks. This
will let us prevent others (in non-Py code) from being introduced.
Kyr Shatskyy [Tue, 30 Oct 2018 13:17:05 +0000 (14:17 +0100)]
orchestra: add remote.sh commands analogous to misc.sh
Adds a remote.sh similar to misc.sh, in fact a shortcut for remote.run,
but return output instead of proc
Example:
my_name = Remote('127.0.0.1').sh('whoami')
Adds a remote.sh_file run a script as file on a remote with or without sudo
Example 1: Run python script
Remote('127.0.0.1').sh_file("#!/usr/bin/env python3\n"
"import sys\n"
"print(sys.version_info)")
Example 2: Run script as root
Remote('user@host.domain').sh_file("whoami", sudo=True,
label="who-am-i-for-the-real")
Example 3: Run script as other user
Remote('user@host.domain').sh_file("whoami", sudo='nobody', )
Kyrylo Shatskyy [Sun, 28 Oct 2018 17:56:14 +0000 (18:56 +0100)]
Fix ipv4 and ipv6 address logging for Remote.run
The Remote class does not respect ip addresses
when it comes to define shortnames. As a result,
the hostname is not shown correctly in the log.
For ipv4 it only shows first number of the octet.
For ipv6 it even does not allow to proceed,
and raises exception in orchestra.run.
Kefu Chai [Fri, 26 Oct 2018 06:18:34 +0000 (14:18 +0800)]
orchestra.run: log the ssh command without prefix
run() also supports single string, but if we pass a longstring literal
which contains "\n", it renders log difficult to read.
in this change, multi-line command is logged in multiple lines,
and print the "prefix" in the first line, then print the command in
following lines without "prefix".
Kyr Shatskyy [Thu, 26 Oct 2017 16:15:41 +0000 (18:15 +0200)]
openstack: add --test-repo CLI option
Add custom repos before installing rpm packages on test nodes.
Repository can be specified as a NAME:URL pair. Several repos
can be provided by specifying the option multiple times.
For example,
--test-repo foo:http://example.com/repo/foo \
--test-repo bar:http://example.com/repo/bar
gives two test package repositories named "foo" and "bar".
Nathan Cutler [Wed, 30 Aug 2017 10:18:30 +0000 (12:18 +0200)]
install: rpm: only one option per variable
Although the variables are entitled, e.g., "pkg_mng_opts" they
really can only contain at most one option. Here's what happens
when they contain more than one:
Nathan Cutler [Wed, 30 Aug 2017 09:55:15 +0000 (11:55 +0200)]
Override failing package signature checks
The RPMs built by teuthology's buildpackages task are not signed and
after a recent zypper update the install task started to fail with
File 'repomd.xml' from repository 'ceph-rpm-under-test' is unsigned, continue? [yes/no] (no): no
Error building the cache: [ceph-rpm-under-test|http://149.202.175.91/ceph-rpm-sle12-x86_64-basic/sha1/3804e807353c9d125753b1cf4f6405f79db83d4e/x86_64] Valid metadata not found at specified URL
Nathan Cutler [Thu, 11 May 2017 20:18:55 +0000 (22:18 +0200)]
packaging: add ceph-rpm-under-test zypper repo with high priority
Otherwise the ceph packages from OBS are preferred because RPM evaluates, e.g.,
12.0.2+git.1493341348.9148e53 as a higher version number than
12.0.2-276.gf27d4b00ed.
Nathan Cutler [Sat, 11 Feb 2017 08:21:16 +0000 (09:21 +0100)]
packaging: call the zypper repo ceph-rpm-under-test
The zypper repo must have a name/alias and "ceph-rpm-under-test" seems better
than just "ceph-rpm"; it's a repo containing the ceph RPMs that are being
tested.
Nathan Cutler [Fri, 10 Feb 2017 14:38:06 +0000 (15:38 +0100)]
setup: do not set ceph_qa_suite_git_url in ~/.teuthology.yaml
When this value is set, it is necessary to explicitly give --suite-repo and
--suite-branch. We would rather have the defaults for these come from
--ceph-repo and --ceph.
Nathan Cutler [Wed, 15 Mar 2017 14:12:37 +0000 (15:12 +0100)]
openstack: upload absolute paths to remote teuthology machine
If any absolute paths are given on the teuthology-openstack command line,
assume they are local YAML files and upload them to the remote teuthology
machine before running teuthology-suite.
If the local YAML file (absolute path) is PATH, the remote
path will be /home/ubuntu/yaml/$PATH. The remote file is clobbered if
it already exists.
projects / teuthology.git / log