nuke: s/yaml.load/yaml.safe_load/

to silence following warning:

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1508 from lxbsz/ulimit

task/install/adjust-ulimits: use a larger numbers for open files

Reviewed-by: Kefu Chai <kchai@redhat.com>

adjust-ulimits: use a larger numbers for open files

For some cephfs test case it will run handreds of threads and with
some mounts in each, we need a larger limit on the number of open
file descriptors.

Fixes: https://tracker.ceph.com/issues/45829
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1507 from tchaikov/wip-docs-virtualenv-py2

docs: create a python3 virtualenv instead a py2 one

Reviewed-by: Jos Collin <jcollin@redhat.com>

docs: create a python3 virtualenv instead a py2 one

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1506 from sunilkumarn417/master

eliminating kernel task duplication

Merge pull request #1505 from ceph/bootstrap-fedora-rhel

bootstrap: python3 packages for fedora and redhat based systems

python3 packages for fedora and redhat based systems

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

eliminating kernel task duplication

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1503 from ceph/wip-valgrind-exits-on-error

teuthology/misc: Valgrind exits on first detected error.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/misc: Valgrind exits on first detected error.

Valgrind v3.14 brought the `--exit-on-first-error` option
which causes termination of the process on first encountered
error. Although it does nothing for memory leaks, it should
help with pin-pointing e.g. `UninitValue` without imposing
the costly `--track-origins=yes`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1502 from ceph/fix-darwin

tox: Avoid system site-packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1490 from kshtsk/wip-auto-t

teuthology-suite: automate -t argument default value

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tox: Avoid system site-packages

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-suite: automate -t argument default value

This change is required to determine which teuthology branch should
be used for scheduling a run for the given ceph branch.

Drop the code using --ceph-branch and when --teuthology-branch is not
supplied, since this mechanism is not used and not supported anymore.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1499 from sebastian-philipp/write-file-unicode

orchestra/run: py2 fix for copy_and_close

Merge pull request #1497 from ceph/drop-mysql-dep

Drop MySQL dependency

Drop results.generate_coverage()

This was just an oversight from #1419

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop MySQL dependency

I think it's been years since anything in the project used the
mysqlclient package, so let's stop making everyone install it and everything it requires.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/run: py2 fix for copy_and_close

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1495 from kshtsk/plgs-logs

provision/pelagos: improve logging

provision/pelagos: improve logging

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1496 from tchaikov/wip-bootstrap-py3

bootstrap: s/python-pip/python3-pip/ for python3 on debian derivatives

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1498 from ceph/wip-waitlonger

console.py: Wait longer for IPMI timeouts

console.py: Wait longer for IPMI timeouts

The smithi have really crappy/flaky BMCs.  I occasionally see `'wait for power on' reached maximum tries (5) after waiting for 20.0 seconds` but when I go to check the BMC, it just needed some more time.  So let's give it more time.

Signed-off-by: David Galloway <dgallowa@redhat.com>

bootstrap: s/python-pip/python3-pip/ for python3 on debian derivatives

python-pip was removed in debian/bullseye and ubuntu/focal, so let's
replace it with python3-pip. and more importantly, for python3 support
we should install python3-pip instead.

Fixes: https://tracker.ceph.com/issues/45384
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1491 from ShraddhaAg/update-docs

docs: add docs.ceph.com link in README.rst

Merge pull request #1486 from kshtsk/wip-sbtd-email

run_tasks: add email notification for sleep-before-teardown

Merge pull request #1492 from toabctl/fix-opensuse-py3-bootstrap

Fix opensuse py3 bootstrap

Merge pull request #1493 from sebastian-philipp/toml

setup.py: add toml for qa/tasks/cephadm.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

setup.py: add toml for qa/tasks/cephadm.py

podman's /etc/containers/registries.conf is written
in toml. In order to modify it, we need to parse the
config.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1482 from kshtsk/wip-desc-without-yaml

suite/build_matrix: get rid of '.yaml' in descriptions

run_tasks: add email notification for sleep-before-teardown

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstap: Drop --setuptools from virtualenv installation

--setuptools is a no-op since ~2013 so drop the switch.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

bootstrap: Support python3 on openSUSE/SLES

Install the correct system packages when using a python3 env. Similar
to what is done in the ubuntu/debian case.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

docs: add docs.ceph.com link in README.rst

This commit adds a hyperlink to Running Integration Tests Using
Teuthology in docs/README.rst under the section Test Suites.

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Merge pull request #1489 from ceph/ignore-package-check

run: exclude internal check_packages and buildpackages_prep for redhat

skip internal check_packages and buildpackages_prep for redhat config

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Merge pull request #1474 from tchaikov/wip-no-clear-exec-py3

teuthology/run_tasks: do not call sys.exc_clear() for clearing exception

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #1473 from kshtsk/wip-schedule-once

suite: schedule first and last in suite jobs once

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1483 from kshtsk/wip-rename-describe-tests

rename teuthology-describe-tests to teuthology-describe

describe_tests: improve summary output

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology-describe-tests: rename to teuthology-describe

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe-tests: rename short options for better usability

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/build_matrix: get rid of '.yaml' in descriptions

Test descriptions have structured representation
fragments the test config is composed from.
It's worth noticing that no other files except yaml
are used for description composition.
Although having the .yaml extentions in the description
looks redundant and has no use in the code.

This change removes many '.yaml' sufixes from
descriptions which make it more compact and easy
to read.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1479 from ceph/rh-internal-tasks

fixes for redhat internal tasks

added todo for support of submodules

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

fixes for redhat internal tasks

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Merge pull request #1480 from ceph/fog-sentinel

provision.fog: Add optional sentinel file

Merge pull request #1248 from kshtsk/wip-calc

teuthology-describe-tests: add --summary command

provision.fog: Add optional sentinel file

Similar to how provision.cloud.openstack works, wait for a particular
file to be created before considering provisioning to be finished.

Unlike the cloud.openstack module, teuthology doesn't control the entire
process, so to avoid breaking any labs that aren't using a sentinel
file, make the feature optional and default to being inoperative.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1478 from kshtsk/wip-revert-add-retry-exc

Revert "teuthology/orchestra/connection: add retry exceptions"

Revert "teuthology/orchestra/connection: add retry exceptions"

This reverts commit 1e30d6f624921ae98c27981d55e06c29d1b0fdd0.

teuthology-describe-tests: add --summary command

Add --summary argument to teuthology-describe-tests for estimating
run capacity, printing descriptions, dumping matrix, yaml fragments.

Refactor get_combinations code to use the same code that
is used by teuthology-suite for filtering.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1476 from ShraddhaAg/update-requirements

docs: acknowledge py2 and py3 where requirements.txt is mentioned

docs: acknowledge py2 and py3 where requirements.txt is mentioned

This commit removes mention of deleted file requirements.txt and
acknowledges the two separate requirements files, requirements2.txt
and requirements3.txt, for python2 and python3 respectively, in the
following files:

1. MANIFEST.in
2. docs/INSTALL.rst

suite/run: isolate description filter code

Refactor collect_jobs in order to isolate descriptions filter code
in order to make it possible to reuse it and test better.

Introduce --filter-fragment option defaults to False, to align code
logic with --filter help and documentation.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #1466 from batrick/i45438

teuthology/orchestra/connection: add retry exceptions

Reviewed-by: Kefu Chai <kchai@redhat.com>

teuthology-suite: improve help for --suite-dir arg

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite: schedule first and last in suite jobs once

Only schedule --first-in-suite and --last-in-suite jobs
once per run when --num is provided for a suite.

Fixes: https://tracker.ceph.com/issues/45520
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology/run_tasks: do not call sys.exc_clear() for clearing exception

sys.exc_clear() was removed in Python3, see
https://docs.python.org/3/whatsnew/3.0.html#index-22. so we should not
call it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra/connection: add retry exceptions

Typical connection errors would cause a command to fail.

Fixes: https://tracker.ceph.com/issues/45438
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1366 from kshtsk/wip-filter-all

suite: add filter-all parameter

Merge pull request #1471 from tchaikov/wip-45512

teuthology/orchestra: open log file in Python3 way

Reviewed-by: David Galloway <dgallowa@redhat.com>

teuthology/orchestra: open log file in Python3 way

otherwise we could have
```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'file' is not defined
```

Fixes: https://tracker.ceph.com/issues/45512
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1463 from ceph/rh-changes

various updates to redhat internal install task

suite: add --filter-all parameter

Add --filter-all parameter to 'teuthology-suite' to allow
precise test case selection. The --filter-in makes use of filter list
with logical 'or' operation to include test cases.
The --filter-out takes filter list and excludes any test case which
match any from the filter list.

This patch provides with 'and' logical operation for a given filter list.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fixes to install task to handle extra system pkgs

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>
Fix unit test cases

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

various updates to redhat internal install task

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1470 from ceph/flake8-py3

fix some old flake8 issues

Fix flake8 issues

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #1469 from tchaikov/wip-45255

Revert "orchestra/remote.py: use ensure_online() helper in run()"

Revert "orchestra/remote.py: use ensure_online() helper in run()"

This reverts commit 478bb3f661621c38ac0b9bb21389cc5b225c318d.

Fixes: https://tracker.ceph.com/issues/45255
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1465 from tchaikov/remote-sudo-tar

teuthology/orchestra/remote.py: write remote file without sudo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/orchestra/remote.py: write remote file without sudo

the remote file is created using "ubuntu" user. in ubuntu xenial,
superuser is able to write to that file using tar, it seems that "tar"
starts another process when writing to the dest file specified by "-f",
and that process does not have the privilege for writing that file.

so when we are trying to archive a directory on ubuntu/focal test node,
we have following error:
```
tar (child): /tmp/tmp.vkl0kAtc06: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
tar: /tmp/tmp.vkl0kAtc06: Cannot write: Broken pipe
tar: Child returned status 2
tar: Error is not recoverable: exiting now
```

and this is reproduciable with tar 1.30:
```
$ touch /tmp/helloworld
$ mkdir /tmp/foobar
$ sudo tar czf /tmp/helloworld -C /tmp/foobar -- .
tar (child): /tmp/helloworld: Cannot open: Permission denied
tar (child): Error is not recoverable: exiting now
tar: /tmp/helloworld: Cannot write: Broken pipe
tar: Child returned status 2
tar: Error is not recoverable: exiting now
```

but we can workaround this by writing the temp file using "ubuntu" user, like
```
sudo tar czf - -C /tmp/foobar -- . > /tmp/helloworld
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1464 from sunilkumarn417/master

updated pid method to retrieve correct daemon process id

updated pid method to retrieve correct daemon process id.

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1462 from sebastian-philipp/cephadmunit-signal-py3

cephadmunit: Python3 fix: signal.SIGHUP is no longer an int.

cephadmunit: Python3 fix: signal.SIGHUP is no longer an int.

py2: `str(signal.SIGHUP) == "1"`
py3: `str(signal.SIGHUP) == "signal.SIGHUP"`

Fixes: https://tracker.ceph.com/issues/45297
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1460 from ceph/wip-focal

opsys: Add Focal Fossa

opsys: Add Focal Fossa

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1452 from toabctl/add-cli-docs

Add cli docs

docs: Drop mention of teuthology-coverage

The command was dropped in 865a74b750.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

docs: Link commands to generated help

This is useful to quickly show the available CLI help for the
different commands.
For that, auto generate labels based on the headline via a Sphinx
extension.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

docs: Add CLI help to docs

It's useful to have the command help available in the
documentation.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

docs: Use python3 for building docs

python2 is EOL.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #1451 from toabctl/bump-cffi

requirements: Bump cffi version to 1.14.0

Reviewed-by: Kefu Chai <kchai@redhat.com>

requirements: Bump cffi version to 1.14.0

python 3.8 needs a newer cffi version. So bump it. Otherwise, calling
"./bootstrap" with python3.8 will fail.
This is also done for the python2 requirements to be in sync.

Fixes: https://tracker.ceph.com/issues/45132
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #1450 from tchaikov/wip-suite-filter

teuthology/suite: extract match() for filtering jobs

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

teuthology/suite: extract match() for filtering jobs

for better readablity

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1449 from kshtsk/wip-45105

lock/cli: fix broken teuthology-lock --summary for py3

Reviewed-by: Kefu Chai <kchai@redhat.com>

lock/cli: fix broken teuthology-lock --summary for py3

Convert None type to empty str in sorted key lambda
The usage of .format method is also adjusted for empty
machine type values

Fixes: https://tracker.ceph.com/issues/45105
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1443 from kshtsk/wip-sleep-before-unwinding

suite: Implement --sleep-before-teardown option

suite/run: add more interaction for sleep-before-teardown

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1422 from kshtsk/wip-py3-enable

enable py3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1445 from tchaikov/wip-safe-while

teuthology: pass integer as "tries" to safe_while()

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology: pass integer as "tries" to safe_while()

in Python3, `a / b` could return a float, and `safe_while()` uses
`itertools.islice()` under the hood, where `stop` should be None or an
integer. so let's use `a // b` instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

suite: Implement --sleep-before-teardown option

With this option, all jobs in the run sleep for the given number of seconds as
the very last thing they do, regardless of success or failure.

Use case: interactive debugging

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite: add non interactive mode argument

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1444 from kshtsk/wip-fix-coredump

task.internal.archive: fix 'Permission denied' error when coredump found

Reviewed-by: Kefu Chai <kchai@redhat.com>