worker: fix crash on try_delete_jobs

In some circumstances workers try to delete a job which
is not created yet. It quits immediately with unhandled
exception error.

We do not want to allow crashes whatever error happens,
because worker can continue working and process other
jobs correctly.

Also some schedulers might not add corresponding
first in suite and last in suite jobs to the reporter,
since it is not required.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1521 from kshtsk/fix-install-repos

run: fix typo after refactor

run: fix typo after refactor

This addresses issue introduced by PR 1520 commit:
b9fe404403db8f983a402ed64b85a2a292de66af

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1520 from tchaikov/wip-check-per-job-verify_ceph_hash

suite/run.py: check config['verify_ceph_hash'] before verifying ceph …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/run: do not check ceph packages if verify_ceph_hash is disabled

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/run: use default value for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

suite/run.py: check config['verify_ceph_hash'] before verifying ceph packages

there is chance that we don't have ceph packages built for the
combination of specified os_type and os_version, for instance, in the
case of rados/thash_old_clients, older ceph clients are installed on
el7, but the ceph cluster is deployed using cephadm, which in turn pull
ceph container images built using the ceph being tested and el8.

since we've dropped the build of master on el7, there is no need to
verify if ceph package is available if cephadm is used for deploying the
cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1519 from tchaikov/wip-query-packages-of-el8

teuthology/suite: use rhel8 for querying package version

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology: use rhel8 for querying package version

as we don't build packages for el7 on master anymore. but we do build
packages for nautilus and octopus.

* suite: use rhel8 for querying package version
* orchestra/opsys: update default OS versions
  centos/rhel 7.6 -> 8.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1516 from kshtsk/wip-increase-clock-timeout

teuthology/task/clock: increase timeout

Merge pull request #1514 from sunilkumarn417/master

Fixed empty string split issue

teuthology/task/clock: increase timeout

Increase timeout because 1 minute is not always enough
to configure ntp.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fixed empty string split issue

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1510 from tchaikov/wip-drop-py2

drop python2 support

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

rename requirements3.txt to requirements.txt

since there is no need to differentiate requirements3.txt from
requirements2.txt.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Added comment explaining the usage of the unpatched kill

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Used unpatched os.kill with test_basic in test_exit.py

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Changed the import of mock to unittest.mock

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

teuthology/orchestra/run: drop code handling py2

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/suite: drop six.moves.input()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/provision: drop six/string_types

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/run_tasks.py: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra: drop six.StringIO

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/provision: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

misc, repo_utils: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/suite: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra: drop six

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology: drop six.ensure_str()

use get_file(...).decode() instead

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/kill: drop six.ensure_str

Signed-off-by: Kefu Chai <kchai@redhat.com>

tasks: drop six.StringIO

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/task: drop six.string_types

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/task: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology: drop six.reraise()

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: update pacakge metadata to support python3

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/test: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

drop python2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1509 from tchaikov/wip-secure-yaml-load

nuke: s/yaml.load/yaml.safe_load/

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox.ini: use python3 for testing openstack tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

nuke: s/yaml.load/yaml.safe_load/

to silence following warning:

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1508 from lxbsz/ulimit

task/install/adjust-ulimits: use a larger numbers for open files

Reviewed-by: Kefu Chai <kchai@redhat.com>

adjust-ulimits: use a larger numbers for open files

For some cephfs test case it will run handreds of threads and with
some mounts in each, we need a larger limit on the number of open
file descriptors.

Fixes: https://tracker.ceph.com/issues/45829
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1507 from tchaikov/wip-docs-virtualenv-py2

docs: create a python3 virtualenv instead a py2 one

Reviewed-by: Jos Collin <jcollin@redhat.com>

docs: create a python3 virtualenv instead a py2 one

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1506 from sunilkumarn417/master

eliminating kernel task duplication

Merge pull request #1505 from ceph/bootstrap-fedora-rhel

bootstrap: python3 packages for fedora and redhat based systems

python3 packages for fedora and redhat based systems

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

eliminating kernel task duplication

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1503 from ceph/wip-valgrind-exits-on-error

teuthology/misc: Valgrind exits on first detected error.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/misc: Valgrind exits on first detected error.

Valgrind v3.14 brought the `--exit-on-first-error` option
which causes termination of the process on first encountered
error. Although it does nothing for memory leaks, it should
help with pin-pointing e.g. `UninitValue` without imposing
the costly `--track-origins=yes`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1502 from ceph/fix-darwin

tox: Avoid system site-packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1490 from kshtsk/wip-auto-t

teuthology-suite: automate -t argument default value

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tox: Avoid system site-packages

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-suite: automate -t argument default value

This change is required to determine which teuthology branch should
be used for scheduling a run for the given ceph branch.

Drop the code using --ceph-branch and when --teuthology-branch is not
supplied, since this mechanism is not used and not supported anymore.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1499 from sebastian-philipp/write-file-unicode

orchestra/run: py2 fix for copy_and_close

Merge pull request #1497 from ceph/drop-mysql-dep

Drop MySQL dependency

Drop results.generate_coverage()

This was just an oversight from #1419

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop MySQL dependency

I think it's been years since anything in the project used the
mysqlclient package, so let's stop making everyone install it and everything it requires.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/run: py2 fix for copy_and_close

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1495 from kshtsk/plgs-logs

provision/pelagos: improve logging

provision/pelagos: improve logging

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1496 from tchaikov/wip-bootstrap-py3

bootstrap: s/python-pip/python3-pip/ for python3 on debian derivatives

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1498 from ceph/wip-waitlonger

console.py: Wait longer for IPMI timeouts

console.py: Wait longer for IPMI timeouts

The smithi have really crappy/flaky BMCs.  I occasionally see `'wait for power on' reached maximum tries (5) after waiting for 20.0 seconds` but when I go to check the BMC, it just needed some more time.  So let's give it more time.

Signed-off-by: David Galloway <dgallowa@redhat.com>

bootstrap: s/python-pip/python3-pip/ for python3 on debian derivatives

python-pip was removed in debian/bullseye and ubuntu/focal, so let's
replace it with python3-pip. and more importantly, for python3 support
we should install python3-pip instead.

Fixes: https://tracker.ceph.com/issues/45384
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1491 from ShraddhaAg/update-docs

docs: add docs.ceph.com link in README.rst

Merge pull request #1486 from kshtsk/wip-sbtd-email

run_tasks: add email notification for sleep-before-teardown

Merge pull request #1492 from toabctl/fix-opensuse-py3-bootstrap

Fix opensuse py3 bootstrap

Merge pull request #1493 from sebastian-philipp/toml

setup.py: add toml for qa/tasks/cephadm.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

setup.py: add toml for qa/tasks/cephadm.py

podman's /etc/containers/registries.conf is written
in toml. In order to modify it, we need to parse the
config.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1482 from kshtsk/wip-desc-without-yaml

suite/build_matrix: get rid of '.yaml' in descriptions

run_tasks: add email notification for sleep-before-teardown

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstap: Drop --setuptools from virtualenv installation

--setuptools is a no-op since ~2013 so drop the switch.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

bootstrap: Support python3 on openSUSE/SLES

Install the correct system packages when using a python3 env. Similar
to what is done in the ubuntu/debian case.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

docs: add docs.ceph.com link in README.rst

This commit adds a hyperlink to Running Integration Tests Using
Teuthology in docs/README.rst under the section Test Suites.

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Merge pull request #1489 from ceph/ignore-package-check

run: exclude internal check_packages and buildpackages_prep for redhat

skip internal check_packages and buildpackages_prep for redhat config

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Merge pull request #1474 from tchaikov/wip-no-clear-exec-py3

teuthology/run_tasks: do not call sys.exc_clear() for clearing exception

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #1473 from kshtsk/wip-schedule-once

suite: schedule first and last in suite jobs once

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1483 from kshtsk/wip-rename-describe-tests

rename teuthology-describe-tests to teuthology-describe

describe_tests: improve summary output

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology-describe-tests: rename to teuthology-describe

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe-tests: rename short options for better usability

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/build_matrix: get rid of '.yaml' in descriptions

Test descriptions have structured representation
fragments the test config is composed from.
It's worth noticing that no other files except yaml
are used for description composition.
Although having the .yaml extentions in the description
looks redundant and has no use in the code.

This change removes many '.yaml' sufixes from
descriptions which make it more compact and easy
to read.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1479 from ceph/rh-internal-tasks

fixes for redhat internal tasks

added todo for support of submodules

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

fixes for redhat internal tasks

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Merge pull request #1480 from ceph/fog-sentinel

provision.fog: Add optional sentinel file

Merge pull request #1248 from kshtsk/wip-calc

teuthology-describe-tests: add --summary command

provision.fog: Add optional sentinel file

Similar to how provision.cloud.openstack works, wait for a particular
file to be created before considering provisioning to be finished.

Unlike the cloud.openstack module, teuthology doesn't control the entire
process, so to avoid breaking any labs that aren't using a sentinel
file, make the feature optional and default to being inoperative.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1478 from kshtsk/wip-revert-add-retry-exc

Revert "teuthology/orchestra/connection: add retry exceptions"

Revert "teuthology/orchestra/connection: add retry exceptions"

This reverts commit 1e30d6f624921ae98c27981d55e06c29d1b0fdd0.

teuthology-describe-tests: add --summary command

Add --summary argument to teuthology-describe-tests for estimating
run capacity, printing descriptions, dumping matrix, yaml fragments.

Refactor get_combinations code to use the same code that
is used by teuthology-suite for filtering.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1476 from ShraddhaAg/update-requirements

docs: acknowledge py2 and py3 where requirements.txt is mentioned

docs: acknowledge py2 and py3 where requirements.txt is mentioned

This commit removes mention of deleted file requirements.txt and
acknowledges the two separate requirements files, requirements2.txt
and requirements3.txt, for python2 and python3 respectively, in the
following files:

1. MANIFEST.in
2. docs/INSTALL.rst

suite/run: isolate description filter code

Refactor collect_jobs in order to isolate descriptions filter code
in order to make it possible to reuse it and test better.

Introduce --filter-fragment option defaults to False, to align code
logic with --filter help and documentation.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

Merge pull request #1466 from batrick/i45438

teuthology/orchestra/connection: add retry exceptions

Reviewed-by: Kefu Chai <kchai@redhat.com>

teuthology-suite: improve help for --suite-dir arg

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite: schedule first and last in suite jobs once

Only schedule --first-in-suite and --last-in-suite jobs
once per run when --num is provided for a suite.

Fixes: https://tracker.ceph.com/issues/45520
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology/run_tasks: do not call sys.exc_clear() for clearing exception

sys.exc_clear() was removed in Python3, see
https://docs.python.org/3/whatsnew/3.0.html#index-22. so we should not
call it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra/connection: add retry exceptions

Typical connection errors would cause a command to fail.

Fixes: https://tracker.ceph.com/issues/45438
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>