build(deps): bump cryptography from 3.2 to 3.3.2

Bumps [cryptography](https://github.com/pyca/cryptography) from 3.2 to 3.3.2.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/3.2...3.3.2)

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1609 from ceph/dependabot/pip/httplib2-0.19.0

build(deps): bump httplib2 from 0.18.0 to 0.19.0

Merge pull request #1610 from jdurgin/wip-supervisor-timeouts

supervisor: improve error handling for dead jobs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

supervisor: send paddles the reason a jobs is marked dead

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

supervisor: kill processes before gathering logs

When we hit the max job timeout, we need to stop the test programs
before collecting logs or else we run into errors like 'file size
changed while zipping' trying to compress them, and we can't save them
or stop the job.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

nuke: allow not rebooting again

The default behavior was changed to always reboot in
1d47a121b385e2656e9314e9d63faf68a8e865e4 but the --reboot-all option
remained. Keep the original option around for compatibility with
existing scripts.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

nuke: add option to preserve logs on remote machines

This will be helpful for killing jobs that hit the max_job_timeout
while still being able to collect logs from them.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

build(deps): bump httplib2 from 0.18.0 to 0.19.0

Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.18.0 to 0.19.0.
- [Release notes](https://github.com/httplib2/httplib2/releases)
- [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG)
- [Commits](https://github.com/httplib2/httplib2/compare/v0.18.0...v0.19.0)

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1608 from kshtsk/fix-docs

readme: fix teuthology docs link at docs.ceph.com

Merge pull request #1601 from sebastian-philipp/prio-add-job-count

teuthology-suite: Add job count to priority error msg.

readme: fix teuthology docs link at docs.ceph.com

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1607 from kshtsk/ver-1.1.0

version: increase version to 1.1.0 since we have dispatcher

Merge pull request #1606 from kshtsk/supervisor-log

dispatcher: add .log extension for supervisor log

version: increase version to 1.1.0 since we have dispatcher

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

dispatcher: add .log extension for supervisor log

It would be great to have an extension for easy log identification.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1605 from jdurgin/wip-supervisor-connect-error

dispatcher/supervisor: always unlock machines and save status

dispatcher/supervisor: always unlock machines and save status

If we can't connect to the machines anymore, we still need to clean
up.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1604 from jdurgin/wip-dispatcher-commit-bug

dispatcher/repo_utils: handle missing commits better

Reviewed-by: David Galloway <dgallowa@redhat.com>

dispatcher: keep operating if preparing a job fails

prep_job() handles updating the job status already.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

repo_utils: clone entire branch if commit is specified

If the commit is not the head of the branch, we need more history to be
able to check it out.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

worker: handle CommitNotFoundErrors

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1603 from jdurgin/wip-dispatcher-bug

dispatcher: allow empty os_type for fake config

Reviewed-by: David Galloway <dgallowa@redhat.com>

dispatcher: allow empty os_type for fake config

This is the same default as reimaging uses,
though it's not too important in the supervisor.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

teuthology-suite: Add job count to priority error msg.

Don't let users guess the job count.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1546 from ShraddhaAg/add-minimal-dispatcher

Add teuthology-dispatcher

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1599 from ceph/wip-exact-commits

Use the same version of teuthology and ceph-qa-suite for a whole run

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1598 from SrinivasaBharath/wip-deb-rm

task/install/redhat: Removing packages based on OS in cleanup task

Merge branch 'master' into add-minimal-dispatcher

Merge pull request #1600 from rzarzynski/wip-valgrind-controllable-exit

teuthology/misc: make the Valgrind's early exit configurable.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/misc: make the Valgrind's early exit configurable.

This commit is a follow-up to a98eb3e1405c8ca8f6933eb0356c03955e4e2e83
where Valgrind has been configured to exit on first-seen error as it
was (wrongly!) assumed that all components are green when it comes
to the Valgrind verification.
This assumption turned out to be broken for RGW which got a few issues
over the course as a result of having the Valgrind checks knocked out
as a side effect of the python3 transition [1]. In the consequence,
multiple problems accumulated and introducing a mechanism to disable
the early exit to e.g. develop a list of these issues looks desirable.

[1]: https://github.com/ceph/teuthology/pull/1503#issuecomment-762837504

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

worker, run: use exact commits for teuthology and qa suite

This ensures we use the same version across all jobs in a run.

We already have suite_sha1 set by older versions of teuthology, but
for folks who haven't updated their suite command, and thus don't set
teuthology_sha1 in the job config, look up the sha1 of in the worker.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

repo_utils: allow fetching a specific sha1 to per-commit directories

Using a checkout of a single branch used by potentially many
workers/teuthology processes can result in errors when one job
updates the local branch while another job is reading it.

This causes issues particularly easily when using non-master
teuthology branches, and with the teuthology-dispatcher.

This also allows us to guarantee we're using the same
version across an entire run, even if e.g. the master
qa suite is updated between jobs.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

suite: pass the commit of teuthology with each job's config

We already pass the suite sha1, but do not use it yet. This is the
missing piece to be able to use the same version of everything across
a whole run.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

repo_utils: allow checking out a specific commit

Since we're cloning a particular branch with git clone --shallow,
assume we're still passed a branch that contains the commit. Otherwise
we'd waste time and space cloning all the branches in the repo.

Assume that this is only used for checking out a particular sha1 once,
to avoid repetitive work.

There are two use cases for these utilities:

1) on the user's machine when they're scheduling a suite - there it
will make sense to maintain a single checkout of a branch e.g. teuthology master

2) on the queue consumer side - here it's best if we use the same commit for an
entire run, so checking out by sha1 makes more sense

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

task/install/redhat: Removing packages based on OS in cleanup task

Signed-off-by: Bharath <skanta@redhat.com>

Merge pull request #1597 from jdurgin/wip-workunit-fingerprint

exceptions: only use one of label or command for fingerprint

Reviewed-by: Neha Ojha <nojha@redhat.com>

exceptions: only use one of label or command for fingerprint

Commands like those running workunits include the ceph sha1 being
tested, so they're not useful for grouping. This also lets us group
together other tests if we like, for example to map tests with small
differences in configuration to the same fingerprint for sentry.

Also use the plain command, it's already a string at this point
so there's no reason to add spaces between its characters.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #1595 into master

* refs/pull/1595/head:
orchestra: squelch Traceback for expected auth failures

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra: squelch Traceback for expected auth failures

The Traceback clutters the log and messes up greps for Tracebacks.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1593 from jdurgin/wip-sentry-sdk

sentry: use new library and group CommandFailedErrors more finely

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

exceptions: group CommandFailedErrors in sentry more finely

By default sentry uses the stack trace / error type / rough error
message, which ends up with many failures from different workunits
grouped together. Include the actual command run, the exit status, and
the optional label to group these more accurately. This will group
failures of the same workunit together, for example.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

run_tasks: use new sentry_sdk

raven has been deprecated

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1591 from rakeshgm/typo-correct

run_tasks: correct typo tuethology -> teuthology

run_tasks: correct typo tuethology -> teuthology

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Merge pull request #1590 from lxbsz/task_ship_utilities

teuthology: run the ship_utilities task only once

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

teuthology: run the ship_utilities task only once

This will make sure that the utilities won't removed until the last
user is unwound.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1589 from kshtsk/wip-wait

Add teuthology-wait command

scripts: add wait script for watching run

While using teuthology-suite with --wait option it is
usefull sometimes to split the suite scheduling and
the run waiting. For example, when using tools like
Jenkins we might want to schedule a suite, report
about successful schedule and start waiting only in
the next steps.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite: improve info message about waiting the run

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1584 from kshtsk/wip-quiet-run

orchestra: introduce quiet mode for remote.run

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1588 from lxbsz/install1

install: pass the 'shaman' to Shaman class

install: pass the 'shaman' to Shaman class

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1585 from kshtsk/wip-rocket

teuthology-suite: add Rocket.Chat notification

teuthology-suite: add Rocket.Chat notification

Add Rocket.Chat notification for sleep before teardown.
For details see https://rocket.chat/

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1578 from lxbsz/override

packaging: try noarch repo if arch repo doesn't exist

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1581 from ideepika/add-interactive-on-error

 teuthology command: introduce --interactive-on-error flag

Merge pull request #1586 from tchaikov/wip-gzip-archive

task.internal.archive: gzip archived file size > 128MB

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

task.internal.archive: gzip archived file size > 128MB

* misc: add an optional write_to argument to misc.pull_directory()
        so the caller can optionally specify the function to write
        to local file.
* task/internal: add a global option "log-compress-min-size" which
        defaults to "128MB". if the size of a file pulled from remote
        host is greater or equal to the specified size, it will be
        compressed with gzip with the extension of ".gz" before
        stored in the archive directory.

Signed-off-by: Kefu Chai <kchai@redhat.com>

docs/detailed_test_config: document "archive-on-error" option

Signed-off-by: Kefu Chai <kchai@redhat.com>

packageing: add force_noarch option support

Add one "force_noarch : True" option to force to use the "noarch"
to build the uri, False as default.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

orchestra: introduce quiet mode for remote.run

Applied changes:

- Add quiet option to remote.run and subsidiary function calls
- Logging commands now directed to DEBUG instead of INFO logger

This is usefull when we want suppress logs for some kind of commmands like
reading binary files or logging useless data to stdout/stderr as well as
dumping some vulnarable information.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1575 from mdw-at-linuxbox/ssh-ecdsa

orchestra/connection: accept ecdsa (and future) host key types.

Reviewed-by: Kefu Chai <kchai@redhat.com>

orchestra/connection: accept ecdsa (and future) host key types.

Out of the box, centos 8 ssh daemon makes this file,
/etc/ssh/ssh_host_ecdsa_key.pub
containing a key of type "ecdsa-sha2-nistp256", which was
not recognized by the existing teuthology logic.
Use logic in paramiko.hostkeys to recognize the new key types.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

docs/detailed_test_config: document ``--block`` option

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

teuthology command: introduce --interactive-on-error flag

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #1580 from ceph/rh_ds_yml

redhat downstream yaml location changed and redhat install tasks to accept lists

Merge pull request #1577 from dmick/wip-container-build-complete

Check shaman not only for repo but for build complete

redhat ds yaml location changed and redhat install tasks to accept lists

fetch the dowstream yaml location from
downstream teuth_config and redhat install tasks
to accept lists too

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Check shaman not only for repo but for build complete

The package repo might be done, but the build must be totally
complete for the container to be present.  For a specific build
(in global variables, for now), demand that the build be complete
before claiming packages exist.  This allows suite jobs to
fail or --newest to continue searching when the container isn't
done.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1572 from lxbsz/deb

task/install/deb: retry installing the package if resource temporarily unavailable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1574 from ceph/dependabot/pip/cryptography-3.2

build(deps): bump cryptography from 2.8 to 3.2

Merge pull request #1576 from mdw-at-linuxbox/centos-8-stream

misc: CentOSStream => rpm

misc: CentOSStream => rpm

lsb_release -is identifies CentOS 8 Stream as CentOSStream.
Teuthology should recognize this as 'rpm'.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #1559 from ryg-/wip-fix-domain

Makes (de)canonicalize_hostname ready to empty domains

build(deps): bump cryptography from 2.8 to 3.2

Bumps [cryptography](https://github.com/pyca/cryptography) from 2.8 to 3.2.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/2.8...3.2)

Signed-off-by: dependabot[bot] <support@github.com>

Makes (de)canonicalize_hostname ready to empty domains

teuthology/misc.py : addied support of empty domains to
 misc.canonicalize_hostname and misc.decanonicalize_hostname.

teuthology/test/test_misc.py: added test cases for new code

Signed-off-by: Roman Grigorev <rgrigorev@suse.de>

deb: retry installing the package if resource temporarily unavailable

Fixes: https://tracker.ceph.com/issues/46878
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1560 from ryg-/wip-pelagos-speedup-parking

provision/pelagos: improve parking speed

provision/pelagos: Speed up pelagos node parking

Speed up the process of booting nodes to a special image that
is excluded node from access from other nodes.

Signed-off-by: Roman Grigorev <rgrigorev@suse.de>

Merge pull request #1568 from ceph/wip-rpm-upgrade

task/install/rpm: fix issued with upgrading packages under DNF

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #1554 into master

* refs/pull/1554/head:
rpm: retry installing the package if the mirror server is busy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

task/install/rpm: fix issued with upgrading packages under DNF

yum is an alias for dnf on modern RHEL-like systems but it's not
100% compatible. The "install" command does not upgrade packages
unless you give it a specific version to install. Additionally,
the code was incorrectly inserting two blank arguments that resulted
in "yum install" failing.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #1567 from tchaikov/wip-pip-use-feature

bootstrap: correct the check for --use-feature=2020-resolver

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

bootstrap: correct the check for --use-feature=2020-resolver

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1566 from toabctl/fix-pip-new-resolver

bootstrap: Do not use --use-feature=2020-resolver for pip upgrade

Reviewed-by: Kefu Chai <kchai@redhat.com>

bootstrap: Conditionally use --use-feature=2020-resolver for pip

pip might be too old so the --use-feature flag is not available. That
leads to:

$ ./bootstrap
Removing old virtualenv because it uses system site-packages
created virtual environment CPython3.8.5.final.0-64 in 173ms
  creator CPython3Posix(dest=/home/tom/devel/ceph/teuthology/virtualenv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tom/.local/share/virtualenv)
    added seed packages: pip==20.1.1, setuptools==50.3.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Usage:
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

no such option: --use-feature

So for the pip upgrade itself, check if the flag is available. From
there on, use the new resolver.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #1563 from tchaikov/pip-resolver

bootstrap: use pip --use-feature=2020-resolver

Reviewed-by: David Galloway <dgallowa@redhat.com>

bootstrap: use pip --use-feature=2020-resolver

follow the suggestion from pip

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1561 from tchaikov/wip-doc-rtd

doc: add requirements.txt

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: add requirements.txt

so we can built this project using readthedocs.org

Signed-off-by: Kefu Chai <kchai@redhat.com>

dispatcher: make archive flag optional

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Merge pull request #1557 from kshtsk/wip-id_rsa-0500

Use remote.write_file instead of misc.create_file

task/ssh_keys: use remote.write_file instead misc.create_file

Because misc.create_file with data arguments creates file,
changes permissions, and later appends the data using
misc.append_lines_to_file, this logic makes the algorythm
above to fail if the file created without write permissions.

    2020-09-05T10:00:32.003 INFO:teuthology.task.ssh_keys:pushing keys to smithi086.front.sepia.ceph.com for ubuntu
    2020-09-05T10:00:32.003 INFO:teuthology.orchestra.run.smithi086:> rm -f -- /home/ubuntu/.ssh/id_rsa
    2020-09-05T10:00:32.046 INFO:teuthology.orchestra.run.smithi086:> touch /home/ubuntu/.ssh/id_rsa && chmod 500 -- /home/ubuntu/.ssh/id_rsa
    2020-09-05T10:00:32.095 INFO:teuthology.orchestra.run.smithi086:> set -ex
    2020-09-05T10:00:32.096 INFO:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/.ssh/id_rsa conv=notrunc oflag=append
    2020-09-05T10:00:32.140 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd of=/home/ubuntu/.ssh/id_rsa conv=notrunc oflag=append
    2020-09-05T10:00:32.142 INFO:teuthology.orchestra.run.smithi086.stderr:dd: failed to open '/home/ubuntu/.ssh/id_rsa': Permission denied
    2020-09-05T10:00:32.142 DEBUG:teuthology.orchestra.run:got remote process result: 1

However we have more advanced remote.write_file function now,
which does not have such issues and moreover creates file
with the data provided in a single hop without trying to
download the file locally.

Related-to: 243ff3bbf218102b24ed992e3931bf8c76cdaadd
Related-to: 45aba9cf8d72fd30ca607200694c638cfd240fd8
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/ansible: reset fail_log file index

The yaml.safe_load reads the fail_log opened file
and shifts the offset to the end of stream.
However in case of error we need to shift offset
to the begin of the file stream, so we can read
data again.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/remote: raise FileNotFoundError in read_file

Raise FileNotFoundError exception if the file passed to
remote.read_file was not found.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1538 from kshtsk/wip-remote-write-file

Add remote file operations

orchestra.remote: add move_file, copy_file and read_file

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

misc: optimize append_lines_to_file() to reuse remote.write_file

The misc.append_lines_to_file method has been using get
remote file to temporary file and then write back to
remote host.

The orchestra.remote.write_file method has 'append' parameter
when using which there is no data downloaded from remote host.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

misc: deprecate (sudo_)write_file methods

Deprecate misc.write_file() and misc.sudo_write_file()
in favor of the orchestra.remote package methods.
The code of the misc's methods is now calling the
remote's ones.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/remote: add (sudo_)write_file method

Add advanced write_file and sudo_write_file to remote context
for more intuitive and handy coding

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>