bootstrap: Tolerate a missing lsb_release

This fixes the lack of support for CentOS 9.Stream

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1846 from ceph/stdin-killer

teuthology/task/install: add stdin-killer helper

This helper tool runs commands which may or may not take data on stdin.
Like "daemon-helper", if stdin signals EOF, stdin-killer will kill the
command but only as a last resort. It forwards EOF to the command by
closing the command's stdin (pipe) and then waiting a configurable
amount of time for the command to gracefully exit.

Additionally, if stdout or stderr are hung up -- i.e. the ssh parent
process has terminated -- then stdin-killer also detects this and
initiates the graceful shutdown of the command. This is something
daemon-helper does not do.

In general, this tool is a superior replacement of the daemon-helper
tool because you can write to the command's stdin normally.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.cfg: install binary helpers

These are used by vstart_runner.py for local dev operations. Install
them so they are available in the virtualenv bin directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install: reorganize binary helpers

We intend to install these so move them into an appropriately named
directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1803 from jdurgin/wip-configurable-timeouts

Merge pull request #1851 from ceph/reimage-failures

fog: Verify reimaged machine OS

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1850 from ceph/unmask-unlock-response

Merge pull request #1849 from ceph/prom-reimage-results

exporter: Instrument node reimaging success/fail

lock.ops.unlock_one: Fail sooner on 403, with msg

In the case of e.g. owners values not matching on an unlock attempt, we
were exhausting all retries and failing to display the exact reason for
the unlock failure. We can simply break on 403 errors and let the rest
of the function do its thing.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: make timeouts configurable

This will help with the sepia lab, being able to increase these
temporarily to handle a new fog server that is sometimes exceeding the
hardcoded timeouts.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

contextutil: allow safe_while to use an explicit timeout

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

nuke: Fix an import issue

Signed-off-by: Zack Cerza <zack@redhat.com>

lock.ops.reimage_machines: Drop incorrect log msg

This message was being logged when the reimage started, not finished.

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Instrument node reimaging success/fail

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1847 from ceph/fog-timeout

fog: Increase timeout in wait_for_deploy_task()

test_fog: Fix up test for preceding commit "Increase timeout"

Signed-off-by: Dan Mick <dmick@redhat.com>

fog: Increase timeout in wait_for_deploy_task()

When too many reimaging ops are running concurrently, we're seeing
timeouts. This isn't a true fix, but should help things until we've got
one.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1843 from dmick/wip-kernel-sort

Merge pull request #1844 from cbodley/wip-install-copr

task: install supports enable_coprs array

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Zack Cerza <zcerza@redhat.com>

task: install supports enable_coprs array

enable the installation of packages in fedora copr repositories

Signed-off-by: Casey Bodley <cbodley@redhat.com>

task/kernel.py: sort installed kernels by version

rpm -q --last sorts by timestamp-of-install, which does not
necessarily correlate with "latest version".  sort -rV does.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1804 from ceph/dependabot/pip/wheel-0.38.1

build(deps): bump wheel from 0.36.2 to 0.38.1

build(deps): bump wheel from 0.36.2 to 0.38.1

Bumps [wheel](https://github.com/pypa/wheel) from 0.36.2 to 0.38.1.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.36.2...0.38.1)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1831 from kamoltat/wip-ksirivad-rerun-readme

teuthology-suite: --seed & --subset now also stored in teuthology.log, config.yaml and orig.config.yaml
Reviewed-by: Zack Cerza <zcerza@redhat.com>

Merge pull request #1834 from ceph/disp_ls_remote

worker.prep_job: Skip job if ls_remote fails

worker.prep_job: Skip job if ls_remote fails

This is preferable to letting the dispatcher die.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite/run.py: Added seed and subset to base_config

In addition to being stored in results.log
`--seed` and `--subset` are now also stored in:

`teuthlogy.log`, `config.yaml` and `orig.config.yaml`.

Fixes: https://tracker.ceph.com/issues/59300
Signed-off-by: Kamoltat <ksirivad@redhat.com>

teuthology-suite: Log errors & warnings if results.log is missing during --rerun

Notify the user if `results.log` is missing when
they issue a rerun.

Also, edited teuthology-suite doc
to inform the user about how `--rerun` by
default parse `--seed`, `--subset` and
`--no-nested-subset` from `results.log`
by default

Fixes: https://tracker.ceph.com/issues/59300
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #1832 from ceph/deps

Merge pull request #1827 from ceph/tox-no-osp

Drop argparse as a requirement

It's part of the standard library.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements.txt: Add prometheus_client

This should have been added a couple PRs back.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1833 from ceph/gha-ubuntu-version

bootstrap: apt-get update before installing

Signed-off-by: Zack Cerza <zack@redhat.com>

.github: Explicitly define test matrix

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1824 from batrick/parallel-gzip

teuthology: do not compress tarballs when pulling dir

Where we use this, it's for pulling log files that are already
compressed. Do not waste time double compressing!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: give verbose gzip output

For future analysis.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: use medium compression

To speed things up.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: parallelize gzip

Our machines have lots of cores, use them!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1826 from ceph/job-time

tox: Don't run openstack by default

It's quite time-consuming, and we're not sure if it's in use at all.

Signed-off-by: Zack Cerza <zack@redhat.com>

sueprvisor: Do not instrument certain job times

This should only really include first/last-in-suite jobs.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1819 from ceph/timing

task.internal: Fix import of get_builder_project

This is another problematic circular import.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.kernel: Make task() a context manager

This will cause it to be instrumented for timing via the bits in
run_tasks.

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Make JobResults a singleton

This should avoid "Duplicated timeseries in CollectorRegistry" errors.

Signed-off-by: Zack Cerza <zack@redhat.com>

docs: Add new document for teuthology-exporter

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Add instrumentation for reimaging time

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils: Add instrumentation for bootstrap time

And fix a couple of linter errors.

Signed-off-by: Zack Cerza <zack@redhat.com>

run_tasks: Add instrumentation for task time

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Add instrumentation for job time

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Add instrumentation for locking time

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1825 from ceph/cl-import

nuke: Fix import of task.internal.check_lock

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1823 from ceph/fix-kill

task.cephmetrics: Fix broken import

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix more circular imports

Signed-off-by: Zack Cerza <zack@redhat.com>

Test all module imports

Signed-off-by: Zack Cerza <zack@redhat.com>

Restore some import statements

Some of the circular import fixes were overkill.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1822 from ceph/fix-kill

kill.kill_job: Tweak job status reporting

kill.kill_job: Tweak job status reporting

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 45117a8781ce6438f22140c00d05435c7781648b)

Merge pull request #1817 from ceph/exporter

Fix a circular import

This was exposed during development of the exporter.

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Set flake8 max line length to 100

It's what black uses.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-exporter: Report job status

Signed-off-by: Zack Cerza <zack@redhat.com>

Add teuthology-exporter, a Prometheus exporter

Signed-off-by: Zack Cerza <zack@redhat.com>

Add active_machine_types config item

I'm adding this so that the exporter can know which machines types to
care about.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1818 from ceph/fix-nop

task.ansible: Fix an IndexError w/ 0 remotes

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1815 from kshtsk/wip-sle15sp3

orchestra/opsys: add sle 15 sp5

Merge pull request #1816 from ceph/paddles-fault-tolerance

supervisor: Ignore some reporting failures

Specifically, the pings coming from the watchdog loop

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Do not time out when locking machines

Signed-off-by: Zack Cerza <zack@redhat.com>

safe_while: Retry forever when tries == -1

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1812 from ceph/orch-ssh-key

orchestra/opsys: add sle 15 sp5

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1814 from NitzanMordhai/wip-nitzan-scrapy-deadjobs-nonetype-match

scrapy: comparing dead job skip typeNone

Merge pull request #1793 from cybozu/fix-docker-compose-postgres

Specify PostgreSQL v14 to avoid permission errors.

scrapy: comparing dead job skip typeNone
SequenceMatcher can't get NoneType when comparing 2 dead jobs,
adding check and compare for those jobs.

Fixes: https://tracker.ceph.com/issues/58264
Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>

orchestra.connection: Look for 'ssh_key' in config

On systems where SSH is configured using ~/.ssh/config.d/, we don't have
a great way of locating and processing the full configuration, so we
might not find the correct key to use. Allow the user to specify one key
to use for all test nodes if they wish.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1811 from ceph/wip-58491

misc.get_system_type: Use Remote.os

misc.get_system_type: Use Remote.os

... instead of lsb_release, as newer distros don't ship it.

Fixes: #58491
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1810 from ceph/selinux-denial-fix

task/selinux: Fix regressed grepping of audit logs

task/selinux: Fix regressed grepping of audit logs

Fixes: https://tracker.ceph.com/issues/58610
Introduced: a106217
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1805 from ceph/el-py39

bootstrap: Correct python3 package names for EL8/9

Merge pull request #1807 from kshtsk/wip-update-leap

orchestra/opsys: add newer sle and leap versions

orchestra/opsys: add newer sle and leap versions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstrap: Correct python3 package names for EL8/9

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1806 from ceph/fix-pytest

tests: Fix warnings, deprecations, and broken tests

Merge pull request #1802 from ceph/faster-newest

suite: Speed up 'newest' feature

tox.ini: Update to reflect passenv change

See the second bullet point here:
https://tox.wiki/en/4.0.3/faq.html#tox-4-changed-ini-rules

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix bugs revealed by mock's recent change

Once the asserts were corrected, these tests started failing.
Fortunately, they didn't seem to be masking any actual bugs.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix broken Mock assert statements

This breaks (or, rather, reveals the broken state of) several unit
tests. A subsequent commit will fix them.

See https://github.com/testing-cabal/mock/commit/21787a948cf1186f3bd38462d5b912784fa81218

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Make sure to patch repo_utils ops

These tests were hitting the network.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Rename setup() and teardown() methods

These were nose methods, and pytest is dropping support for them.

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix 'invalid escape sequence' deprecation warnings

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1799 from dmick/wip-bootstrap

bootstrap: rpm systems: use --whatprovides to test for package presence