PhysicalConsole: Tolerate invalid UTF-8 characters

... in pexpect.spawn() calls.

Fixes: https://tracker.ceph.com/issues/62286
Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole.check_status(): Use log.exception

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1877 from ceph/sentry-ae

supervisor: Fix an AttributeError in reimage()

supervisor: Fix an AttributeError in reimage()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1859 from ceph/quiet-urllib3

Turn down logging for urllib3.util.retry

Merge pull request #1875 from ceph/reimage-errs-sentry

Report reimage failures to Sentry

Merge pull request #1876 from ceph/afa-sort

task.ansible.FailureAnalyzer: Sort failure items

task.ansible.FailureAnalyzer: Sort failure items

To reduce unecessary duplication in e.g. Sentry.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1874 from ceph/fix-fog-timeout

fog: Fix a connection timeout bug

Merge pull request #1873 from ceph/console-log

orchestra.console: Scope loggers to shortname

supervisor.reimage: Report failures to Sentry

Signed-off-by: Zack Cerza <zack@redhat.com>

Move Sentry reporting logic to utils

Signed-off-by: Zack Cerza <zack@redhat.com>

FOG._wait_for_ready(): Catch ConnectionErrors

Instead of just ConnectionResetErrors, which inherit from
ConnectionError

Signed-off-by: Zack Cerza <zack@redhat.com>

remote: Raise ConnectionError when appropriate

Instead of just Exception.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.console: Scope loggers to shortname

This will make reading console debug logging easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1871 from ceph/sentry-ansible

exceptions.AnsibleFailedError: Add fingerprint()

This will cause Sentry to group events by their failure reasons, rather
than lumping all AnsibleFailedErrors together

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1869 from dmick/wip-pexpect

orchestra/console: log output from pexpect commands

orchestra/console: log output from pexpect commands

in case anything weird is being noticed and communicated by
ipmitool, try to display anything it says

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1870 from ceph/pyyaml-fix

Pin PyYAML to fix CI breakage

Update pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Exclude PyYAML 5.4.0,5.4.1

See https://github.com/yaml/pyyaml/issues/601

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1866 from ceph/aa-fix

Merge pull request #1867 from ceph/keyscan-timout

misc.ssh_keyscan: Always retry, and retry more

We started seeing reimage failures with errors like:
"teuthology.exceptions.MaxWhileTries: 'ssh_keyscan $host' reached
maximum tries (6) after waiting for 5 seconds"

Let's be quite a bit more generous.

Signed-off-by: Zack Cerza <zack@redhat.com>

TestFailureAnalyzer: Add tests for dropped items

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: Drop malformed records

If host_obj is the wrong type, we won't be able to extract anything
useful from it. In these cases, we'll end up using the raw string as we
used to do.

Signed-off-by: Zack Cerza <zack@redhat.com>

Ansible._handle_failure: YAMLErrors are special

Return to treating them differently, but also continue to catch other
exceptions here.

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: Look for SSH errors

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: items -> values

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1865 from ceph/ansible-fail-tolerate-exceptions

Ansible._handle_failure: Catch all Exceptions

Ansible._handle_failure: Catch all Exceptions

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1864 from ceph/analyze-ansible

ansible: Try to summarize failure logs

The failure logs we capture are sometimes helpful, but are often too
long, too complex, and too noisy to understand. Sentry also struggles to
associate related failures because of the presence of unique data such
as timestamps and URLs.

While I don't see a quick and generic solution to this, there are
several common failure modes that can easily be summarized. This commit
begins that work by looking for errors caused by network outages.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_ansible: Use mock_open()

This provides a more complete interface than what we were constructing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1863 from ceph/fog-wfr-eoferror

FOG._wait_for_ready: Tolerate EOFError

FOG._wait_for_ready: Tolerate EOFError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1862 from ceph/retry-sentinel-connreset

Remote.reconnect(): Use a default timeout of 30s

And rewrite with safe_while.

Signed-off-by: Zack Cerza <zack@redhat.com>

FOG._wait_for_ready: Tolerate ConnectionResetError

Signed-off-by: Zack Cerza <zack@redhat.com>

contextutil: Remove leftover print statement

Looks like this was missed during PR submission/review in 8f8d05852

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1858 from ceph/exporter-restart

Merge pull request #1860 from ceph/disp-rc

dispatcher: Return the highest of the jobs' RCs

dispatcher: Return the highest of the jobs' RCs

This is so that ceph-devstack can report job failures

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1855 from ceph/ssh-ux

Improve error message when there is no SSH key

Turn down logging for urllib3.util.retry

This quiets messages like: "Converted retries value: 10 ->
Retry(total=10, connect=None, read=None, redirect=None, status=None)"

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Restart every 24h

A design limitation of prometheus-client's multiprocessing mode is that
each process creates files to store its own metrics; the exporter then
has to read each file, even if the process which created it is dead.

This results in request latency growing over time, to the point of
multiple seconds when the file count gets into the thousands. This
eventually results in prometheus failing to fetch, leaving gaps in our
data.

We can work around this by restarting at a regular interval; 24h seems
like a fine place to start.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1856 from ceph/fog-debug

fog: Add more debug logging

FOG._wait_for_ready(): Use instance logger

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher/supervisor: Set root logger level

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Add more debug logging

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1854 from ceph/bootstrap-c9s

Improve error message when there is no SSH key

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1853 from ceph/reimage-no-ctx

bootstrap: Tolerate a missing lsb_release

This fixes the lack of support for CentOS 9.Stream

Signed-off-by: Zack Cerza <zack@redhat.com>

provision: Avoid a possible AttributeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1846 from ceph/stdin-killer

teuthology/task/install: add stdin-killer helper

This helper tool runs commands which may or may not take data on stdin.
Like "daemon-helper", if stdin signals EOF, stdin-killer will kill the
command but only as a last resort. It forwards EOF to the command by
closing the command's stdin (pipe) and then waiting a configurable
amount of time for the command to gracefully exit.

Additionally, if stdout or stderr are hung up -- i.e. the ssh parent
process has terminated -- then stdin-killer also detects this and
initiates the graceful shutdown of the command. This is something
daemon-helper does not do.

In general, this tool is a superior replacement of the daemon-helper
tool because you can write to the command's stdin normally.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.cfg: install binary helpers

These are used by vstart_runner.py for local dev operations. Install
them so they are available in the virtualenv bin directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install: reorganize binary helpers

We intend to install these so move them into an appropriately named
directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1803 from jdurgin/wip-configurable-timeouts

Merge pull request #1851 from ceph/reimage-failures

fog: Verify reimaged machine OS

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1850 from ceph/unmask-unlock-response

Merge pull request #1849 from ceph/prom-reimage-results

exporter: Instrument node reimaging success/fail

lock.ops.unlock_one: Fail sooner on 403, with msg

In the case of e.g. owners values not matching on an unlock attempt, we
were exhausting all retries and failing to display the exact reason for
the unlock failure. We can simply break on 403 errors and let the rest
of the function do its thing.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: make timeouts configurable

This will help with the sepia lab, being able to increase these
temporarily to handle a new fog server that is sometimes exceeding the
hardcoded timeouts.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

contextutil: allow safe_while to use an explicit timeout

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

nuke: Fix an import issue

Signed-off-by: Zack Cerza <zack@redhat.com>

lock.ops.reimage_machines: Drop incorrect log msg

This message was being logged when the reimage started, not finished.

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Instrument node reimaging success/fail

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1847 from ceph/fog-timeout

fog: Increase timeout in wait_for_deploy_task()

test_fog: Fix up test for preceding commit "Increase timeout"

Signed-off-by: Dan Mick <dmick@redhat.com>

fog: Increase timeout in wait_for_deploy_task()

When too many reimaging ops are running concurrently, we're seeing
timeouts. This isn't a true fix, but should help things until we've got
one.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1843 from dmick/wip-kernel-sort

Merge pull request #1844 from cbodley/wip-install-copr

task: install supports enable_coprs array

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Zack Cerza <zcerza@redhat.com>

task: install supports enable_coprs array

enable the installation of packages in fedora copr repositories

Signed-off-by: Casey Bodley <cbodley@redhat.com>

task/kernel.py: sort installed kernels by version

rpm -q --last sorts by timestamp-of-install, which does not
necessarily correlate with "latest version".  sort -rV does.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1804 from ceph/dependabot/pip/wheel-0.38.1

build(deps): bump wheel from 0.36.2 to 0.38.1

build(deps): bump wheel from 0.36.2 to 0.38.1

Bumps [wheel](https://github.com/pypa/wheel) from 0.36.2 to 0.38.1.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.36.2...0.38.1)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1831 from kamoltat/wip-ksirivad-rerun-readme

teuthology-suite: --seed & --subset now also stored in teuthology.log, config.yaml and orig.config.yaml
Reviewed-by: Zack Cerza <zcerza@redhat.com>

Merge pull request #1834 from ceph/disp_ls_remote

worker.prep_job: Skip job if ls_remote fails

worker.prep_job: Skip job if ls_remote fails

This is preferable to letting the dispatcher die.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite/run.py: Added seed and subset to base_config

In addition to being stored in results.log
`--seed` and `--subset` are now also stored in:

`teuthlogy.log`, `config.yaml` and `orig.config.yaml`.

Fixes: https://tracker.ceph.com/issues/59300
Signed-off-by: Kamoltat <ksirivad@redhat.com>

teuthology-suite: Log errors & warnings if results.log is missing during --rerun

Notify the user if `results.log` is missing when
they issue a rerun.

Also, edited teuthology-suite doc
to inform the user about how `--rerun` by
default parse `--seed`, `--subset` and
`--no-nested-subset` from `results.log`
by default

Fixes: https://tracker.ceph.com/issues/59300
Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #1832 from ceph/deps

Merge pull request #1827 from ceph/tox-no-osp

Drop argparse as a requirement

It's part of the standard library.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements.txt: Add prometheus_client

This should have been added a couple PRs back.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1833 from ceph/gha-ubuntu-version

bootstrap: apt-get update before installing

Signed-off-by: Zack Cerza <zack@redhat.com>

.github: Explicitly define test matrix

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1824 from batrick/parallel-gzip

teuthology: do not compress tarballs when pulling dir

Where we use this, it's for pulling log files that are already
compressed. Do not waste time double compressing!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: give verbose gzip output

For future analysis.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: use medium compression

To speed things up.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: parallelize gzip

Our machines have lots of cores, use them!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1826 from ceph/job-time

tox: Don't run openstack by default

It's quite time-consuming, and we're not sure if it's in use at all.

Signed-off-by: Zack Cerza <zack@redhat.com>

sueprvisor: Do not instrument certain job times

This should only really include first/last-in-suite jobs.

Signed-off-by: Zack Cerza <zack@redhat.com>