misc._ssh_keyscan: Sort keys before returning any

ssh-keyscan's output is unsorted, so this function wasn't deterministic.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1886 from ceph/update-paramiko

Merge pull request #1884 from kamoltat/wip-ksirivad-fix-62445

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log
Reviewed-by Zack Cerza <zcerza@redhat.com>

Merge pull request #1885 from ceph/fix-docs-build

Fix docs build
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Update paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

tox: Avoid buggy sphinx versions

See https://github.com/ceph/teuthology/pull/1884

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Drop license_file

It's deprecated in favor of `license_files`, but the default value is
sufficient.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log

Problem:

- confusing warning message stating that
the back trace is malformed

- We kept adding to the backtrace buffer
even when we exceeded the `MAX_BT_LINES`

Solution:

- Correct the warning message to be
"Ignoring backtrace that exceeds MAX_BT_LINES"
- reset the buffer once we exceeded MAX_BT_LINES
- Added some cases where we detect start/end of back trace.

Fixes:https://tracker.ceph.com/issues/62445

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1883 from ceph/nuke-desc-typeerror

nuke: Avoid a TypeError w/ null node description

nuke: Avoid a TypeError w/ null node description

This avoids a `TypeError: argument of type 'NoneType' is not iterable`
when nuking a node whose description is None.

ex: https://sentry.ceph.com/share/issue/91172146663f4c71a6cbfe43725b2e07/

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1882 from ceph/sentry-reimage-taskname

supervisor.reimage(): Improve Sentry reporting

supervisor.reimage(): Improve Sentry reporting

Set the `task` tag value to 'reimage' when reporting reimage failures to
Sentry, to make searching for them in its UI easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1881 from ceph/stdin-killer-setpgrp

stdin-killer: do not setpgrp is already leader

stdin-killer: do not setpgrp if already leader

Fixes failure like:

    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:Traceback (most recent call last):
    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:  File "/usr/bin/stdin-killer", line 213, in <module>
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:    os.setpgrp()
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:PermissionError: [Errno 1] Operation not permitted

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1880 from ceph/wip-62286

PhysicalConsole: Tolerate invalid UTF-8 characters

... in pexpect.spawn() calls.

Fixes: https://tracker.ceph.com/issues/62286
Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole.check_status(): Use log.exception

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1845 from NitzanMordhai/wip-nitzan-correct-typo-osd-default-pool-size

Merge pull request #1877 from ceph/sentry-ae

supervisor: Fix an AttributeError in reimage()

supervisor: Fix an AttributeError in reimage()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1859 from ceph/quiet-urllib3

Turn down logging for urllib3.util.retry

Merge pull request #1875 from ceph/reimage-errs-sentry

Report reimage failures to Sentry

Merge pull request #1876 from ceph/afa-sort

task.ansible.FailureAnalyzer: Sort failure items

task.ansible.FailureAnalyzer: Sort failure items

To reduce unecessary duplication in e.g. Sentry.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1874 from ceph/fix-fog-timeout

fog: Fix a connection timeout bug

Merge pull request #1873 from ceph/console-log

orchestra.console: Scope loggers to shortname

supervisor.reimage: Report failures to Sentry

Signed-off-by: Zack Cerza <zack@redhat.com>

Move Sentry reporting logic to utils

Signed-off-by: Zack Cerza <zack@redhat.com>

FOG._wait_for_ready(): Catch ConnectionErrors

Instead of just ConnectionResetErrors, which inherit from
ConnectionError

Signed-off-by: Zack Cerza <zack@redhat.com>

remote: Raise ConnectionError when appropriate

Instead of just Exception.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.console: Scope loggers to shortname

This will make reading console debug logging easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1871 from ceph/sentry-ansible

exceptions.AnsibleFailedError: Add fingerprint()

This will cause Sentry to group events by their failure reasons, rather
than lumping all AnsibleFailedErrors together

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1869 from dmick/wip-pexpect

orchestra/console: log output from pexpect commands

orchestra/console: log output from pexpect commands

in case anything weird is being noticed and communicated by
ipmitool, try to display anything it says

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1870 from ceph/pyyaml-fix

Pin PyYAML to fix CI breakage

Update pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Exclude PyYAML 5.4.0,5.4.1

See https://github.com/yaml/pyyaml/issues/601

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1866 from ceph/aa-fix

Merge pull request #1867 from ceph/keyscan-timout

misc.ssh_keyscan: Always retry, and retry more

We started seeing reimage failures with errors like:
"teuthology.exceptions.MaxWhileTries: 'ssh_keyscan $host' reached
maximum tries (6) after waiting for 5 seconds"

Let's be quite a bit more generous.

Signed-off-by: Zack Cerza <zack@redhat.com>

TestFailureAnalyzer: Add tests for dropped items

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: Drop malformed records

If host_obj is the wrong type, we won't be able to extract anything
useful from it. In these cases, we'll end up using the raw string as we
used to do.

Signed-off-by: Zack Cerza <zack@redhat.com>

Ansible._handle_failure: YAMLErrors are special

Return to treating them differently, but also continue to catch other
exceptions here.

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: Look for SSH errors

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: items -> values

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1865 from ceph/ansible-fail-tolerate-exceptions

Ansible._handle_failure: Catch all Exceptions

Ansible._handle_failure: Catch all Exceptions

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1864 from ceph/analyze-ansible

ansible: Try to summarize failure logs

The failure logs we capture are sometimes helpful, but are often too
long, too complex, and too noisy to understand. Sentry also struggles to
associate related failures because of the presence of unique data such
as timestamps and URLs.

While I don't see a quick and generic solution to this, there are
several common failure modes that can easily be summarized. This commit
begins that work by looking for errors caused by network outages.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_ansible: Use mock_open()

This provides a more complete interface than what we were constructing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1863 from ceph/fog-wfr-eoferror

FOG._wait_for_ready: Tolerate EOFError

FOG._wait_for_ready: Tolerate EOFError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1862 from ceph/retry-sentinel-connreset

Remote.reconnect(): Use a default timeout of 30s

And rewrite with safe_while.

Signed-off-by: Zack Cerza <zack@redhat.com>

FOG._wait_for_ready: Tolerate ConnectionResetError

Signed-off-by: Zack Cerza <zack@redhat.com>

contextutil: Remove leftover print statement

Looks like this was missed during PR submission/review in 8f8d05852

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1858 from ceph/exporter-restart

Merge pull request #1860 from ceph/disp-rc

dispatcher: Return the highest of the jobs' RCs

dispatcher: Return the highest of the jobs' RCs

This is so that ceph-devstack can report job failures

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1855 from ceph/ssh-ux

Improve error message when there is no SSH key

Turn down logging for urllib3.util.retry

This quiets messages like: "Converted retries value: 10 ->
Retry(total=10, connect=None, read=None, redirect=None, status=None)"

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Restart every 24h

A design limitation of prometheus-client's multiprocessing mode is that
each process creates files to store its own metrics; the exporter then
has to read each file, even if the process which created it is dead.

This results in request latency growing over time, to the point of
multiple seconds when the file count gets into the thousands. This
eventually results in prometheus failing to fetch, leaving gaps in our
data.

We can work around this by restarting at a regular interval; 24h seems
like a fine place to start.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1856 from ceph/fog-debug

fog: Add more debug logging

FOG._wait_for_ready(): Use instance logger

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher/supervisor: Set root logger level

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Add more debug logging

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1854 from ceph/bootstrap-c9s

Improve error message when there is no SSH key

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1853 from ceph/reimage-no-ctx

bootstrap: Tolerate a missing lsb_release

This fixes the lack of support for CentOS 9.Stream

Signed-off-by: Zack Cerza <zack@redhat.com>

provision: Avoid a possible AttributeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1846 from ceph/stdin-killer

teuthology/task/install: add stdin-killer helper

This helper tool runs commands which may or may not take data on stdin.
Like "daemon-helper", if stdin signals EOF, stdin-killer will kill the
command but only as a last resort. It forwards EOF to the command by
closing the command's stdin (pipe) and then waiting a configurable
amount of time for the command to gracefully exit.

Additionally, if stdout or stderr are hung up -- i.e. the ssh parent
process has terminated -- then stdin-killer also detects this and
initiates the graceful shutdown of the command. This is something
daemon-helper does not do.

In general, this tool is a superior replacement of the daemon-helper
tool because you can write to the command's stdin normally.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.cfg: install binary helpers

These are used by vstart_runner.py for local dev operations. Install
them so they are available in the virtualenv bin directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install: reorganize binary helpers

We intend to install these so move them into an appropriately named
directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1803 from jdurgin/wip-configurable-timeouts

Merge pull request #1851 from ceph/reimage-failures

fog: Verify reimaged machine OS

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1850 from ceph/unmask-unlock-response

Merge pull request #1849 from ceph/prom-reimage-results

exporter: Instrument node reimaging success/fail

lock.ops.unlock_one: Fail sooner on 403, with msg

In the case of e.g. owners values not matching on an unlock attempt, we
were exhausting all retries and failing to display the exact reason for
the unlock failure. We can simply break on 403 errors and let the rest
of the function do its thing.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: make timeouts configurable

This will help with the sepia lab, being able to increase these
temporarily to handle a new fog server that is sometimes exceeding the
hardcoded timeouts.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

contextutil: allow safe_while to use an explicit timeout

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

nuke: Fix an import issue

Signed-off-by: Zack Cerza <zack@redhat.com>

lock.ops.reimage_machines: Drop incorrect log msg

This message was being logged when the reimage started, not finished.

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Instrument node reimaging success/fail

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1847 from ceph/fog-timeout

fog: Increase timeout in wait_for_deploy_task()

test_fog: Fix up test for preceding commit "Increase timeout"

Signed-off-by: Dan Mick <dmick@redhat.com>

fog: Increase timeout in wait_for_deploy_task()

When too many reimaging ops are running concurrently, we're seeing
timeouts. This isn't a true fix, but should help things until we've got
one.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1843 from dmick/wip-kernel-sort

suite: fix typo osd pool default size
The 'osd pool default size' config had typo, we didn't took
it and actually ignored it until this fix.
Since we never use it and fixing the typo will cause all
suites to use it, I'll delete it completely and if we
want it in the future, lets fix it then.

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #1844 from cbodley/wip-install-copr

task: install supports enable_coprs array

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Zack Cerza <zcerza@redhat.com>

task: install supports enable_coprs array

enable the installation of packages in fedora copr repositories

Signed-off-by: Casey Bodley <cbodley@redhat.com>

task/kernel.py: sort installed kernels by version

rpm -q --last sorts by timestamp-of-install, which does not
necessarily correlate with "latest version".  sort -rV does.

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1804 from ceph/dependabot/pip/wheel-0.38.1

build(deps): bump wheel from 0.36.2 to 0.38.1

build(deps): bump wheel from 0.36.2 to 0.38.1

Bumps [wheel](https://github.com/pypa/wheel) from 0.36.2 to 0.38.1.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.36.2...0.38.1)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1831 from kamoltat/wip-ksirivad-rerun-readme

teuthology-suite: --seed & --subset now also stored in teuthology.log, config.yaml and orig.config.yaml
Reviewed-by: Zack Cerza <zcerza@redhat.com>

Merge pull request #1834 from ceph/disp_ls_remote

worker.prep_job: Skip job if ls_remote fails

worker.prep_job: Skip job if ls_remote fails

This is preferable to letting the dispatcher die.

Signed-off-by: Zack Cerza <zack@redhat.com>