Merge pull request #1902 from ceph/dispatcher-quiet

dispatcher: Dont spam the journal

Merge pull request #1792 from VallariAg/unittest-xml-scanner

orch/run: Add unit test xml scanner

util/scanner: add UnitTestScanner.num_of_total_failures

In UnitTestScanner's final error message, add total count of failures
before the first error occurance, like "(total x failed) <message>".
Another minor change: add "..." if the failure reason is more than 200 chars.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

add utils/tests/test_scanner.py

and test_run_unit_test in test_remote.py

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

add Scanner, UnitTestScanner, ValgrindScanner

1. add 'run_unit_test' to Remote
2. create util/scanner.py
3. new exception: UnitTestError
4. add `lxml` dependency in setup.cfg

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

supervisor: Drop job output

It gets logged to its own file in the job archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1901 from ceph/fog-debug-quieter

fog: Drop request debug logging

dispatcher: Drop supervisor output

It gets logged to its own file in the job archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Drop request debug logging

It's too noisy.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1892 from ceph/devstack-simplified

Add containers/teuthology-dev

Add containers/teuthology-dev

This is nearly identical to docs/docker-compose/teuthology, but with
some changes to better work with ceph-devstack. The bits in
docs/docker-compose should be able to be adapted easily to work with
this container.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1895 into main

* refs/pull/1895/head:
install/bin/stdin-killer: macOs (Darwin) compatibility

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1894 from VallariAg/fix-readthedocs-builds

fix readthedocs PR builds

readthedocs: fix 'The configuration key "build.image" is deprecated'

builds are failing because support for deprecated “build.image” is
fully removed by readthedocs, need to use "build.os" instead.

ref: https://blog.readthedocs.com/use-build-os-config/
error: https://readthedocs.org/projects/teuthology/builds/22250705/

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

install/bin/stdin-killer: macOs (Darwin) compatibility

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #1887 from ceph/paramiko-eoferror

orchestra: Tolerate EOFError during connect

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1888 from ceph/keyscan-tweak

misc._ssh_keyscan: Sort keys before returning any

ssh-keyscan's output is unsorted, so this function wasn't deterministic.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra: Move connection exception handling

... to inside the retry loop. Also, add an increment to the safe_while
instance we use.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra: Treat EOFError as SSHException

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1886 from ceph/update-paramiko

Merge pull request #1884 from kamoltat/wip-ksirivad-fix-62445

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log
Reviewed-by Zack Cerza <zcerza@redhat.com>

Merge pull request #1885 from ceph/fix-docs-build

Fix docs build
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Update paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

tox: Avoid buggy sphinx versions

See https://github.com/ceph/teuthology/pull/1884

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Drop license_file

It's deprecated in favor of `license_files`, but the default value is
sufficient.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log

Problem:

- confusing warning message stating that
the back trace is malformed

- We kept adding to the backtrace buffer
even when we exceeded the `MAX_BT_LINES`

Solution:

- Correct the warning message to be
"Ignoring backtrace that exceeds MAX_BT_LINES"
- reset the buffer once we exceeded MAX_BT_LINES
- Added some cases where we detect start/end of back trace.

Fixes:https://tracker.ceph.com/issues/62445

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1883 from ceph/nuke-desc-typeerror

nuke: Avoid a TypeError w/ null node description

nuke: Avoid a TypeError w/ null node description

This avoids a `TypeError: argument of type 'NoneType' is not iterable`
when nuking a node whose description is None.

ex: https://sentry.ceph.com/share/issue/91172146663f4c71a6cbfe43725b2e07/

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1882 from ceph/sentry-reimage-taskname

supervisor.reimage(): Improve Sentry reporting

supervisor.reimage(): Improve Sentry reporting

Set the `task` tag value to 'reimage' when reporting reimage failures to
Sentry, to make searching for them in its UI easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1881 from ceph/stdin-killer-setpgrp

stdin-killer: do not setpgrp is already leader

stdin-killer: do not setpgrp if already leader

Fixes failure like:

    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:Traceback (most recent call last):
    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:  File "/usr/bin/stdin-killer", line 213, in <module>
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:    os.setpgrp()
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:PermissionError: [Errno 1] Operation not permitted

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1880 from ceph/wip-62286

PhysicalConsole: Tolerate invalid UTF-8 characters

... in pexpect.spawn() calls.

Fixes: https://tracker.ceph.com/issues/62286
Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole.check_status(): Use log.exception

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1845 from NitzanMordhai/wip-nitzan-correct-typo-osd-default-pool-size

Merge pull request #1877 from ceph/sentry-ae

supervisor: Fix an AttributeError in reimage()

supervisor: Fix an AttributeError in reimage()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1859 from ceph/quiet-urllib3

Turn down logging for urllib3.util.retry

Merge pull request #1875 from ceph/reimage-errs-sentry

Report reimage failures to Sentry

Merge pull request #1876 from ceph/afa-sort

task.ansible.FailureAnalyzer: Sort failure items

task.ansible.FailureAnalyzer: Sort failure items

To reduce unecessary duplication in e.g. Sentry.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1874 from ceph/fix-fog-timeout

fog: Fix a connection timeout bug

Merge pull request #1873 from ceph/console-log

orchestra.console: Scope loggers to shortname

supervisor.reimage: Report failures to Sentry

Signed-off-by: Zack Cerza <zack@redhat.com>

Move Sentry reporting logic to utils

Signed-off-by: Zack Cerza <zack@redhat.com>

FOG._wait_for_ready(): Catch ConnectionErrors

Instead of just ConnectionResetErrors, which inherit from
ConnectionError

Signed-off-by: Zack Cerza <zack@redhat.com>

remote: Raise ConnectionError when appropriate

Instead of just Exception.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.console: Scope loggers to shortname

This will make reading console debug logging easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1871 from ceph/sentry-ansible

exceptions.AnsibleFailedError: Add fingerprint()

This will cause Sentry to group events by their failure reasons, rather
than lumping all AnsibleFailedErrors together

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1869 from dmick/wip-pexpect

orchestra/console: log output from pexpect commands

orchestra/console: log output from pexpect commands

in case anything weird is being noticed and communicated by
ipmitool, try to display anything it says

Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1870 from ceph/pyyaml-fix

Pin PyYAML to fix CI breakage

Update pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Exclude PyYAML 5.4.0,5.4.1

See https://github.com/yaml/pyyaml/issues/601

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1866 from ceph/aa-fix

Merge pull request #1867 from ceph/keyscan-timout

misc.ssh_keyscan: Always retry, and retry more

We started seeing reimage failures with errors like:
"teuthology.exceptions.MaxWhileTries: 'ssh_keyscan $host' reached
maximum tries (6) after waiting for 5 seconds"

Let's be quite a bit more generous.

Signed-off-by: Zack Cerza <zack@redhat.com>

TestFailureAnalyzer: Add tests for dropped items

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: Drop malformed records

If host_obj is the wrong type, we won't be able to extract anything
useful from it. In these cases, we'll end up using the raw string as we
used to do.

Signed-off-by: Zack Cerza <zack@redhat.com>

Ansible._handle_failure: YAMLErrors are special

Return to treating them differently, but also continue to catch other
exceptions here.

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: Look for SSH errors

Signed-off-by: Zack Cerza <zack@redhat.com>

ansible.FailureAnalyzer: items -> values

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1865 from ceph/ansible-fail-tolerate-exceptions

Ansible._handle_failure: Catch all Exceptions

Ansible._handle_failure: Catch all Exceptions

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1864 from ceph/analyze-ansible

ansible: Try to summarize failure logs

The failure logs we capture are sometimes helpful, but are often too
long, too complex, and too noisy to understand. Sentry also struggles to
associate related failures because of the presence of unique data such
as timestamps and URLs.

While I don't see a quick and generic solution to this, there are
several common failure modes that can easily be summarized. This commit
begins that work by looking for errors caused by network outages.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_ansible: Use mock_open()

This provides a more complete interface than what we were constructing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1863 from ceph/fog-wfr-eoferror

FOG._wait_for_ready: Tolerate EOFError

FOG._wait_for_ready: Tolerate EOFError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1862 from ceph/retry-sentinel-connreset

Remote.reconnect(): Use a default timeout of 30s

And rewrite with safe_while.

Signed-off-by: Zack Cerza <zack@redhat.com>

FOG._wait_for_ready: Tolerate ConnectionResetError

Signed-off-by: Zack Cerza <zack@redhat.com>

contextutil: Remove leftover print statement

Looks like this was missed during PR submission/review in 8f8d05852

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1858 from ceph/exporter-restart

Merge pull request #1860 from ceph/disp-rc

dispatcher: Return the highest of the jobs' RCs

dispatcher: Return the highest of the jobs' RCs

This is so that ceph-devstack can report job failures

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1855 from ceph/ssh-ux

Improve error message when there is no SSH key

Turn down logging for urllib3.util.retry

This quiets messages like: "Converted retries value: 10 ->
Retry(total=10, connect=None, read=None, redirect=None, status=None)"

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Restart every 24h

A design limitation of prometheus-client's multiprocessing mode is that
each process creates files to store its own metrics; the exporter then
has to read each file, even if the process which created it is dead.

This results in request latency growing over time, to the point of
multiple seconds when the file count gets into the thousands. This
eventually results in prometheus failing to fetch, leaving gaps in our
data.

We can work around this by restarting at a regular interval; 24h seems
like a fine place to start.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1856 from ceph/fog-debug

fog: Add more debug logging

FOG._wait_for_ready(): Use instance logger

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher/supervisor: Set root logger level

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Add more debug logging

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1854 from ceph/bootstrap-c9s

Improve error message when there is no SSH key

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1853 from ceph/reimage-no-ctx

bootstrap: Tolerate a missing lsb_release

This fixes the lack of support for CentOS 9.Stream

Signed-off-by: Zack Cerza <zack@redhat.com>

provision: Avoid a possible AttributeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1846 from ceph/stdin-killer

teuthology/task/install: add stdin-killer helper

This helper tool runs commands which may or may not take data on stdin.
Like "daemon-helper", if stdin signals EOF, stdin-killer will kill the
command but only as a last resort. It forwards EOF to the command by
closing the command's stdin (pipe) and then waiting a configurable
amount of time for the command to gracefully exit.

Additionally, if stdout or stderr are hung up -- i.e. the ssh parent
process has terminated -- then stdin-killer also detects this and
initiates the graceful shutdown of the command. This is something
daemon-helper does not do.

In general, this tool is a superior replacement of the daemon-helper
tool because you can write to the command's stdin normally.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

setup.cfg: install binary helpers

These are used by vstart_runner.py for local dev operations. Install
them so they are available in the virtualenv bin directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/task/install: reorganize binary helpers

We intend to install these so move them into an appropriately named
directory.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1803 from jdurgin/wip-configurable-timeouts

Merge pull request #1851 from ceph/reimage-failures

fog: Verify reimaged machine OS

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1850 from ceph/unmask-unlock-response

Merge pull request #1849 from ceph/prom-reimage-results

exporter: Instrument node reimaging success/fail