Add teuthology-node-cleanup command

This replaces teuthology-nuke --stale

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove nuke: Rework unlocking

This commit re-implements functionality that was removed with the nuke system.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove nuke: Trivial changes

This commit contains trivial changes like reference removals, docs changes, and
removal of dead code.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove nuke: deletions

This commit contains only full file deletions, and the relocation of
nuke.actions.clear_firewall() to nuke/__init__.py to retain compatibility with
older ceph.git tasks.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1914 from ceph/lock-leaks

supervisor: Disregard nuke-on-error when unlocking

test_exit: Drop bad test_noop

This test races with other tests because Exiter doesn't have a great way to
remove all installed handlers. This is a test-only issue, so we can drop this
test.

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Disregard nuke-on-error when unlocking

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1913 from ceph/wip-64193

supervisor: Do not nuke nodes after jobs finish

supervisor: Do not nuke nodes after jobs finish

This was causing a bad race condition, where we could unlock a node, then unlock
it again via the nuke process after a different job had locked it.

Fixes: https://tracker.ceph.com/issues/64193
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1912 from ceph/kill-report-dead

kill: After killing a run, report it as dead

kill: After killing a run, report it as dead

In case processes died a messy death.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1909 from ceph/deps

Update dependencies

JobProcesses: Ignore zombies safely

Signed-off-by: Zack Cerza <zack@redhat.com>

find_dispatcher_processes: Ignore zombies safely

Signed-off-by: Zack Cerza <zack@redhat.com>

Install ansible collections individually

Going forward, we can maintain our specific collection requirements in
requirements.yml.

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop ansible for ansible-core

The 'ansible' PyPI package installs _all_ collections, which ends up being
~60% the total size of our virtualenv.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Move some deps to setup.cfg

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: python_requires>=3.8

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils.fetch_repo: Use less retries

If a particular branch cannot successfully bootstrap, it can cause an accidental
DoS.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update via pip-compile -U

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Pin urllib3 for botocore

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update pyjwt

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update gevent

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update configobj

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update certifi

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update requests

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update PyYAML

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Move openstack to its own variant

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update cryptography

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1906 from ceph/kill-unbound

kill.kill_processes: Fix possibly-unbound variables

kill.kill_processes: Fix possibly-unbound variables

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1903 from ceph/wip-package-queries

suite: Improve package query caching

Merge pull request #1900 from ceph/systemd

Add systemd units for exporter and dispatcher

run.util.find_git_parents: Drop refresh()

This takes a long time, and can time out. The mirror is updated every ten
minutes automatically.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1896 from ceph/dependabot/pip/urllib3-1.26.18

build(deps): bump urllib3 from 1.26.6 to 1.26.18

build(deps): bump urllib3 from 1.26.6 to 1.26.18

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.6 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.6...1.26.18)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

run: Fix some pyright errors

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.opsys: Add some newer OS codenames

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Remove some gitbuilder-related tests

Signed-off-by: Zack Cerza <zack@redhat.com>

Make logs slightly quieter during scheduling

Particularly in non-verbose mode.

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils.ls_remote: Memoize

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Improve package query caching

We had our own "system" for caching, but it had the unfortunate characteristic
 of being a big bowl of spaghetti. While eating said pasta I also noticed we
had two competing "distro defaults" concepts - so that let me delete even more
code. Yum!

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1899 from ceph/kill-proc-perms

Merge pull request #1902 from ceph/dispatcher-quiet

dispatcher: Dont spam the journal

Merge pull request #1792 from VallariAg/unittest-xml-scanner

orch/run: Add unit test xml scanner

util/scanner: add UnitTestScanner.num_of_total_failures

In UnitTestScanner's final error message, add total count of failures
before the first error occurance, like "(total x failed) <message>".
Another minor change: add "..." if the failure reason is more than 200 chars.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

add utils/tests/test_scanner.py

and test_run_unit_test in test_remote.py

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

add Scanner, UnitTestScanner, ValgrindScanner

1. add 'run_unit_test' to Remote
2. create util/scanner.py
3. new exception: UnitTestError
4. add `lxml` dependency in setup.cfg

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

kill: Don't unlock nodes if killing procs fails

... so that we don't unlock nodes while their jobs are running.

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Drop job output

It gets logged to its own file in the job archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1901 from ceph/fog-debug-quieter

fog: Drop request debug logging

dispatcher: Drop supervisor output

It gets logged to its own file in the job archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Drop request debug logging

It's too noisy.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add systemd units for exporter and dispatcher

These are copies of what is currently in use in sepia.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1892 from ceph/devstack-simplified

Add containers/teuthology-dev

Add containers/teuthology-dev

This is nearly identical to docs/docker-compose/teuthology, but with
some changes to better work with ceph-devstack. The bits in
docs/docker-compose should be able to be adapted easily to work with
this container.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1895 into main

* refs/pull/1895/head:
install/bin/stdin-killer: macOs (Darwin) compatibility

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1894 from VallariAg/fix-readthedocs-builds

fix readthedocs PR builds

readthedocs: fix 'The configuration key "build.image" is deprecated'

builds are failing because support for deprecated “build.image” is
fully removed by readthedocs, need to use "build.os" instead.

ref: https://blog.readthedocs.com/use-build-os-config/
error: https://readthedocs.org/projects/teuthology/builds/22250705/

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

install/bin/stdin-killer: macOs (Darwin) compatibility

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #1887 from ceph/paramiko-eoferror

orchestra: Tolerate EOFError during connect

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1888 from ceph/keyscan-tweak

misc._ssh_keyscan: Sort keys before returning any

ssh-keyscan's output is unsorted, so this function wasn't deterministic.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra: Move connection exception handling

... to inside the retry loop. Also, add an increment to the safe_while
instance we use.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra: Treat EOFError as SSHException

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1886 from ceph/update-paramiko

Merge pull request #1884 from kamoltat/wip-ksirivad-fix-62445

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log
Reviewed-by Zack Cerza <zcerza@redhat.com>

Merge pull request #1885 from ceph/fix-docs-build

Fix docs build
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Update paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

tox: Avoid buggy sphinx versions

See https://github.com/ceph/teuthology/pull/1884

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Drop license_file

It's deprecated in favor of `license_files`, but the default value is
sufficient.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log

Problem:

- confusing warning message stating that
the back trace is malformed

- We kept adding to the backtrace buffer
even when we exceeded the `MAX_BT_LINES`

Solution:

- Correct the warning message to be
"Ignoring backtrace that exceeds MAX_BT_LINES"
- reset the buffer once we exceeded MAX_BT_LINES
- Added some cases where we detect start/end of back trace.

Fixes:https://tracker.ceph.com/issues/62445

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1883 from ceph/nuke-desc-typeerror

nuke: Avoid a TypeError w/ null node description

nuke: Avoid a TypeError w/ null node description

This avoids a `TypeError: argument of type 'NoneType' is not iterable`
when nuking a node whose description is None.

ex: https://sentry.ceph.com/share/issue/91172146663f4c71a6cbfe43725b2e07/

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1882 from ceph/sentry-reimage-taskname

supervisor.reimage(): Improve Sentry reporting

supervisor.reimage(): Improve Sentry reporting

Set the `task` tag value to 'reimage' when reporting reimage failures to
Sentry, to make searching for them in its UI easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1881 from ceph/stdin-killer-setpgrp

stdin-killer: do not setpgrp is already leader

stdin-killer: do not setpgrp if already leader

Fixes failure like:

    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:Traceback (most recent call last):
    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:  File "/usr/bin/stdin-killer", line 213, in <module>
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:    os.setpgrp()
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:PermissionError: [Errno 1] Operation not permitted

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1880 from ceph/wip-62286

PhysicalConsole: Tolerate invalid UTF-8 characters

... in pexpect.spawn() calls.

Fixes: https://tracker.ceph.com/issues/62286
Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole.check_status(): Use log.exception

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1845 from NitzanMordhai/wip-nitzan-correct-typo-osd-default-pool-size

Merge pull request #1877 from ceph/sentry-ae

supervisor: Fix an AttributeError in reimage()

supervisor: Fix an AttributeError in reimage()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1859 from ceph/quiet-urllib3

Turn down logging for urllib3.util.retry

Merge pull request #1875 from ceph/reimage-errs-sentry

Report reimage failures to Sentry

Merge pull request #1876 from ceph/afa-sort

task.ansible.FailureAnalyzer: Sort failure items

task.ansible.FailureAnalyzer: Sort failure items

To reduce unecessary duplication in e.g. Sentry.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1874 from ceph/fix-fog-timeout

fog: Fix a connection timeout bug

Merge pull request #1873 from ceph/console-log

orchestra.console: Scope loggers to shortname

supervisor.reimage: Report failures to Sentry

Signed-off-by: Zack Cerza <zack@redhat.com>

Move Sentry reporting logic to utils

Signed-off-by: Zack Cerza <zack@redhat.com>

FOG._wait_for_ready(): Catch ConnectionErrors

Instead of just ConnectionResetErrors, which inherit from
ConnectionError

Signed-off-by: Zack Cerza <zack@redhat.com>

remote: Raise ConnectionError when appropriate

Instead of just Exception.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.console: Scope loggers to shortname

This will make reading console debug logging easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1871 from ceph/sentry-ansible

exceptions.AnsibleFailedError: Add fingerprint()

This will cause Sentry to group events by their failure reasons, rather
than lumping all AnsibleFailedErrors together

Signed-off-by: Zack Cerza <zack@redhat.com>