teuthology/task/install: apply extra_system_packages overrides

Currently the install task requires these to be defined only in the definition
of the install task itself. Overrides are ignored.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #1926 into main

* refs/pull/1926/head:
teuthology/suite: initialize lua prng using run's seed

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1932 from ceph/fix-test-admins-kill

kill: Fix test-admins' ability to kill

kill: Fix test-admins' ability to kill

By testing access to /bin/true, we were getting false negatives as we meant
to be testing for access to /bin/kill. With our configuration, any sudo access
indicates access to /bin/kill.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1929 from lxbsz/wip-64471

suite: add the kdb option support

Merge pull request #1930 from VallariAg/fix-cephadmunit-start

fix cephadmunit start() method

orchestra/daemon/cephadmunit.py: set is_started in start()

In CephadmUnit.start() method, `is_started` isn't set to
true. When running() is called after start(), then it would
return false, which is not correct since the daemon has been
started by calling start().
This commit fixes that issue.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

suite: add the kdb option support

This will allow us to disable the kdb when running the jobs.

URL: https://tracker.ceph.com/issues/64471
Signed-off-by: Xiubo Li <xiubli@redhat.com>

orchestra/daemon/cephadmunit.py: fix start() method

In CephadmUnit.start() method, explicitly pass start_cmd as
"args" keyword because Remote.run expects keyword arguments.

Fixes: https://tracker.ceph.com/issues/65162
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

teuthology/suite: initialize lua prng using run's seed

When a script may use Lua's prng, we want it to produce the same sequence
during a rerun.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1923 from ceph/poweron-timeout

orchestra.PhysicalConsole: Increase timeout

orchestra.PhysicalConsole: Increase timeout

... from 40s to 120s. A physical host being slightly slow to boot should not
cause a reimage failure.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1919 from ceph/wip-kernel-image-version

kernel: make get_image_version() work for rpm

Reviewed-by: Ramana Raja <rraja@redhat.com>

kernel: make get_image_version() work for rpm

At some point in the past, the layout of the rpm package has changed.
There is no file matching "/boot/vmlinuz-" there anymore, instead there
is "vmlinuz" file at the root of the modules directory.  For reference:

deb:

-rw-r--r-- root/root  11527168 2024-02-13 16:25 ./boot/vmlinuz-6.8.0-rc4-ga64ccd305b28
-rw-r--r-- root/root     72614 2024-02-13 16:25 ./lib/modules/6.8.0-rc4-ga64ccd305b28/modules.order

rpm:

/lib/modules/6.8.0-rc4-ga64ccd305b28/modules.order
/lib/modules/6.8.0-rc4-ga64ccd305b28/vmlinuz

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1918 from VallariAg/tapi-threads

teuthology/__init__.py: don't patch threads when running via teuthology_api

teuthology/__init__.py: don't patch threads when running via teuthology_api

The project [teuthology-api](https://github.com/ceph/teuthology-api)
requires threads to be not patched.
Currently, we are using "teuth-api" branch of teuthology where threads are
not patched. With this commit, we'll be able to use the "main" branch as
a dependency.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #1914 from ceph/lock-leaks

supervisor: Disregard nuke-on-error when unlocking

test_exit: Drop bad test_noop

This test races with other tests because Exiter doesn't have a great way to
remove all installed handlers. This is a test-only issue, so we can drop this
test.

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Disregard nuke-on-error when unlocking

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1913 from ceph/wip-64193

supervisor: Do not nuke nodes after jobs finish

supervisor: Do not nuke nodes after jobs finish

This was causing a bad race condition, where we could unlock a node, then unlock
it again via the nuke process after a different job had locked it.

Fixes: https://tracker.ceph.com/issues/64193
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1912 from ceph/kill-report-dead

kill: After killing a run, report it as dead

kill: After killing a run, report it as dead

In case processes died a messy death.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1909 from ceph/deps

Update dependencies

JobProcesses: Ignore zombies safely

Signed-off-by: Zack Cerza <zack@redhat.com>

find_dispatcher_processes: Ignore zombies safely

Signed-off-by: Zack Cerza <zack@redhat.com>

Install ansible collections individually

Going forward, we can maintain our specific collection requirements in
requirements.yml.

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop ansible for ansible-core

The 'ansible' PyPI package installs _all_ collections, which ends up being
~60% the total size of our virtualenv.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Move some deps to setup.cfg

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: python_requires>=3.8

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils.fetch_repo: Use less retries

If a particular branch cannot successfully bootstrap, it can cause an accidental
DoS.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update via pip-compile -U

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Pin urllib3 for botocore

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update pyjwt

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update gevent

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update configobj

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update certifi

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update requests

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update PyYAML

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Move openstack to its own variant

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update cryptography

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1906 from ceph/kill-unbound

kill.kill_processes: Fix possibly-unbound variables

kill.kill_processes: Fix possibly-unbound variables

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1903 from ceph/wip-package-queries

suite: Improve package query caching

Merge pull request #1900 from ceph/systemd

Add systemd units for exporter and dispatcher

run.util.find_git_parents: Drop refresh()

This takes a long time, and can time out. The mirror is updated every ten
minutes automatically.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1896 from ceph/dependabot/pip/urllib3-1.26.18

build(deps): bump urllib3 from 1.26.6 to 1.26.18

build(deps): bump urllib3 from 1.26.6 to 1.26.18

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.6 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.6...1.26.18)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

run: Fix some pyright errors

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.opsys: Add some newer OS codenames

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Remove some gitbuilder-related tests

Signed-off-by: Zack Cerza <zack@redhat.com>

Make logs slightly quieter during scheduling

Particularly in non-verbose mode.

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils.ls_remote: Memoize

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Improve package query caching

We had our own "system" for caching, but it had the unfortunate characteristic
 of being a big bowl of spaghetti. While eating said pasta I also noticed we
had two competing "distro defaults" concepts - so that let me delete even more
code. Yum!

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1899 from ceph/kill-proc-perms

Merge pull request #1902 from ceph/dispatcher-quiet

dispatcher: Dont spam the journal

Merge pull request #1792 from VallariAg/unittest-xml-scanner

orch/run: Add unit test xml scanner

util/scanner: add UnitTestScanner.num_of_total_failures

In UnitTestScanner's final error message, add total count of failures
before the first error occurance, like "(total x failed) <message>".
Another minor change: add "..." if the failure reason is more than 200 chars.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

add utils/tests/test_scanner.py

and test_run_unit_test in test_remote.py

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

add Scanner, UnitTestScanner, ValgrindScanner

1. add 'run_unit_test' to Remote
2. create util/scanner.py
3. new exception: UnitTestError
4. add `lxml` dependency in setup.cfg

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

kill: Don't unlock nodes if killing procs fails

... so that we don't unlock nodes while their jobs are running.

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Drop job output

It gets logged to its own file in the job archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1901 from ceph/fog-debug-quieter

fog: Drop request debug logging

dispatcher: Drop supervisor output

It gets logged to its own file in the job archive.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Drop request debug logging

It's too noisy.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add systemd units for exporter and dispatcher

These are copies of what is currently in use in sepia.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1892 from ceph/devstack-simplified

Add containers/teuthology-dev

Add containers/teuthology-dev

This is nearly identical to docs/docker-compose/teuthology, but with
some changes to better work with ceph-devstack. The bits in
docs/docker-compose should be able to be adapted easily to work with
this container.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1895 into main

* refs/pull/1895/head:
install/bin/stdin-killer: macOs (Darwin) compatibility

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1894 from VallariAg/fix-readthedocs-builds

fix readthedocs PR builds

readthedocs: fix 'The configuration key "build.image" is deprecated'

builds are failing because support for deprecated “build.image” is
fully removed by readthedocs, need to use "build.os" instead.

ref: https://blog.readthedocs.com/use-build-os-config/
error: https://readthedocs.org/projects/teuthology/builds/22250705/

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

install/bin/stdin-killer: macOs (Darwin) compatibility

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #1887 from ceph/paramiko-eoferror

orchestra: Tolerate EOFError during connect

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1888 from ceph/keyscan-tweak

misc._ssh_keyscan: Sort keys before returning any

ssh-keyscan's output is unsorted, so this function wasn't deterministic.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra: Move connection exception handling

... to inside the retry loop. Also, add an increment to the safe_while
instance we use.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra: Treat EOFError as SSHException

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1886 from ceph/update-paramiko

Merge pull request #1884 from kamoltat/wip-ksirivad-fix-62445

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log
Reviewed-by Zack Cerza <zcerza@redhat.com>

Merge pull request #1885 from ceph/fix-docs-build

Fix docs build
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Update paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

tox: Avoid buggy sphinx versions

See https://github.com/ceph/teuthology/pull/1884

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Drop license_file

It's deprecated in favor of `license_files`, but the default value is
sufficient.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/scrape: Fix bad backtrace parsing in Teuthology.log

Problem:

- confusing warning message stating that
the back trace is malformed

- We kept adding to the backtrace buffer
even when we exceeded the `MAX_BT_LINES`

Solution:

- Correct the warning message to be
"Ignoring backtrace that exceeds MAX_BT_LINES"
- reset the buffer once we exceeded MAX_BT_LINES
- Added some cases where we detect start/end of back trace.

Fixes:https://tracker.ceph.com/issues/62445

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1883 from ceph/nuke-desc-typeerror

nuke: Avoid a TypeError w/ null node description

nuke: Avoid a TypeError w/ null node description

This avoids a `TypeError: argument of type 'NoneType' is not iterable`
when nuking a node whose description is None.

ex: https://sentry.ceph.com/share/issue/91172146663f4c71a6cbfe43725b2e07/

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1882 from ceph/sentry-reimage-taskname

supervisor.reimage(): Improve Sentry reporting

supervisor.reimage(): Improve Sentry reporting

Set the `task` tag value to 'reimage' when reporting reimage failures to
Sentry, to make searching for them in its UI easier.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1881 from ceph/stdin-killer-setpgrp

stdin-killer: do not setpgrp is already leader

stdin-killer: do not setpgrp if already leader

Fixes failure like:

    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:Traceback (most recent call last):
    2023-08-03T19:40:10.942 INFO:teuthology.orchestra.run.smithi100.stderr:  File "/usr/bin/stdin-killer", line 213, in <module>
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:    os.setpgrp()
    2023-08-03T19:40:10.943 INFO:teuthology.orchestra.run.smithi100.stderr:PermissionError: [Errno 1] Operation not permitted

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1880 from ceph/wip-62286

PhysicalConsole: Tolerate invalid UTF-8 characters

... in pexpect.spawn() calls.

Fixes: https://tracker.ceph.com/issues/62286
Signed-off-by: Zack Cerza <zack@redhat.com>

PhysicalConsole.check_status(): Use log.exception

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1845 from NitzanMordhai/wip-nitzan-correct-typo-osd-default-pool-size

Merge pull request #1877 from ceph/sentry-ae

supervisor: Fix an AttributeError in reimage()

supervisor: Fix an AttributeError in reimage()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1859 from ceph/quiet-urllib3

Turn down logging for urllib3.util.retry