suite.util.find_git_parents: Fix KeyError

And clean up the code a bit

Signed-off-by: Zack Cerza <zack@redhat.com>

suite.Run.schedule_jobs: Unmask error in dry-run

For whatever reason we were suppressing an error related to unavailable
packages.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1953 from ceph/newer-oses

Various updates to support newer OSes

opsys: Tweak OS.__eq__ to work with CentOS Stream

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Simplify installed OS verification

Signed-off-by: Zack Cerza <zack@redhat.com>

opsys: Handle CentOS Stream more appropriately

This fixes an inconsistency where we'd change "centosX.stream" to "centosX" when
a job starts, submitting that to paddles.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode_start.sh: Detect operating system

... instead of hardcoding, which is more error-prone.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Bump testnode image to 22.04

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Fix locale

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Pin teuthology to Ubuntu 22.04

Until it can get updated.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Look for CentOS X.Stream images

When attempting to provision a CentOS node, if the user leaves out ".stream",
try searching for that as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Don't force English locale

Signed-off-by: Zack Cerza <zack@redhat.com>

containers/teuthology-dev: Update to noble

Signed-off-by: Zack Cerza <zack@redhat.com>

opsys: Add codenames for noble & centos 10

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Add test for parsing Centos 9 os_release

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1951 into main

* refs/pull/1951/head:
teuthology/suite: use valid subset default for rerun without --subset

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1944 from cbodley/wip-66096

suite: check build_complete for centos9

teuthology/suite/test: update unit tests to reflect c9

Signed-off-by: Laura <lflores@redhat.com>

suite: check build_complete for centos9

with all ceph releases moving to centos 9-based containers, teuthology
should prevent scheduling jobs until those container builds finish

update the DEFAULT_OS_VERSION for centos to 9 so it checks for those
containers when no --distro-version is specified

Fixes: https://tracker.ceph.com/issues/66096
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1950 from rishabh-d-dave/bootstrap-fedora-fix

boostrap: expect "FedoraLinux" along with "Fedora"

boostrap: expect "FedoraLinux" along with "Fedora"

On Fedora 38 -

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"
ID=fedora
VERSION_ID=38

Also see: https://fedoraproject.org/wiki/Changes/Fedora_Linux_in_os-release.

With this "./boostrap install" works fine on Fedora 38.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

teuthology/suite: use valid subset default for rerun without --subset

Fixes: https://tracker.ceph.com/issues/66302
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1948 from ceph/py312-centos

bootstrap: Improve Python version detection; default to 3.12 on CentOS & Fedora

Merge pull request #1942 from VallariAg/fix-install-task

task/install: Fix AssertionError

bootstrap: Default to py3.12 on CentOS & Fedora

Fixes: https://tracker.ceph.com/issues/66178
Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Improve Python version detection

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1947 from ceph/wip-pynose

replace deprecated nose dependency with pynose

Reviewed-by: Zack Cerza <zack@redhat.com>

Merge pull request #1946 from batrick/paddles-rerun

teuthology/suite: use paddles solely for rerun info

replace deprecated nose dependency with pynose

after upgrading teuthology to centos 9 and python 3.10, the rgw
multisite test suite now fails to invoke nosetests with:

> AttributeError: module 'collections' has no attribute 'Callable'

this is a known issue tracked in https://github.com/nose-devs/nose/issues/1127

Shilpa found a new 'pynose' package which claims that:

> This version of nose is compatible with Python 3.7+ (including 3.13+).

replace the nose dependency that i initially added in
https://github.com/ceph/teuthology/pull/1069 with the current version
1.5.1 of pynose

to prepare this pr, i first changed setup.cfg then ran:

> ./update-requirements.sh -P "pynose==1.5.1"

Fixes: https://tracker.ceph.com/issues/66183
Signed-off-by: Casey Bodley <cbodley@redhat.com>

teuthology/suite: use paddles solely for rerun info

teuthology.front no longer allows access to the run directories for general
users. In fact, it's not required because we are already getting run
information from paddles.

This commit makes it more stringent in --rerun handling: if
seed/subset/no_nested_subset do not match, exit failure rather than try to
continue. Without a subset, many suites are so large that teuthology-suite will
run for several minutes before ultimately failing to schedule due to the
quantity of jobs. It also doesn't make sense to rerun a suite when these values
do not match.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

task/install: fix AssertionError due to overrides

Fixes:
https://tracker.ceph.com/issues/66093
https://tracker.ceph.com/issues/66133

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge PR #1943 into main

* refs/pull/1943/head:
teuthology/task/install: conditionally merge extra_system_packages override

Reviewed-by: Vallari Agrawal <val.agl002@gmail.com>

teuthology/task/install: conditionally merge extra_system_packages override

Fixes: https://tracker.ceph.com/issues/66093
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1940 from ceph/py310

 Fix CI failures related to Ansible Galaxy

Merge pull request #1915 from ceph/rm-nuke

Removing nuke

Merge PR #1935 into main

* refs/pull/1935/head:
teuthology: modify logic to check for multiple completed builds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1941 from batrick/install-override-extra_system_packages

teuthology/task/install: apply extra_system_packages overrides

teuthology/task/install: apply extra_system_packages overrides

Currently the install task requires these to be defined only in the definition
of the install task itself. Overrides are ignored.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

docker-compose: Install locales-all for ansible

This commit originally included an ansible-core version update, but that was
moved to a different commit.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Bump ansible-core to 2.16.6

First, I updated the version in setup.cfg, then ran:
  ./update-requirements.sh -P "ansible-core==2.16.6"

Signed-off-by: Zack Cerza <zack@redhat.com>

Bump minimum Python version to 3.10

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology: modify logic to check for multiple completed builds

The current logic assumes that there is only one build for each distro/flavor
per SHA1. However, there is a bug in the jenkins infrastrucutre that sometimes
causes multiple builds to trigger for one SHA1. In many of these cases, the first
build succeeds, but the second fails. Teuthology only looks at the latest build,
notices that it failed, and gives up. However, with this logic, teuthology can
go back farther and notice that there is indeed a successful build earlier in the
lineup.

Here is an example in which the first centos 8 x86_64 build succeeded, but a second
build on top of it failed. Teuthology could only detect the latest failed build:
https://shaman.ceph.com/builds/ceph/wip-pdonnell-testing-20240503.010653-debug/ec1d3bd17a3db9d74296aa618f8d63c801bb647e/

Addresses this failure in teuthology:
lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-suite -v -m smithi -c wip-pdonnell-testing-20240503.010653-debug -s fs --subset 111/12000 -p 75 --dry-run
2024-05-03 16:39:35,231.231 INFO:teuthology.suite:Using random seed=9685
2024-05-03 16:39:35,232.232 INFO:teuthology.suite.run:kernel sha1: distro
2024-05-03 16:39:35,673.673 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci.git wip-pdonnell-testing-20240503.010653-debug -> ec1d3bd17a3db9d74296aa618f8d63c801bb647e
2024-05-03 16:39:35,673.673 INFO:teuthology.suite.run:ceph sha1: ec1d3bd17a3db9d74296aa618f8d63c801bb647e
2024-05-03 16:39:35,674.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ec1d3bd17a3db9d74296aa618f8d63c801bb647e
2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default
2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/8 arm64 default
2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/9 x86_64 crimson
2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/9 x86_64 default
2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/8 arm64 default
2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/8 x86_64 crimson
2024-05-03 16:39:36,177.177 DEBUG:teuthology.packaging:build: centos/8 x86_64 default
2024-05-03 16:39:36,178.178 INFO:teuthology.suite.util:Container build incomplete
Traceback (most recent call last):
  File "./teuthology/virtualenv/bin/teuthology-suite", line 8, in <module>
    sys.exit(main())
  File "/cephfs/home/lflores/teuthology/scripts/suite.py", line 226, in main
    return teuthology.suite.main(args)
  File "/cephfs/home/lflores/teuthology/teuthology/suite/__init__.py", line 143, in main
    run = Run(conf)
  File "/cephfs/home/lflores/teuthology/teuthology/suite/run.py", line 56, in __init__
    self.base_config = self.create_initial_config()
  File "/cephfs/home/lflores/teuthology/teuthology/suite/run.py", line 94, in create_initial_config
    self.choose_ceph_version(ceph_hash)
  File "/cephfs/home/lflores/teuthology/teuthology/suite/run.py", line 216, in choose_ceph_version
    util.schedule_fail(msg, self.name, dry_run=self.args.dry_run)
  File "/cephfs/home/lflores/teuthology/teuthology/suite/util.py", line 77, in schedule_fail
    raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling lflores-2024-05-03_16:39:35-fs-wip-pdonnell-testing-20240503.010653-debug-distro-default-smithi failed: Packages for os_type 'centos', flavor default and ceph hash 'ec1d3bd17a3db9d74296aa618f8d63c801bb647e' not found

More work should be done to fix the "double build" issue in jenkins, so this can be thought of as a workaround.

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge PR #1926 into main

* refs/pull/1926/head:
teuthology/suite: initialize lua prng using run's seed

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1932 from ceph/fix-test-admins-kill

kill: Fix test-admins' ability to kill

kill: Fix test-admins' ability to kill

By testing access to /bin/true, we were getting false negatives as we meant
to be testing for access to /bin/kill. With our configuration, any sudo access
indicates access to /bin/kill.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1929 from lxbsz/wip-64471

suite: add the kdb option support

Merge pull request #1930 from VallariAg/fix-cephadmunit-start

fix cephadmunit start() method

orchestra/daemon/cephadmunit.py: set is_started in start()

In CephadmUnit.start() method, `is_started` isn't set to
true. When running() is called after start(), then it would
return false, which is not correct since the daemon has been
started by calling start().
This commit fixes that issue.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

suite: add the kdb option support

This will allow us to disable the kdb when running the jobs.

URL: https://tracker.ceph.com/issues/64471
Signed-off-by: Xiubo Li <xiubli@redhat.com>

orchestra/daemon/cephadmunit.py: fix start() method

In CephadmUnit.start() method, explicitly pass start_cmd as
"args" keyword because Remote.run expects keyword arguments.

Fixes: https://tracker.ceph.com/issues/65162
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

teuthology/suite: initialize lua prng using run's seed

When a script may use Lua's prng, we want it to produce the same sequence
during a rerun.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1923 from ceph/poweron-timeout

orchestra.PhysicalConsole: Increase timeout

orchestra.PhysicalConsole: Increase timeout

... from 40s to 120s. A physical host being slightly slow to boot should not
cause a reimage failure.

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/docker-compose/testnode: Don't kill sshd

docker-compose exits non-zero if a container goes down while it's running.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1919 from ceph/wip-kernel-image-version

kernel: make get_image_version() work for rpm

Reviewed-by: Ramana Raja <rraja@redhat.com>

kernel: make get_image_version() work for rpm

At some point in the past, the layout of the rpm package has changed.
There is no file matching "/boot/vmlinuz-" there anymore, instead there
is "vmlinuz" file at the root of the modules directory.  For reference:

deb:

-rw-r--r-- root/root  11527168 2024-02-13 16:25 ./boot/vmlinuz-6.8.0-rc4-ga64ccd305b28
-rw-r--r-- root/root     72614 2024-02-13 16:25 ./lib/modules/6.8.0-rc4-ga64ccd305b28/modules.order

rpm:

/lib/modules/6.8.0-rc4-ga64ccd305b28/modules.order
/lib/modules/6.8.0-rc4-ga64ccd305b28/vmlinuz

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1918 from VallariAg/tapi-threads

teuthology/__init__.py: don't patch threads when running via teuthology_api

teuthology/__init__.py: don't patch threads when running via teuthology_api

The project [teuthology-api](https://github.com/ceph/teuthology-api)
requires threads to be not patched.
Currently, we are using "teuth-api" branch of teuthology where threads are
not patched. With this commit, we'll be able to use the "main" branch as
a dependency.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Add teuthology-node-cleanup command

This replaces teuthology-nuke --stale

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove nuke: Rework unlocking

This commit re-implements functionality that was removed with the nuke system.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove nuke: Trivial changes

This commit contains trivial changes like reference removals, docs changes, and
removal of dead code.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove nuke: deletions

This commit contains only full file deletions, and the relocation of
nuke.actions.clear_firewall() to nuke/__init__.py to retain compatibility with
older ceph.git tasks.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1914 from ceph/lock-leaks

supervisor: Disregard nuke-on-error when unlocking

test_exit: Drop bad test_noop

This test races with other tests because Exiter doesn't have a great way to
remove all installed handlers. This is a test-only issue, so we can drop this
test.

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Disregard nuke-on-error when unlocking

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1913 from ceph/wip-64193

supervisor: Do not nuke nodes after jobs finish

supervisor: Do not nuke nodes after jobs finish

This was causing a bad race condition, where we could unlock a node, then unlock
it again via the nuke process after a different job had locked it.

Fixes: https://tracker.ceph.com/issues/64193
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1912 from ceph/kill-report-dead

kill: After killing a run, report it as dead

kill: After killing a run, report it as dead

In case processes died a messy death.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1909 from ceph/deps

Update dependencies

JobProcesses: Ignore zombies safely

Signed-off-by: Zack Cerza <zack@redhat.com>

find_dispatcher_processes: Ignore zombies safely

Signed-off-by: Zack Cerza <zack@redhat.com>

Install ansible collections individually

Going forward, we can maintain our specific collection requirements in
requirements.yml.

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop ansible for ansible-core

The 'ansible' PyPI package installs _all_ collections, which ends up being
~60% the total size of our virtualenv.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Move some deps to setup.cfg

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: python_requires>=3.8

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils.fetch_repo: Use less retries

If a particular branch cannot successfully bootstrap, it can cause an accidental
DoS.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update via pip-compile -U

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: Pin urllib3 for botocore

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update pyjwt

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update paramiko

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update gevent

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update configobj

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update certifi

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update requests

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update PyYAML

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Move openstack to its own variant

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update cryptography

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update pip-tools

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1906 from ceph/kill-unbound

kill.kill_processes: Fix possibly-unbound variables

kill.kill_processes: Fix possibly-unbound variables

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1903 from ceph/wip-package-queries

suite: Improve package query caching

Merge pull request #1900 from ceph/systemd

Add systemd units for exporter and dispatcher

run.util.find_git_parents: Drop refresh()

This takes a long time, and can time out. The mirror is updated every ten
minutes automatically.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1896 from ceph/dependabot/pip/urllib3-1.26.18

build(deps): bump urllib3 from 1.26.6 to 1.26.18

build(deps): bump urllib3 from 1.26.6 to 1.26.18

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.6 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.6...1.26.18)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

run: Fix some pyright errors

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra.opsys: Add some newer OS codenames

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Remove some gitbuilder-related tests

Signed-off-by: Zack Cerza <zack@redhat.com>