timer: use datetime.timezone.utc for python 3.10

Python 3.10 doesn't have datetime.UTC yet as Python 3.12 does

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/ssh_keys: fix invalid escape sequence '\w'

Fix syntax warning message:

    <unknown>:68: SyntaxWarning: invalid escape sequence '\w'
    teuthology/task/ssh_keys.py:57: SyntaxWarning: invalid escape sequence '\w'
      match = re.match('[\w-]+ {key} \S+@\S+'.format(key=re.escape(ssh_key)), line_to_test)
    teuthology/task/ssh_keys.py:68: SyntaxWarning: invalid escape sequence '\w'
      match = re.match('[\w-]+ \S+ {username}@\S+'.format(username=username), line_to_test)

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/mpi: fix invalid escape sequence

    teuthology/task/mpi.py:21: SyntaxWarning: invalid escape sequence '\s'
      version = re.search("^\s+Version:\s+(.+)$", version_str, re.MULTILINE).group(1)

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

util/scanner: get rid of FutureWarning message

Get rid of the warning:

    teuthology/util/test/test_scanner.py::TestValgrindScanner::test_scan_all_files
      /Users/kyr/kshtsk/teuthology/teuthology/util/scanner.py:133: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
        if not xml_tree:

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

timer: fix deprecation warning for utcfromtimestamp

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

lock/cli: fix invalid escape sequence

Address warning message which is going to be an error in future
python versions:

  SyntaxWarning: invalid escape sequence '\w'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/kernel: fix invalid escape sequence

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1988 from kshtsk/wip-github-checkout-v4

workflows: use latest checkout and setup-python actions

workflows: use latest checkout and setup-python actions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1985 from kshtsk/wip-access-denied

Fix access denied from psutil on macOS

kill: handle AccessDenied exception

Fix unhandled exception psutilAccessDenied in process_matches_run().

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

dispatcher: fix AccessDenied on process lookup

On macOS dispatcher while trying to go through process list gets stuck
on some of the system processes like launchd, logd, systemstats, etc.
and quites unexpectedly with PermissionError and psutil.AccessDenied
exceptions.

Fixes: https://tracker.ceph.com/issues/67313
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1987 from ceph/docker-compose

Migrate to docker compose v2

Merge pull request #1982 from ceph/rm-worker

Remove worker.py

Migrate to docker compose v2

https://github.com/actions/runner-images/issues/9692
https://docs.docker.com/compose/migrate/

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove worker.py

Originally, #1960 was intended to do this, but in some review back-and-forth the
commit removing the file was dropped. Let's actually remove the file.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1960 from ceph/worker-supervisor

Finish removing teuthology-worker

dispatcher: Temporarily pass through to supervisor

The old dispatcher expects to be able to invoke the supervisor via
`teuthology-dispatcher --supervisor`, so add this compatibility shim for the
time being.

Signed-off-by: Zack Cerza <zack@redhat.com>

Finish removing teuthology-worker

The dispatcher and supervisor were added in #1546, but code was copied and
pasted into the new modules, leaving the worker untouched. Also untouched were
the unit tests, meaning that the dispatcher and supervisor were never unit
tested. As the copied code changed, the dispatcher and supervisor were not being
tested for regressions, while the worker - which wasn't being anymore - had
passing unit tests, giving some false sense of security.

This commit removes the old worker code, and adapts the old worker tests to
apply to the dispatcher and supervisor. It also splits out teuthology-supervisor
into its own command.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1977 from ceph/utcnow

Replace deprecated datetime.utcnow() calls

Replace deprecated datetime.utcnow() calls

See https://blog.miguelgrinberg.com/post/it-s-time-for-a-change-datetime-utcnow-is-now-deprecated

Signed-off-by: Zack Cerza <zack@redhat.com>

ci: Test on Ubuntu 24.04 with python3.12

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1975 from ceph/test-debug

Get the 'tests' task working again

Merge pull request #1980 from ceph/kill-job-paddles-data

kill.kill_job: Get job data from paddles

kill.kill_job: Get job data from paddles

... if we can't get it from the filesystem. This is necessary because of the
teuthology archive's restricted permissions on teuthology.front.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1979 from ceph/issue-66893-fix

find_git_parents: Fix githelper URL template

find_git_parents: Fix githelper URL template

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1976 from ceph/issue-66893

find_git_parents: Refresh mirror when required

Merge pull request #1974 from ceph/wip-selinux-ignore-agetty

teuthology/task/selinux: ignore denials in agetty

find_git_parents: Refresh mirror when required

We used to do the refresh every time, which wasn't good. Then we didn't do it at
all, which was fine until someone tried to schedule against a commit in the few
minutes before the mirror picked it up. Let's try doing it when necessary :)

Fixes: #66893
Signed-off-by: Zack Cerza <zack@redhat.com>

test_report: Use module-level tests and fixtures

Signed-off-by: Zack Cerza <zack@redhat.com>

Task.__init__: ctx param is mandatory

This reflects reality and also fixes some linter errors.

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.deep_merge: Rewrite for readability

Partially enabled by not passing MappingProxyType to it any longer.

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.deep_merge,merge_configs: Add type hinting

Signed-off-by: Zack Cerza <zack@redhat.com>

run: Use job config's archive dir if specified

Signed-off-by: Zack Cerza <zack@redhat.com>

Run.report_outcome: Drop fake_ctx

Signed-off-by: Zack Cerza <zack@redhat.com>

test_fog: fix test conflict with local config

Signed-off-by: Zack Cerza <zack@redhat.com>

test_report: Use a tmp_path fixture

Signed-off-by: Zack Cerza <zack@redhat.com>

test_ansible: Set ctx.archive to empty string

Scheduled jobs are failing because they try to write to
/home/ansible.log.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Rewrite

This task has been broken for a very long time. It hasn't passed in years, and
has additionally been masking test failures including some regressions.

This commit:
* Fixes masking of failures in setup/teardown methods
* Actually runs all of our tests
* Excludes tests that need remote nodes when they are not available
* Improves logging and reporting
* Adds support for running in CLI mode

Signed-off-by: Zack Cerza <zack@redhat.com>

Move LoggerFile to teuthology.util

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/task/selinux: ignore denials in agetty

This was tracked in [1], but the addition of the context to the
allowlist in ceph.git commit 7a6389272aff ("qa: ignore container
checkpoint/restore related selinux denials for centos9") doesn't
cover krbd suite which doesn't pull in any distro snippets.

Since this denial has nothing to do with Ceph [2], let's ignore it
globally.

[1] http://tracker.ceph.com/issues/64616
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2259622

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1921 from phlogistonjohn/jjm-ssh-config-hostname

orchestra/connection: ssh configuration options

Merge pull request #1972 from VallariAg/fix-teuthology-kill

teuthology/kill.py: Fix "TypeError: 'NoneType' object is not iterable"

Create teuthology/test/test_kill.py

Add TestFindTargets in test_kill for different
test cases of find_targets function.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #1898 from fazledyn-or/Fix_Improper_Method_Call

Fixed Improper Method Call: Replaced `mktemp`

Merge pull request #1969 from kshtsk/wip-alma-rocky

bootstrap: add AlmaLinux and RockyLinux case

Merge branch 'ceph:main' into Fix_Improper_Method_Call

Merge pull request #1965 from ceph/dclo

.github: Don't run dev_container on PRs or tags

Merge pull request #1964 from ceph/bs-av-check

bootstrap: Fix ansible version check

Merge pull request #1967 from ceph/dependabot/pip/jinja2-3.1.4

build(deps): bump jinja2 from 3.1.2 to 3.1.4

bootstrap: add AlmaLinux and RockyLinux case

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

build(deps): bump jinja2 from 3.1.2 to 3.1.4

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.2 to 3.1.4.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.2...3.1.4)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1962 from kshtsk/wip-opensuse-ce77b04

bootstrap: Tolerate a missing lsb_release

.github: Don't run dev_container on PRs or tags

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Fix ansible version check

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1961 from VallariAg/add-daemon-finished

orchestra/daemon: add finished() method in DaemonState and CephadmUnit

requirements.txt: pump lupa from 2.0 to 2.2

Fixes: https://tracker.ceph.com/issues/66768
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

bootstrap: use python3.12 now for SUSE based repo

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

bootstrap: Tolerate a missing lsb_release

Add missing keywords for openSUSE Leap and Tumbleweed

Fixes: ce77b04aabe8e17e6bd70ad5812233b5772c50bc
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

orchestra/daemon/cephadmunit.py: fix method kill_cmd()

This commit changes kill command from docker to systemctl.
This is because services are managed by systemd, not docker.
Also fixes a typo in service name.

Fixes: https://tracker.ceph.com/issues/66742
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

orchestra/daemon: add finished() method in DaemonState and CephadmUnit

These are to be used by DaemonWatchdog to determine if
DaemonState/CephadmUnit daemons are active or finished.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

teuthology/kill.py: Fix "TypeError: 'NoneType' object is not iterable"

Fixes: https://tracker.ceph.com/issues/66440#note-1
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #1959 from ceph/kill-running

kill: Fix killing by owner

Merge pull request #1954 from ceph/dev-container

.github: Add workflow to build dev container

kill: Fix killing by owner

The list comprehension was incorrect.
Fixes: https://tracker.ceph.com/issues/66440
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1949 from ceph/nodecleanup

Fixes for teuthology-node-cleanup
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1955 from ceph/ipmi

console: Rework power on/off retry mechanism

lock.ops.unlock_safe: Remove stray comment

Signed-off-by: Zack Cerza <zack@redhat.com>

node_cleanup: Fix invocations without --owner

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1958 from ceph/suite-dryrun-pkgs

suite.Run.schedule_jobs: Unmask error in dry-run

suite.util.find_git_parents: Fix KeyError

And clean up the code a bit

Signed-off-by: Zack Cerza <zack@redhat.com>

suite.Run.schedule_jobs: Unmask error in dry-run

For whatever reason we were suppressing an error related to unavailable
packages.

Signed-off-by: Zack Cerza <zack@redhat.com>

.github: Add workflow to build dev container

It's multi-arch, too!

Signed-off-by: Zack Cerza <zack@redhat.com>

console: Rework power on/off retry mechanism

We were only ever issuing the inital request once; Now, after five unsuccessful
checks to see that the node is in the desired state, we re-issue the request but
do not reset the clock.
We were also checking the power state immediately after sending the request,
which is unhelpful as the operation will never complete that quickly. Now, we
sleep at least once.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1953 from ceph/newer-oses

Various updates to support newer OSes

opsys: Tweak OS.__eq__ to work with CentOS Stream

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Simplify installed OS verification

Signed-off-by: Zack Cerza <zack@redhat.com>

opsys: Handle CentOS Stream more appropriately

This fixes an inconsistency where we'd change "centosX.stream" to "centosX" when
a job starts, submitting that to paddles.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode_start.sh: Detect operating system

... instead of hardcoding, which is more error-prone.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Bump testnode image to 22.04

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Fix locale

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Pin teuthology to Ubuntu 22.04

Until it can get updated.

Signed-off-by: Zack Cerza <zack@redhat.com>

fog: Look for CentOS X.Stream images

When attempting to provision a CentOS node, if the user leaves out ".stream",
try searching for that as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Don't force English locale

Signed-off-by: Zack Cerza <zack@redhat.com>

containers/teuthology-dev: Update to noble

Signed-off-by: Zack Cerza <zack@redhat.com>

opsys: Add codenames for noble & centos 10

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Add test for parsing Centos 9 os_release

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1951 into main

* refs/pull/1951/head:
teuthology/suite: use valid subset default for rerun without --subset

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1944 from cbodley/wip-66096

suite: check build_complete for centos9

teuthology/suite/test: update unit tests to reflect c9

Signed-off-by: Laura <lflores@redhat.com>

suite: check build_complete for centos9

with all ceph releases moving to centos 9-based containers, teuthology
should prevent scheduling jobs until those container builds finish

update the DEFAULT_OS_VERSION for centos to 9 so it checks for those
containers when no --distro-version is specified

Fixes: https://tracker.ceph.com/issues/66096
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1950 from rishabh-d-dave/bootstrap-fedora-fix

boostrap: expect "FedoraLinux" along with "Fedora"

boostrap: expect "FedoraLinux" along with "Fedora"

On Fedora 38 -

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"
ID=fedora
VERSION_ID=38

Also see: https://fedoraproject.org/wiki/Changes/Fedora_Linux_in_os-release.

With this "./boostrap install" works fine on Fedora 38.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

teuthology/suite: use valid subset default for rerun without --subset

Fixes: https://tracker.ceph.com/issues/66302
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1948 from ceph/py312-centos

bootstrap: Improve Python version detection; default to 3.12 on CentOS & Fedora

lock.query.find_stale_locks: Fix a false positive

Also fix the return type and docstring, which were inconsistent.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1942 from VallariAg/fix-install-task

task/install: Fix AssertionError

bootstrap: Default to py3.12 on CentOS & Fedora

Fixes: https://tracker.ceph.com/issues/66178
Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Improve Python version detection

Signed-off-by: Zack Cerza <zack@redhat.com>