Merge pull request #2002 from ceph/node-cleanup-false-pos

Merge pull request #2004 from ceph/suite-user

suite: use owner value for user name

Merge pull request #2007 from ceph/wip-68285

run: Fix reference to ctx for interactive jobs

Merge pull request #2006 from joscollin/wip-B68037-drop-packages_2-1

teuthology/task: drop packages that are not built

run: Fix reference to ctx for interactive jobs

Fixes: https://tracker.ceph.com/issues/68285
Signed-off-by: Zack Cerza <zack@redhat.com>

test_run: Add test_main_interactive

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/task: drop packages that are not built

Fixes the error `Unable to find a match: python3-ceph`.

Fixes: https://tracker.ceph.com/issues/68037
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #2005 from ceph/wip-B68037-drop-packages

teuthology/task: drop packages that are not built

Merge pull request #1936 from amathuria/wip-amat-fix-suite-sha1

suite/run: Add a option to specify suite sha1

teuthology/task: drop packages that are not built

FIxes: https://tracker.ceph.com/issues/68037
Signed-off-by: Jos Collin <jcollin@redhat.com>

suite: use owner value for user name

Signed-off-by: Zack Cerza <zack@redhat.com>

lock: Avoid querying paddles for non-jobs

When we encounter a node that's locked with a description that doesn't look
like it points to a job, avoid the inevitable 404 we'd get from paddles. Without
this, the cleanup process gets short-circuited.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2001 from ceph/bootstrap

bootstrap: Simplify dependency checking/install

bootstrap: Simplify dependency checking/install

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1997 from kshtsk/wip-exclude-tests-from-wheel

setup.cfg: exclude tests from wheel package

Merge pull request #2000 from ceph/user

orchestra: Respect username if set in ssh config

orchestra: Respect username if set in ssh config

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: exclude tests from packaging

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1995 from kshtsk/wip-fix-1816

contextutil.safe_while does not sleep between tries if needed to loop infinitely

Merge pull request #1999 from ljflores/wip-revert-tracker-66742

Revert "orchestra/daemon/cephadmunit.py: fix method kill_cmd()"

Revert "orchestra/daemon/cephadmunit.py: fix method kill_cmd()"

This reverts commit 2c6d64a222b4d38d4782ed407e045a86df0a6524.

Related tracker: https://tracker.ceph.com/issues/66883

Merge pull request #1983 from ceph/expiry

Add job expiration dates

Merge pull request #1990 from kshtsk/wip-downburst-fixes

bunch of downburst fixes

supervisor: Check for job expiration

This commit isn't strictly necessary for the feature's implementation, but will
allow testing the feature on the production teuthology cluster before merging.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add job expiration dates

This feature has two parts:
* Specifying expiration dates when scheduling test runs
* A global maximum age

Expiration dates are provided by passing `--expire` to `teuthology-suite` with
a relative value like `1d` (one day), `1w` (one week), or an absolute value like
`1999-12-31_23:59:59`.

A new configuration item, `max_job_age`, is specified in seconds. This defaults
to two weeks.

When the dispatcher checks the queue for the next job to run, it will first
compare the job's `timestamp` value - which reflects the time the job was
scheduled. If more than `max_job_age` seconds have passed, the job is skipped
and marked dead. It next checks for an `expire` value; if that value is in the
past, the job is skipped and marked dead. Otherwise, it will be run as usual.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Ensure teuthology config is consistent between tests

test_init.py was making modifications to the config object that persisted
between tests. When I fixed that, initially some tests in test_run_.py started
failing because of settings in my local ~/.teuthology.yaml. This change causes
all of the tests in suite.test to use default config values.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_misc: Speed up and fix two test bugs

One test had a missing assert; another had a comparison that would never fire
because of an expected exception being raised during the call.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-dev: Allow passing more suite args

Signed-off-by: Zack Cerza <zack@redhat.com>

util.time: Add parse_timestamp

And move the format string to the time module.

Signed-off-by: Zack Cerza <zack@redhat.com>

util.time: Add new submodule with parse_offset()

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/siteconfig.rst: Update references to worker

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1996 from kshtsk/wip-shlex-instead-of-pipes

orcherstra/run: don't use pipes, but shlex

orcherstra/run: don't use pipes, but shlex

Finally get rid of deprecation warning for 'pipes':

    teuthology/orchestra/run.py:12
      /teuthology/teuthology/orchestra/run.py:12: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
        import pipes

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

misc: allow _ssh_keyscan return None again

A host may not have any keys, so _ssh_keyscan should return None,
instead of failing with error:

    IndexError: list index out of range

Fixes: dbd55e37563ce0fde01e1bfa19ca2fd2c03f0194
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: add PATH to environment

Need to passthrough the PATH to downburst environment,
otherwise on some system like macos where mkisofs installed
via homebrew it cannot be found.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: ignore "username@" prefix in hostname

If downburst gets hostname as an argument which does not have
a username@ prefix (does not include @-symbol) there is an
index out of range error will occur, which we, obviously,
don't want and wish to allow hostnames don't use login names.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1989 from kshtsk/wip-stop-node

lock/ops: try to stop_node before unlock

Merge pull request #1991 from kshtsk/wip-arbitrary-bootstrap-venv

bootstrap: fix VENV variable utilization

Merge pull request #1992 from kshtsk/wip-deprecation-warnings

get rid of deprecation warnings

Merge pull request #1993 from kshtsk/wip-allow-apple-git

test/test_repo_utils: correctly parse system git version

lock/query: make use of tries for list_locks()

The list_locks has been updated with new argument 'tries' in PR#1816,
but by unfortunate mistake hasn't been used yet.

Fixes: 55886eac960cfc6f311fb4c9812311c420761793
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

suite/util: list_lock() once for get_arch()

We only want to try to list_locks() for getting arch by
machine_type when scheduling a suite. Otherwise we've
got into an infinite loop and console log flooded
with useless error messages.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

contextutil: sleep between tries in safe_while again

This patch fixes an issue introduced within PR #1816,
that when tries are < 0 (e.g. -1) there is no sleep
occurred between tries.

It also fixes error message, so we repot correct
number of tries we've done now.

Fixes: ee43f87cfe88a8155566b662750d663c1d8742ff
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1981 from ceph/import-tests

Disable prometheus when exporter isn't running

Merge PR #1994 into main

* refs/pull/1994/head:
teuthology/suite: add len() lupa function

Reviewed-by: Samuel Just <sjust@redhat.com>

teuthology/suite: add len() lupa function

Namespacing for this is weird which is why I missed it.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

timer: use datetime.timezone.utc for python 3.10

Python 3.10 doesn't have datetime.UTC yet as Python 3.12 does

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

test/test_repo_utils: correctly parse system git version

On macos there are bunch of tests fails because system
git version output cannot be parsed because of extra ending.

Output example for macos:
   git version 2.39.3 (Apple Git-146)
Output example for linux:
   git version 2.45.2

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/ssh_keys: fix invalid escape sequence '\w'

Fix syntax warning message:

    <unknown>:68: SyntaxWarning: invalid escape sequence '\w'
    teuthology/task/ssh_keys.py:57: SyntaxWarning: invalid escape sequence '\w'
      match = re.match('[\w-]+ {key} \S+@\S+'.format(key=re.escape(ssh_key)), line_to_test)
    teuthology/task/ssh_keys.py:68: SyntaxWarning: invalid escape sequence '\w'
      match = re.match('[\w-]+ \S+ {username}@\S+'.format(username=username), line_to_test)

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/mpi: fix invalid escape sequence

    teuthology/task/mpi.py:21: SyntaxWarning: invalid escape sequence '\s'
      version = re.search("^\s+Version:\s+(.+)$", version_str, re.MULTILINE).group(1)

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

util/scanner: get rid of FutureWarning message

Get rid of the warning:

    teuthology/util/test/test_scanner.py::TestValgrindScanner::test_scan_all_files
      /Users/kyr/kshtsk/teuthology/teuthology/util/scanner.py:133: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
        if not xml_tree:

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

timer: fix deprecation warning for utcfromtimestamp

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

lock/cli: fix invalid escape sequence

Address warning message which is going to be an error in future
python versions:

  SyntaxWarning: invalid escape sequence '\w'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/kernel: fix invalid escape sequence

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

bootstrap: fix VENV variable utilization

Allow bootstrap to use path outside of teuthology repo
root directory for building virtualenv.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

exporter: With no exporter running, metrics collection should no-op

The new import tests exposed a ValueError raised by prometheus_client regarding
duplicate metrics in the multiprocess directory.

Signed-off-by: Zack Cerza <zack@redhat.com>

exporter: Catch psutil.AccessDenied

Signed-off-by: Zack Cerza <zack@redhat.com>

scripts: Add test_exporter_.py

Signed-off-by: Zack Cerza <zack@redhat.com>

Rewrite module import tests

These are 38x faster. They also helped me find and fix a bug in the exporter

Signed-off-by: Zack Cerza <zack@redhat.com>

Add types-psutil

This is, of course, for type hinting.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1988 from kshtsk/wip-github-checkout-v4

workflows: use latest checkout and setup-python actions

lock/ops: try to stop_node before unlock

We should try to call stop_node before unlock and don't
even try to do this for vms.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

workflows: use latest checkout and setup-python actions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1985 from kshtsk/wip-access-denied

Fix access denied from psutil on macOS

kill: handle AccessDenied exception

Fix unhandled exception psutilAccessDenied in process_matches_run().

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

dispatcher: fix AccessDenied on process lookup

On macOS dispatcher while trying to go through process list gets stuck
on some of the system processes like launchd, logd, systemstats, etc.
and quites unexpectedly with PermissionError and psutil.AccessDenied
exceptions.

Fixes: https://tracker.ceph.com/issues/67313
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1987 from ceph/docker-compose

Migrate to docker compose v2

Merge pull request #1982 from ceph/rm-worker

Remove worker.py

Migrate to docker compose v2

https://github.com/actions/runner-images/issues/9692
https://docs.docker.com/compose/migrate/

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove worker.py

Originally, #1960 was intended to do this, but in some review back-and-forth the
commit removing the file was dropped. Let's actually remove the file.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1960 from ceph/worker-supervisor

Finish removing teuthology-worker

dispatcher: Temporarily pass through to supervisor

The old dispatcher expects to be able to invoke the supervisor via
`teuthology-dispatcher --supervisor`, so add this compatibility shim for the
time being.

Signed-off-by: Zack Cerza <zack@redhat.com>

Finish removing teuthology-worker

The dispatcher and supervisor were added in #1546, but code was copied and
pasted into the new modules, leaving the worker untouched. Also untouched were
the unit tests, meaning that the dispatcher and supervisor were never unit
tested. As the copied code changed, the dispatcher and supervisor were not being
tested for regressions, while the worker - which wasn't being anymore - had
passing unit tests, giving some false sense of security.

This commit removes the old worker code, and adapts the old worker tests to
apply to the dispatcher and supervisor. It also splits out teuthology-supervisor
into its own command.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1977 from ceph/utcnow

Replace deprecated datetime.utcnow() calls

Replace deprecated datetime.utcnow() calls

See https://blog.miguelgrinberg.com/post/it-s-time-for-a-change-datetime-utcnow-is-now-deprecated

Signed-off-by: Zack Cerza <zack@redhat.com>

ci: Test on Ubuntu 24.04 with python3.12

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1975 from ceph/test-debug

Get the 'tests' task working again

Merge pull request #1980 from ceph/kill-job-paddles-data

kill.kill_job: Get job data from paddles

kill.kill_job: Get job data from paddles

... if we can't get it from the filesystem. This is necessary because of the
teuthology archive's restricted permissions on teuthology.front.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1979 from ceph/issue-66893-fix

find_git_parents: Fix githelper URL template

find_git_parents: Fix githelper URL template

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1976 from ceph/issue-66893

find_git_parents: Refresh mirror when required

Merge pull request #1974 from ceph/wip-selinux-ignore-agetty

teuthology/task/selinux: ignore denials in agetty

find_git_parents: Refresh mirror when required

We used to do the refresh every time, which wasn't good. Then we didn't do it at
all, which was fine until someone tried to schedule against a commit in the few
minutes before the mirror picked it up. Let's try doing it when necessary :)

Fixes: #66893
Signed-off-by: Zack Cerza <zack@redhat.com>

test_report: Use module-level tests and fixtures

Signed-off-by: Zack Cerza <zack@redhat.com>

Task.__init__: ctx param is mandatory

This reflects reality and also fixes some linter errors.

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.deep_merge: Rewrite for readability

Partially enabled by not passing MappingProxyType to it any longer.

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.deep_merge,merge_configs: Add type hinting

Signed-off-by: Zack Cerza <zack@redhat.com>

run: Use job config's archive dir if specified

Signed-off-by: Zack Cerza <zack@redhat.com>

Run.report_outcome: Drop fake_ctx

Signed-off-by: Zack Cerza <zack@redhat.com>

test_fog: fix test conflict with local config

Signed-off-by: Zack Cerza <zack@redhat.com>

test_report: Use a tmp_path fixture

Signed-off-by: Zack Cerza <zack@redhat.com>

test_ansible: Set ctx.archive to empty string

Scheduled jobs are failing because they try to write to
/home/ansible.log.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Rewrite

This task has been broken for a very long time. It hasn't passed in years, and
has additionally been masking test failures including some regressions.

This commit:
* Fixes masking of failures in setup/teardown methods
* Actually runs all of our tests
* Excludes tests that need remote nodes when they are not available
* Improves logging and reporting
* Adds support for running in CLI mode

Signed-off-by: Zack Cerza <zack@redhat.com>

Move LoggerFile to teuthology.util

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/task/selinux: ignore denials in agetty

This was tracked in [1], but the addition of the context to the
allowlist in ceph.git commit 7a6389272aff ("qa: ignore container
checkpoint/restore related selinux denials for centos9") doesn't
cover krbd suite which doesn't pull in any distro snippets.

Since this denial has nothing to do with Ceph [2], let's ignore it
globally.

[1] http://tracker.ceph.com/issues/64616
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2259622

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #1921 from phlogistonjohn/jjm-ssh-config-hostname

orchestra/connection: ssh configuration options

Merge pull request #1972 from VallariAg/fix-teuthology-kill

teuthology/kill.py: Fix "TypeError: 'NoneType' object is not iterable"

Create teuthology/test/test_kill.py

Add TestFindTargets in test_kill for different
test cases of find_targets function.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

suite/run: Add a option to specify suite sha1

The option can be used to specify the workunit sha1, it will override the --suite-branch option if both are provided.
If --suite-repo is specified, it will search for the --suite-sha1 in this repo otherwise it will look for the sha1 in the default suite repo.

Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>