orchestra/opsys: add tumbleweed version to the distro version map

openSUSE Tumbleweed is a rolling release, so it has fixed version number
and changable version id is represented by a date of snapshot,
example of vm image:

openSUSE-Tumbleweed-Minimal-VM.x86_64-1.0.0-Cloud-Snapshot20250224.qcow2

In comparison, openSUSE Leap has regular release, incremental version, for example:
15.0, 15.1, ..., 15.6, etc., and recently introduced 16.0 alpha.

Because there is no sense to stick to the date, it is changing almost daily,
it is suggested to fix Tumbleweed version to 1.0 to distinguish from Leap.
As a side effect, once Teuthology locks a node with imaged Tumbleweed
it updates os version correctly to date of release in paddles, and
pulpito display it interface correspondingly as the date.

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: add 'alma' and 'rocky' to default os map

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: add almalinux versions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: add 'rocky' to DISTRO_CODENAME_MAP

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2025 from ceph/nuke-unlock

supervisor: Unlock nodes after reimage failure

node_active_job: Improve http error handling

Signed-off-by: Zack Cerza <zack1@ibm.com>

node_cleanup: Log owner when query fails

Signed-off-by: Zack Cerza <zack1@ibm.com>

supervisor: Unlock nodes after reimage failure

Signed-off-by: Zack Cerza <zack1@ibm.com>

node_cleanup: Log a summary of what was done

Signed-off-by: Zack Cerza <zack1@ibm.com>

Merge pull request #2024 from kshtsk/wip-64452

Merge pull request #1984 from kshtsk/wip-laptop-dispatcher

orchestra/run: better handle weird args

If one of the command args in remote.run is None then there is exception
occurs while trying to qoute argument list:

   TypeError: expected string or bytes-like object

This fix addresses the issue and gives user better idea what is
happening, and how to fix the error.

Fixes: https://tracker.ceph.com/issues/64452
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2022 from kshtsk/wip-stuck-on-vm-create-fail

teuthology lock is stuck if vm create failed

docs/laptop: update localhost setup instructions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: log multiline stderr on error

For better logs readability just log downburst's stderr
line-by-line.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

lock/ops: don't misguide user while waiting for vms up

The message "virtual machine is still unavailable" is mistakenly
shown to user, when vms are actually available.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

lock/cli: don't update inventory if failed to create a vm

If we were not able to create a vm we don't need to update machine's
inventory, because there is going to be an ssh connection established
to the host and teuthology gets stuck infinitely trying to connect to
the machine which does not exist.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2014 from ceph/wip-nitzan-add-timout-param-stop-type

Merge pull request #2021 from kshtsk/wip-job-archive-in-supervisor

dispatcher/supervisor: use job config only to unlock targets

No need to use serializer, because everything needed to unlock
machines already present in job config.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2019 from ceph/dependabot/pip/jinja2-3.1.5

build(deps): bump jinja2 from 3.1.4 to 3.1.5

Merge pull request #2017 from kshtsk/wip-drop-future-print

teuthology: drop __future__ import for print

build(deps): bump jinja2 from 3.1.4 to 3.1.5

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.4 to 3.1.5.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.4...3.1.5)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #2010 from chrisphoffman/wip-dd-syncseek

teuthology: Add support for seek and sync in write_file

Merge pull request #1998 from phlogistonjohn/jjm-config-alt-path

config: allow reading teuthology config from env var location

Merge pull request #2018 from ceph/devcontainer-no-pat

containers/teuthology-dev: Remove access token

This container is built and pushed via GitHub Actions. GHA likes to provision a
personal access token for each job that gives tightly-scoped access to the git
repository to the job. When we build our container, we end up including
`.git/config`, which contains the token. Later, in ceph-dev-stack's CI, an
`ls-remote` is run against ceph.git, which ends up causing git to prompt for
credentials even though the repo is public. Removing the token should allow
reading all the relevant repos from the built container image.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology: Add tests for seek and sync in write_file

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

teuthology: Add support for seek and sync in write_file

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

config: allow reading teuthology config from env var location

Allow changing the default "user" location of the teuthology
configuration yaml using the (optional) TEUTHOLOGY_CONFIG environment
variable. This change aids my effort to run a customized local
teuthology environment.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #2015 from thuvh/lab_setup_python3

doc: use python3 cause of deprecating of python on ubuntu

doc: use python3 cause of deprecating of python on ubuntu

Signed-off-by: Hoai-Thu Vuong <thuvh87@gmail.com>

Merge pull request #2016 from ceph/dependabot/pip/configobj-5.0.9

build(deps): bump configobj from 5.0.8 to 5.0.9

teuthology: drop __future__ import for print

We don't need to support backward compatibility with python 2.

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@gmail.com>

build(deps): bump configobj from 5.0.8 to 5.0.9

Bumps [configobj](https://github.com/DiffSK/configobj) from 5.0.8 to 5.0.9.
- [Release notes](https://github.com/DiffSK/configobj/releases)
- [Changelog](https://github.com/DiffSK/configobj/blob/release/CHANGES.rst)
- [Commits](https://github.com/DiffSK/configobj/compare/v5.0.8...v5.0.9)

---
updated-dependencies:
- dependency-name: configobj
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

teuthology/misc: Add timeout parameter to stop_daemons_of_type for better flexibility

Updated stop_daemons_of_type to accept a timeout parameter,
allowing dynamic control over the timeout value passed to the
stop function of each daemon.

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #2013 from ceph/dependabot/pip/ansible-core-2.17.7

build(deps): bump ansible-core from 2.16.6 to 2.17.7

build(deps): bump ansible-core from 2.16.6 to 2.17.7

Bumps [ansible-core](https://github.com/ansible/ansible) from 2.16.6 to 2.17.7.
- [Release notes](https://github.com/ansible/ansible/releases)
- [Commits](https://github.com/ansible/ansible/compare/v2.16.6...v2.17.7)

---
updated-dependencies:
- dependency-name: ansible-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #2009 from ceph/fog-image

Merge pull request #2008 from batrick/fragment-merge-base-config

provision.fog: Be more informative re: API errors

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/tasks/kernel: default branch should be "distro"

Generally, we don't want to install any kernel by default. There also is no
"main" branch for the ceph-client.git project.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: merge base_config with other fragments

Presently the code tries to merge the base_config when the worker starts
running. There's no need to construct it this way and it prevents sharing the
"defaults" with the fragment merging infrastructure. It also prevents
overriding defaults like:

    kernel
        branch: wip-pdonnell-i66704
        client:
            branch: wip-pdonnell-i66704
            flavor: default
            kdb: 1
            sha1: 745cacd8f31e50d7f3b6039bbd8c9a8dfc07bf03
        flavor: default
        kdb: 1
        sha1: 745cacd8f31e50d7f3b6039bbd8c9a8dfc07bf03

A YAML fragment can set kernel.client but it cannot delete the defaults for
kernel.(branch|flavor|kdb|sha1) because there's no way to remove YAML elements
via a deep merge.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #2002 from ceph/node-cleanup-false-pos

teuthology/suite: set the kernel branch in the kernel config

For inspection by scripts.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: print base config before config merge

So it's easier to see its value before any configs are generated.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #2004 from ceph/suite-user

suite: use owner value for user name

Merge pull request #2007 from ceph/wip-68285

run: Fix reference to ctx for interactive jobs

Merge pull request #2006 from joscollin/wip-B68037-drop-packages_2-1

teuthology/task: drop packages that are not built

run: Fix reference to ctx for interactive jobs

Fixes: https://tracker.ceph.com/issues/68285
Signed-off-by: Zack Cerza <zack@redhat.com>

test_run: Add test_main_interactive

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/task: drop packages that are not built

Fixes the error `Unable to find a match: python3-ceph`.

Fixes: https://tracker.ceph.com/issues/68037
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #2005 from ceph/wip-B68037-drop-packages

teuthology/task: drop packages that are not built

Merge pull request #1936 from amathuria/wip-amat-fix-suite-sha1

suite/run: Add a option to specify suite sha1

teuthology/task: drop packages that are not built

FIxes: https://tracker.ceph.com/issues/68037
Signed-off-by: Jos Collin <jcollin@redhat.com>

suite: use owner value for user name

Signed-off-by: Zack Cerza <zack@redhat.com>

lock: Avoid querying paddles for non-jobs

When we encounter a node that's locked with a description that doesn't look
like it points to a job, avoid the inevitable 404 we'd get from paddles. Without
this, the cleanup process gets short-circuited.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2001 from ceph/bootstrap

bootstrap: Simplify dependency checking/install

bootstrap: Simplify dependency checking/install

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1997 from kshtsk/wip-exclude-tests-from-wheel

setup.cfg: exclude tests from wheel package

Merge pull request #2000 from ceph/user

orchestra: Respect username if set in ssh config

orchestra: Respect username if set in ssh config

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: exclude tests from packaging

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1995 from kshtsk/wip-fix-1816

contextutil.safe_while does not sleep between tries if needed to loop infinitely

Merge pull request #1999 from ljflores/wip-revert-tracker-66742

Revert "orchestra/daemon/cephadmunit.py: fix method kill_cmd()"

Revert "orchestra/daemon/cephadmunit.py: fix method kill_cmd()"

This reverts commit 2c6d64a222b4d38d4782ed407e045a86df0a6524.

Related tracker: https://tracker.ceph.com/issues/66883

Merge pull request #1983 from ceph/expiry

Add job expiration dates

Merge pull request #1990 from kshtsk/wip-downburst-fixes

bunch of downburst fixes

supervisor: Check for job expiration

This commit isn't strictly necessary for the feature's implementation, but will
allow testing the feature on the production teuthology cluster before merging.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add job expiration dates

This feature has two parts:
* Specifying expiration dates when scheduling test runs
* A global maximum age

Expiration dates are provided by passing `--expire` to `teuthology-suite` with
a relative value like `1d` (one day), `1w` (one week), or an absolute value like
`1999-12-31_23:59:59`.

A new configuration item, `max_job_age`, is specified in seconds. This defaults
to two weeks.

When the dispatcher checks the queue for the next job to run, it will first
compare the job's `timestamp` value - which reflects the time the job was
scheduled. If more than `max_job_age` seconds have passed, the job is skipped
and marked dead. It next checks for an `expire` value; if that value is in the
past, the job is skipped and marked dead. Otherwise, it will be run as usual.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Ensure teuthology config is consistent between tests

test_init.py was making modifications to the config object that persisted
between tests. When I fixed that, initially some tests in test_run_.py started
failing because of settings in my local ~/.teuthology.yaml. This change causes
all of the tests in suite.test to use default config values.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_misc: Speed up and fix two test bugs

One test had a missing assert; another had a comparison that would never fire
because of an expected exception being raised during the call.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-dev: Allow passing more suite args

Signed-off-by: Zack Cerza <zack@redhat.com>

util.time: Add parse_timestamp

And move the format string to the time module.

Signed-off-by: Zack Cerza <zack@redhat.com>

util.time: Add new submodule with parse_offset()

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/siteconfig.rst: Update references to worker

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1996 from kshtsk/wip-shlex-instead-of-pipes

orcherstra/run: don't use pipes, but shlex

orcherstra/run: don't use pipes, but shlex

Finally get rid of deprecation warning for 'pipes':

    teuthology/orchestra/run.py:12
      /teuthology/teuthology/orchestra/run.py:12: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
        import pipes

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

misc: allow _ssh_keyscan return None again

A host may not have any keys, so _ssh_keyscan should return None,
instead of failing with error:

    IndexError: list index out of range

Fixes: dbd55e37563ce0fde01e1bfa19ca2fd2c03f0194
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: add PATH to environment

Need to passthrough the PATH to downburst environment,
otherwise on some system like macos where mkisofs installed
via homebrew it cannot be found.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: ignore "username@" prefix in hostname

If downburst gets hostname as an argument which does not have
a username@ prefix (does not include @-symbol) there is an
index out of range error will occur, which we, obviously,
don't want and wish to allow hostnames don't use login names.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1989 from kshtsk/wip-stop-node

lock/ops: try to stop_node before unlock

Merge pull request #1991 from kshtsk/wip-arbitrary-bootstrap-venv

bootstrap: fix VENV variable utilization

Merge pull request #1992 from kshtsk/wip-deprecation-warnings

get rid of deprecation warnings

Merge pull request #1993 from kshtsk/wip-allow-apple-git

test/test_repo_utils: correctly parse system git version

lock/query: make use of tries for list_locks()

The list_locks has been updated with new argument 'tries' in PR#1816,
but by unfortunate mistake hasn't been used yet.

Fixes: 55886eac960cfc6f311fb4c9812311c420761793
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

suite/util: list_lock() once for get_arch()

We only want to try to list_locks() for getting arch by
machine_type when scheduling a suite. Otherwise we've
got into an infinite loop and console log flooded
with useless error messages.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

contextutil: sleep between tries in safe_while again

This patch fixes an issue introduced within PR #1816,
that when tries are < 0 (e.g. -1) there is no sleep
occurred between tries.

It also fixes error message, so we repot correct
number of tries we've done now.

Fixes: ee43f87cfe88a8155566b662750d663c1d8742ff
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1981 from ceph/import-tests

Disable prometheus when exporter isn't running

Merge PR #1994 into main

* refs/pull/1994/head:
teuthology/suite: add len() lupa function

Reviewed-by: Samuel Just <sjust@redhat.com>

teuthology/suite: add len() lupa function

Namespacing for this is weird which is why I missed it.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

timer: use datetime.timezone.utc for python 3.10

Python 3.10 doesn't have datetime.UTC yet as Python 3.12 does

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

test/test_repo_utils: correctly parse system git version

On macos there are bunch of tests fails because system
git version output cannot be parsed because of extra ending.

Output example for macos:
   git version 2.39.3 (Apple Git-146)
Output example for linux:
   git version 2.45.2

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/ssh_keys: fix invalid escape sequence '\w'

Fix syntax warning message:

    <unknown>:68: SyntaxWarning: invalid escape sequence '\w'
    teuthology/task/ssh_keys.py:57: SyntaxWarning: invalid escape sequence '\w'
      match = re.match('[\w-]+ {key} \S+@\S+'.format(key=re.escape(ssh_key)), line_to_test)
    teuthology/task/ssh_keys.py:68: SyntaxWarning: invalid escape sequence '\w'
      match = re.match('[\w-]+ \S+ {username}@\S+'.format(username=username), line_to_test)

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/mpi: fix invalid escape sequence

    teuthology/task/mpi.py:21: SyntaxWarning: invalid escape sequence '\s'
      version = re.search("^\s+Version:\s+(.+)$", version_str, re.MULTILINE).group(1)

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

util/scanner: get rid of FutureWarning message

Get rid of the warning:

    teuthology/util/test/test_scanner.py::TestValgrindScanner::test_scan_all_files
      /Users/kyr/kshtsk/teuthology/teuthology/util/scanner.py:133: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
        if not xml_tree:

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

timer: fix deprecation warning for utcfromtimestamp

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

lock/cli: fix invalid escape sequence

Address warning message which is going to be an error in future
python versions:

  SyntaxWarning: invalid escape sequence '\w'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/kernel: fix invalid escape sequence

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

bootstrap: fix VENV variable utilization

Allow bootstrap to use path outside of teuthology repo
root directory for building virtualenv.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>