Add jq and curl in docker-compose Dockerfile

Signed-off-by: deepssin <deepssin@redhat.com>

Revive teuthology-openstack

Signed-off-by: deepssin <deepssin@redhat.com>

Merge pull request #2039 from ceph/dependabot/pip/jinja2-3.1.6

build(deps): bump jinja2 from 3.1.5 to 3.1.6

Merge pull request #2035 from VallariAg/ci-integration-logs-archive

ci: save integration logs artifacts

ci: save intergation logs artifacts

Mount volume for archive_dir and then
upload it's logfiles as GA artifacts,
so we can use download-artifacts action
to download logs.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

build(deps): bump jinja2 from 3.1.5 to 3.1.6

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.5...3.1.6)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #2038 from ceph/actions-pinning

workflows/dev_container: Pin docker actions

workflows/dev_container: Pin docker actions

These are the most recent releases for these actions.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2026 from badone/main

task/internal/__init__.py:  Revisit core program extraction regex

task/internal/__init__.py:  Revisit core program extraction regex

Fixes: https://tracker.ceph.com/issues/61586
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #1773 from batrick/1772-revert

Revert "Merge PR #1772 into master"

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #2034 from VallariAg/fix-locked-grace-period

lock/query.py: add grace_time param to node_active_job()

lock/query.py: add grace_time param to node_active_job()

This is because we only want this grace period
when looking for stale jobs. Not when jobs are
being unlocked by the supervisor by unlock_one_safe().

Fixes: https://github.com/ceph/teuthology/pull/2033#issuecomment-2703733380
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #2033 from ceph/node-cleanup-grace

node-cleanup: Grace period for inactive jobs

node-cleanup: Grace period for inactive jobs

Once a job is marked finished, the supervisor may still be waiting to unlock its
nodes. Give jobs five minutes to clean up nodes before we consider them "stale".

Signed-off-by: Zack Cerza <zack@cerza.org>

node-cleanup: Use plain log format

Signed-off-by: Zack Cerza <zack@cerza.org>

node-cleanup: Provide link to job

Instead of just the raw description.

Signed-off-by: Zack Cerza <zack@cerza.org>

Revert "Merge PR #1772 into master"

This reverts commit 082ae7ef4302fa54665ed0a2535e8e254118dcfd, reversing
changes made to 31a356e0c737f1781079045bb8dc366982be8713.

The original ea170935d4b1c78dc6fd6beae6b3fda65b296f57 removed a method that had
been moved to ceph.git but that broke upgrade tests from releases up to Octopus
where get_valgrind_args had not been backported. So a revert was done until all
upgrade paths have the method.

Now that Quincy is EOL, we can revert the revert.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #2030 from jmundack/add_codeowners

Add CODEOWNERS file

Add CODEOWNERS file

Merge pull request #2020 from ceph/dependabot/pip/virtualenv-20.26.6

build(deps): bump virtualenv from 20.25.0 to 20.26.6

Merge pull request #2028 from kshtsk/wip-docs-laptop-dispatcher

docs/laptop: replace workers with dispatcher

Merge pull request #2027 from kshtsk/wip-almarocky

opsys: add AlmaLinux and RockyLinux and Tumbleweed to distro codename map

orchestra/opsys: add tumbleweed version to the distro version map

openSUSE Tumbleweed is a rolling release, so it has fixed version number
and changable version id is represented by a date of snapshot,
example of vm image:

openSUSE-Tumbleweed-Minimal-VM.x86_64-1.0.0-Cloud-Snapshot20250224.qcow2

In comparison, openSUSE Leap has regular release, incremental version, for example:
15.0, 15.1, ..., 15.6, etc., and recently introduced 16.0 alpha.

Because there is no sense to stick to the date, it is changing almost daily,
it is suggested to fix Tumbleweed version to 1.0 to distinguish from Leap.
As a side effect, once Teuthology locks a node with imaged Tumbleweed
it updates os version correctly to date of release in paddles, and
pulpito display it interface correspondingly as the date.

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: add 'alma' and 'rocky' to default os map

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: add almalinux versions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: add 'rocky' to DISTRO_CODENAME_MAP

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

docs/laptop: replace workers with dispatcher

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

Merge pull request #2025 from ceph/nuke-unlock

supervisor: Unlock nodes after reimage failure

node_active_job: Improve http error handling

Signed-off-by: Zack Cerza <zack1@ibm.com>

node_cleanup: Log owner when query fails

Signed-off-by: Zack Cerza <zack1@ibm.com>

supervisor: Unlock nodes after reimage failure

Signed-off-by: Zack Cerza <zack1@ibm.com>

node_cleanup: Log a summary of what was done

Signed-off-by: Zack Cerza <zack1@ibm.com>

Merge pull request #2024 from kshtsk/wip-64452

Merge pull request #1984 from kshtsk/wip-laptop-dispatcher

orchestra/run: better handle weird args

If one of the command args in remote.run is None then there is exception
occurs while trying to qoute argument list:

   TypeError: expected string or bytes-like object

This fix addresses the issue and gives user better idea what is
happening, and how to fix the error.

Fixes: https://tracker.ceph.com/issues/64452
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2022 from kshtsk/wip-stuck-on-vm-create-fail

teuthology lock is stuck if vm create failed

docs/laptop: update localhost setup instructions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: log multiline stderr on error

For better logs readability just log downburst's stderr
line-by-line.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

lock/ops: don't misguide user while waiting for vms up

The message "virtual machine is still unavailable" is mistakenly
shown to user, when vms are actually available.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

lock/cli: don't update inventory if failed to create a vm

If we were not able to create a vm we don't need to update machine's
inventory, because there is going to be an ssh connection established
to the host and teuthology gets stuck infinitely trying to connect to
the machine which does not exist.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2014 from ceph/wip-nitzan-add-timout-param-stop-type

Merge pull request #2021 from kshtsk/wip-job-archive-in-supervisor

dispatcher/supervisor: use job config only to unlock targets

No need to use serializer, because everything needed to unlock
machines already present in job config.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

build(deps): bump virtualenv from 20.25.0 to 20.26.6

Bumps [virtualenv](https://github.com/pypa/virtualenv) from 20.25.0 to 20.26.6.
- [Release notes](https://github.com/pypa/virtualenv/releases)
- [Changelog](https://github.com/pypa/virtualenv/blob/main/docs/changelog.rst)
- [Commits](https://github.com/pypa/virtualenv/compare/20.25.0...20.26.6)

---
updated-dependencies:
- dependency-name: virtualenv
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #2019 from ceph/dependabot/pip/jinja2-3.1.5

build(deps): bump jinja2 from 3.1.4 to 3.1.5

Merge pull request #2017 from kshtsk/wip-drop-future-print

teuthology: drop __future__ import for print

build(deps): bump jinja2 from 3.1.4 to 3.1.5

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.4 to 3.1.5.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.4...3.1.5)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #2010 from chrisphoffman/wip-dd-syncseek

teuthology: Add support for seek and sync in write_file

Merge pull request #1998 from phlogistonjohn/jjm-config-alt-path

config: allow reading teuthology config from env var location

Merge pull request #2018 from ceph/devcontainer-no-pat

containers/teuthology-dev: Remove access token

This container is built and pushed via GitHub Actions. GHA likes to provision a
personal access token for each job that gives tightly-scoped access to the git
repository to the job. When we build our container, we end up including
`.git/config`, which contains the token. Later, in ceph-dev-stack's CI, an
`ls-remote` is run against ceph.git, which ends up causing git to prompt for
credentials even though the repo is public. Removing the token should allow
reading all the relevant repos from the built container image.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology: Add tests for seek and sync in write_file

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

teuthology: Add support for seek and sync in write_file

Signed-off-by: Christopher Hoffman <choffman@redhat.com>

config: allow reading teuthology config from env var location

Allow changing the default "user" location of the teuthology
configuration yaml using the (optional) TEUTHOLOGY_CONFIG environment
variable. This change aids my effort to run a customized local
teuthology environment.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #2015 from thuvh/lab_setup_python3

doc: use python3 cause of deprecating of python on ubuntu

doc: use python3 cause of deprecating of python on ubuntu

Signed-off-by: Hoai-Thu Vuong <thuvh87@gmail.com>

Merge pull request #2016 from ceph/dependabot/pip/configobj-5.0.9

build(deps): bump configobj from 5.0.8 to 5.0.9

teuthology: drop __future__ import for print

We don't need to support backward compatibility with python 2.

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@gmail.com>

build(deps): bump configobj from 5.0.8 to 5.0.9

Bumps [configobj](https://github.com/DiffSK/configobj) from 5.0.8 to 5.0.9.
- [Release notes](https://github.com/DiffSK/configobj/releases)
- [Changelog](https://github.com/DiffSK/configobj/blob/release/CHANGES.rst)
- [Commits](https://github.com/DiffSK/configobj/compare/v5.0.8...v5.0.9)

---
updated-dependencies:
- dependency-name: configobj
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

teuthology/misc: Add timeout parameter to stop_daemons_of_type for better flexibility

Updated stop_daemons_of_type to accept a timeout parameter,
allowing dynamic control over the timeout value passed to the
stop function of each daemon.

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #2013 from ceph/dependabot/pip/ansible-core-2.17.7

build(deps): bump ansible-core from 2.16.6 to 2.17.7

build(deps): bump ansible-core from 2.16.6 to 2.17.7

Bumps [ansible-core](https://github.com/ansible/ansible) from 2.16.6 to 2.17.7.
- [Release notes](https://github.com/ansible/ansible/releases)
- [Commits](https://github.com/ansible/ansible/compare/v2.16.6...v2.17.7)

---
updated-dependencies:
- dependency-name: ansible-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #2009 from ceph/fog-image

Merge pull request #2008 from batrick/fragment-merge-base-config

provision.fog: Be more informative re: API errors

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/tasks/kernel: default branch should be "distro"

Generally, we don't want to install any kernel by default. There also is no
"main" branch for the ceph-client.git project.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: merge base_config with other fragments

Presently the code tries to merge the base_config when the worker starts
running. There's no need to construct it this way and it prevents sharing the
"defaults" with the fragment merging infrastructure. It also prevents
overriding defaults like:

    kernel
        branch: wip-pdonnell-i66704
        client:
            branch: wip-pdonnell-i66704
            flavor: default
            kdb: 1
            sha1: 745cacd8f31e50d7f3b6039bbd8c9a8dfc07bf03
        flavor: default
        kdb: 1
        sha1: 745cacd8f31e50d7f3b6039bbd8c9a8dfc07bf03

A YAML fragment can set kernel.client but it cannot delete the defaults for
kernel.(branch|flavor|kdb|sha1) because there's no way to remove YAML elements
via a deep merge.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #2002 from ceph/node-cleanup-false-pos

teuthology/suite: set the kernel branch in the kernel config

For inspection by scripts.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: print base config before config merge

So it's easier to see its value before any configs are generated.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #2004 from ceph/suite-user

suite: use owner value for user name

Merge pull request #2007 from ceph/wip-68285

run: Fix reference to ctx for interactive jobs

Merge pull request #2006 from joscollin/wip-B68037-drop-packages_2-1

teuthology/task: drop packages that are not built

run: Fix reference to ctx for interactive jobs

Fixes: https://tracker.ceph.com/issues/68285
Signed-off-by: Zack Cerza <zack@redhat.com>

test_run: Add test_main_interactive

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/task: drop packages that are not built

Fixes the error `Unable to find a match: python3-ceph`.

Fixes: https://tracker.ceph.com/issues/68037
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #2005 from ceph/wip-B68037-drop-packages

teuthology/task: drop packages that are not built

Merge pull request #1936 from amathuria/wip-amat-fix-suite-sha1

suite/run: Add a option to specify suite sha1

teuthology/task: drop packages that are not built

FIxes: https://tracker.ceph.com/issues/68037
Signed-off-by: Jos Collin <jcollin@redhat.com>

suite: use owner value for user name

Signed-off-by: Zack Cerza <zack@redhat.com>

lock: Avoid querying paddles for non-jobs

When we encounter a node that's locked with a description that doesn't look
like it points to a job, avoid the inevitable 404 we'd get from paddles. Without
this, the cleanup process gets short-circuited.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #2001 from ceph/bootstrap

bootstrap: Simplify dependency checking/install

bootstrap: Simplify dependency checking/install

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1997 from kshtsk/wip-exclude-tests-from-wheel

setup.cfg: exclude tests from wheel package

Merge pull request #2000 from ceph/user

orchestra: Respect username if set in ssh config

orchestra: Respect username if set in ssh config

Signed-off-by: Zack Cerza <zack@redhat.com>

setup.cfg: exclude tests from packaging

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #1995 from kshtsk/wip-fix-1816

contextutil.safe_while does not sleep between tries if needed to loop infinitely

Merge pull request #1999 from ljflores/wip-revert-tracker-66742

Revert "orchestra/daemon/cephadmunit.py: fix method kill_cmd()"

Revert "orchestra/daemon/cephadmunit.py: fix method kill_cmd()"

This reverts commit 2c6d64a222b4d38d4782ed407e045a86df0a6524.

Related tracker: https://tracker.ceph.com/issues/66883

Merge pull request #1983 from ceph/expiry

Add job expiration dates

Merge pull request #1990 from kshtsk/wip-downburst-fixes

bunch of downburst fixes

supervisor: Check for job expiration

This commit isn't strictly necessary for the feature's implementation, but will
allow testing the feature on the production teuthology cluster before merging.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add job expiration dates

This feature has two parts:
* Specifying expiration dates when scheduling test runs
* A global maximum age

Expiration dates are provided by passing `--expire` to `teuthology-suite` with
a relative value like `1d` (one day), `1w` (one week), or an absolute value like
`1999-12-31_23:59:59`.

A new configuration item, `max_job_age`, is specified in seconds. This defaults
to two weeks.

When the dispatcher checks the queue for the next job to run, it will first
compare the job's `timestamp` value - which reflects the time the job was
scheduled. If more than `max_job_age` seconds have passed, the job is skipped
and marked dead. It next checks for an `expire` value; if that value is in the
past, the job is skipped and marked dead. Otherwise, it will be run as usual.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Ensure teuthology config is consistent between tests

test_init.py was making modifications to the config object that persisted
between tests. When I fixed that, initially some tests in test_run_.py started
failing because of settings in my local ~/.teuthology.yaml. This change causes
all of the tests in suite.test to use default config values.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_misc: Speed up and fix two test bugs

One test had a missing assert; another had a comparison that would never fire
because of an expected exception being raised during the call.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-dev: Allow passing more suite args

Signed-off-by: Zack Cerza <zack@redhat.com>

util.time: Add parse_timestamp

And move the format string to the time module.

Signed-off-by: Zack Cerza <zack@redhat.com>

util.time: Add new submodule with parse_offset()

Signed-off-by: Zack Cerza <zack@redhat.com>