teuthology/repo_utils: Optimize repository cloning via bare repositories and worktrees

Repeatedly performing full clones of the QA suite and other repositories
for different commits or branches slows down testing iterations and
consumes excess disk space and I/O.

This commit introduces an optimization that initializes a single bare
clone of a target repository. It subsequently utilizes git worktree to
provision lightweight, isolated workspaces for requested commits. This
reduces redundant network fetches and significantly accelerates
repository checkout operations.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

teuthology/repo_utils: strip only .git suffix

Previous code with strip any trailing letters in .git.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #2205 from batrick/lupa-2.8

lupa: update to 2.8 and use lua 5.4

lupa: update to 2.8 and use lua 5.4

Do not incidentally upgrade Lua to 5.5.

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #2202 from Adarsha1999/fix/deb-apt-key-https

task/install/deb: fix apt key fetch hanging when http is blocked

task/install/deb: fix apt key fetch hanging when http is blocked

Signed-off-by: Adarsha Dinda <adarshadinda@Adarshas-MacBook-Pro.local>

Merge pull request #2198 from ceph/wip-55347

task/selinux: allowlist iptables changes

Merge pull request #2200 from ceph/update-deps

Update urllib3

Merge pull request #2199 from ceph/quiet-bootstrap

bootstrap: Make dependency install less verbose

Merge pull request #2194 from ceph/dependabot/uv/jinja2-3.1.6

build(deps): bump jinja2 from 3.1.5 to 3.1.6

Update urllib3

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: Make dependency install less verbose

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2196 from kshtsk/wip-interruptable-dispatcher

Cleanup dispatcher code

task/selinux: allowlist iptables changes

Fixes: https://tracker.ceph.com/issues/55347
Signed-off-by: David Galloway <david.galloway@ibm.com>

dispatcher: save initial config as 'orig.config.yaml' again

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2195 from ceph/update-deps

uv.lock --upgrade

dispatcher: save default job config to a separate file

In order to still make it possible to reuse original config
from the queue, don't be confused by 'orig.config.yaml',
which is used in teuthology.run after joining arguments.

So we save it to 'default.config.yaml', and later can
be used like:

```
teuthology path/to/default.config.yaml path/to/targets.yaml
```

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

dispatcher/test: fix config arg

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

supervisor: reuse provided config path to run a job

The supervisor just ignored --job-config argument when
it came to run a job and provided hard coded 'orig.config.yaml'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

dispatcher: cleanup create_job_archive

Get rid of create_job_archive and other cleanups.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

dispatcher: do not fetch main teuthology and qa

Because each run have can have own branch and sha1,
no need to fetch main for teuthology.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

dispatcher: cleanup from result_proc

The teuthology-results is not run in dispatcher anymore,
instead it is started in from the supervisor.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Remove a few unuse dependencies

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock --upgrade

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2188 from ceph/update-deps

Update greenlet

build(deps): bump jinja2 from 3.1.5 to 3.1.6

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.5 to 3.1.6.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.5...3.1.6)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-version: 3.1.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #2189 from ceph/proj-cleanup

Remove requirements.txt and MANIFEST.in

Merge pull request #2187 from kshtsk/wip-overrides-order

Fix overrides order

Remove MANIFEST.in

Signed-off-by: Zack Cerza <zack@cerza.org>

Remove requirements.txt

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade greenlet

Signed-off-by: Zack Cerza <zack@cerza.org>

provision/downburst: preinstall bindutils for rocky to

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

suite: fix order of yaml files overriding

For correct overrides which can be provided in extra yaml arguments
to teuthology-suites the joined yaml of job_config from suite directory
must go first in order.

Fixes: 96f38f2136dd2a4f64cc741919b85dc65a9df536
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2184 from ceph/uv-updates

uv.lock: Update dependencies

Merge pull request #2186 from ceph/rm-ci-workaround

workflows/dependencies: Remove workaround

workflows/dependencies: Remove workaround

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade pytest

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade lxml

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade pyopenssl

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade pygments

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade pyasn1

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade pyjwt

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade cryptography

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Upgrade requests

Signed-off-by: Zack Cerza <zack@cerza.org>

uv.lock: Update to revision 3

This was done by running 'uv lock'.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2185 from ceph/fix-ci

Fix dependencies CI

workflows/dependencies: Install pkgs explicitly

This is to temporarily work around quoting bug in bootstrap, since we run main's
copy first.

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: Install debs quietly

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: Fix quoting

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: Fix apt warning

Signed-off-by: Zack Cerza <zack@cerza.org>

workflows/dependencies: Quiet apt-get

Signed-off-by: Zack Cerza <zack@cerza.org>

workflows/dependencies: Correct venv name

Signed-off-by: Zack Cerza <zack@cerza.org>

workflows/dependencies: Drop virtualenv step

Signed-off-by: Zack Cerza <zack@cerza.org>

workflows/dependencies: Drop apt upgrade step

This was updating all of GNOME, among other things

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2174 from ceph/exporter-fix

exporter: Use own restart method

Merge pull request #2161 from kshtsk/uv2

Move to uv

exporter: Use own restart method

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2172 from ceph/gha-container-merge-fix

workflows/dev_container: Fix merge stage

teuthology: load importlib after monkey patching

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@gmail.com>

bootstrap: make zypper call non-interactive

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@gmail.com>

bootstrap: use "dpkg -s" again

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@gmail.com>

uv.lock: update deps for oauthlib

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

pyproject: add missing oauthlib dep for maas

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

bootstrap: Drop lsb_release support

And also drop an unnecessary tr invocation.

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: fix suse family deps

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

teuthology/Dockerfile: add env setuptools-scm version

Because the teuthology code is building in container after
copying source code from local directory, but no corresponding
git repo copied, the setuptools-scm cannot determine teuthology
version. So in order to produce the package we just provide dummy
version 0.0.0 using environment variable:

  SETUPTOOLS_SCM_PRETEND_VERSION_FOR_TEUTHOLOGY

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

pyproject.toml: use setuptools-scm for uv

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

tox: Use uv to run pytest and sphinx

Signed-off-by: Zack Cerza <zack@cerza.org>

Fix errors reported by ruff

Signed-off-by: Zack Cerza <zack@cerza.org>

Drop flake8 in favor of ruff

For now, use a very narrow ruleset. We should expand to use more later.

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: Fix shellcheck warnings

Conflicts:
        bootstrap

Update install doc to cover uv

Signed-off-by: Zack Cerza <zack@cerza.org>

Use uv's venv path

Signed-off-by: Zack Cerza <zack@cerza.org>

containers: Install pipx

Signed-off-by: Zack Cerza <zack@cerza.org>

Simplify bootstrap script

Much of this is simply removing things we don't have to be doing now that we're
using uv. It also consolidates the different sections for RPM-based distros.

Signed-off-by: Zack Cerza <zack@cerza.org>

Update gevent

To a version that support python 3.13

Signed-off-by: Zack Cerza <zack@cerza.org>

tox: Use tox-uv

Signed-off-by: Zack Cerza <zack@cerza.org>

Drop pip-tools

Signed-off-by: Zack Cerza <zack@cerza.org>
Conflicts:
requirements.txt

Replace pip with uv

Signed-off-by: Zack Cerza <zack@cerza.org>
Conflicts:
requirements.txt
setup.cfg

teuthology-dev: Use args form for entrypoint

This fixes a warning issued by docker/build-push-action

Signed-off-by: Zack Cerza <zack@cerza.org>

workflows/dev_container: Fix merge stage

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2169 from kshtsk/wip-build-essential-missing

containers: add new dependency build-essential

containers: add new dependency build-essential

Fixes: 252ab4f6a44c0d64b1e4cfb2b86e9bf58b5d9ffb
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2171 from zmc/gha-no-push-prs

workflows/dev_container: Don't push for PRs

workflows/dev_container: Don't push for PRs

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2141 from ceph/container-manifest

workflows/dev_container: Fix multi-arch images

Merge PR #2159 into main

* refs/pull/2159/head:
teuthology/suite: log postmerge filtering

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #2160 from Adarsha1999/openstack-rocky10-user-data

openstack: Add cloud-init user-data for Rocky Linux 10 and 10.1

openstack: Add cloud-init user-data for Rocky Linux 10 and 10.1

teuthology/suite: log postmerge filtering

Otherwise it's hard to discern what caused a job to be dropped. The
"postmerge" script itself may be empty but the other filtering still
runs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #2158 from ceph/utf8-fix

bootstrap: Support nonstandard locales

bootstrap: Support nonstandard locales

Some newer systems use e.g. LANG=C.utf8, which is breaking bootstrap since it expects to see e.g. C.utf-8. Instead of trying to parse the values, simply split on '.' and append 'utf-8'.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2155 from tchaikov/seed

schedule: fix first-in-suite option help

schedule: fix first-in-suite option help

The schedule command validates --seed, --subset and
--no-nested-subset together with --first-in-suite, and the suite
runner passes them that way when writing the rerun memo.

Update the help text to match the implemented behavior.

Reported-by: T K Chandra Hasan <t.k.chandra.hasan@ibm.com>
Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #2150 from anshuman-agarwala/python-13-bump

requirements: bumped packages for python3.13

Bumped packages for python3-13

Merge pull request #2148 from ceph/no-ref-sha1

rpm.py: Do not sed ref/sha1

rpm.py: Do not sed ref/sha1

This is a relic from gitbuilder days and causes issues if a dev puts "ref" in the branch name

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #2146 from ceph/reboot-7min

fog: Try ipmi power-cycle if stuck in a reimage reboot hang

Merge pull request #2147 from batrick/kernel-fix

teuthology/task/kernel: always hard reboot