bootstrap: Drop lsb_release support

And also drop an unnecessary tr invocation.

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: fix suse family deps

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>

teuthology/Dockerfile: add env setuptools-scm version

Because the teuthology code is building in container after
copying source code from local directory, but no corresponding
git repo copied, the setuptools-scm cannot determine teuthology
version. So in order to produce the package we just provide dummy
version 0.0.0 using environment variable:

  SETUPTOOLS_SCM_PRETEND_VERSION_FOR_TEUTHOLOGY

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

pyproject.toml: use setuptools-scm for uv

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

tox: Use uv to run pytest and sphinx

Signed-off-by: Zack Cerza <zack@cerza.org>

Fix errors reported by ruff

Signed-off-by: Zack Cerza <zack@cerza.org>

Drop flake8 in favor of ruff

For now, use a very narrow ruleset. We should expand to use more later.

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: Fix shellcheck warnings

Update install doc to cover uv

Signed-off-by: Zack Cerza <zack@cerza.org>

Use uv's venv path

Signed-off-by: Zack Cerza <zack@cerza.org>

containers: Install pipx

Signed-off-by: Zack Cerza <zack@cerza.org>

Simplify bootstrap script

Much of this is simply removing things we don't have to be doing now that we're
using uv. It also consolidates the different sections for RPM-based distros.

Signed-off-by: Zack Cerza <zack@cerza.org>

Update gevent

To a version that support python 3.13

Signed-off-by: Zack Cerza <zack@cerza.org>

tox: Use tox-uv

Signed-off-by: Zack Cerza <zack@cerza.org>

Drop pip-tools

Signed-off-by: Zack Cerza <zack@cerza.org>

Replace pip with uv

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2037 from ceph/suite-distro

suite: Filter YAML fragments for distro, version

Merge pull request #2089 from ceph/ulrif

lock.ops.lock_many: Unlock on reimage failure

Merge pull request #2103 from ceph/wip-extra-system-packages-fixup

task/install: extra_system_packages at the top level can be a list

task/install: don't attempt to handle strings for extra_system_packages

To the best of my knowledge, extra_system_packages has never accepted
a string as a value -- whether at the top level (i.e. on the install
task itself) or on a particular project (e.g. ceph).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

task/install: extra_system_packages at the top level can be a list

Commit fc5a3197f22e ("task/install: allow both list and dicts for
extra_system_packages") introduced a regression for job definitions
where extra_system_packages at the top level (i.e. on the install
task itself) is a list.  For example

  overrides:
    install:
      ceph:
        flavor: default
        sha1: b7d57fefbec68934ae5e6e7d7c48ac09182397a5
      extra_system_packages:
      - pv

produces AssertionError on

  assert isinstance(b, dict)

in deep_merge() when it's called from _override_extra_system_packages().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #2099 from kshtsk/wip-packaging-tests

Fix packaging tests for rocky and friends

orchestra/test: drop opensuse 42.2 from opsys

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@gmail.com>

packaging: fix test for distro version

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: update suse related versions

Signed-off-by: Kyrylo Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2096 from ceph/wip-greg-rocky

cargo-cult some packaging updates for rocky

cargo-cult some packaging updates for rocky

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2084 from kshtsk/wip-install-repos-rpm

support install repos for rpm

Merge pull request #2092 from kshtsk/wip-extra-system-packages

task/install: allow both list and dicts for extra_system_packages

task/install: allow both list and dicts for extra_system_packages

Fixes: https://tracker.ceph.com/issues/73527
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2088 from deepssin/openstack_config

Fix: handle attached_volumes field when fetching volumes from openstack

Openstack: handle attached_volumes field when fetching volumes

Some clouds use attached_volumes instead of os-extended-volumes:volumes_attached

Signed-off-by: deepssin <deepssin@redhat.com>

Merge pull request #2090 from dmick/increase-power-status-timeout

orchestra/console.py: increase 'power status' timeout

orchestra/console.py: increase 'power status' timeout

in set_power() we check to see if the power operation worked by
examining ipmi power status.  Give longer than 1s for the
status command to complete.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

suite.run: Fix parent commit search

We were only looking in ceph.git; look in e.g. ceph-ci.git if that is the ceph
repo being tested.

Signed-off-by: Zack Cerza <zack@cerza.org>

suite: Filter YAML fragments for distro, version

The behavior of --distro{,-version} was never well-defined. With this change,
they can independently be used to provide a default for jobs which don't
otherwise specify a value; and more importantly, perform a post-merge filter
looking at each job's values, rejecting those which don't match. This behavior
should be more intuitive and more easy to understand.

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2086 from ceph/os-opt

openstack: Do not immediately throw ImportError

orchestra/test: Ignore configured ssh key

This removes false negatives when a user has configued an ssh key.

Signed-off-by: Zack Cerza <zack@cerza.org>

openstack: Do not immediately throw ImportError

We would rather not have openstacksdk be a requirement for all installs, since
most do not use it. This change will not disturb people who install the
openstack extra, which pulls in that and other packages.

Signed-off-by: Zack Cerza <zack@cerza.org>

supervisor.reimage: Change error message wording

Signed-off-by: Zack Cerza <zack@cerza.org>

lock.ops.lock_many: Unlock on reimage failure

Signed-off-by: Zack Cerza <zack@cerza.org>

task/install/rpm: don't use builder when using custom repos

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/install/rpm: add/remove yum repos

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/opsys: alma and rocky are rpm distro

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2080 from kshtsk/wip-downburst-defaults

Update downburst defaults

test: update test_bad_version

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: update tests respecting latest defaults

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/remote: fix f-string rendering

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: update defaults for supported distros

Update defaults for the supported os_type/os_version and
instead of always returning default values return the table
received from previous call to `downburst list-json`.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2082 from ceph/arm64-runners

workflows/dev_container: Use native arm64 runners

setup.cfg: Restrict gevent to <25

This should work around the hangs we have been seeing in CI.

Signed-off-by: Zack Cerza <zack@cerza.org>

workflows/dev_container: Use native arm64 runners

Signed-off-by: Zack Cerza <zack@cerza.org>

Merge pull request #2077 from kshtsk/wip-downburst-packages

Introduce multiple machine type instead of single vps for downburst

provision/downburst: support multiple config per machine_type

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

docs/laptop/targets.sql: add semicolon to commit

Apparently semicolon needed after transaction,
otherwise it gets stuck in a transaction queue
and locks the table

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2075 from kshtsk/wip-teuthology-git-url

teuthology/config: add teuthology_git_url option

Merge pull request #2072 from ceph/update-gevent

requirements.txt: Update gevent

Merge pull request #2076 from kshtsk/wip-rocky-alma-9.6

orchestra/opsys: release shift to 9.6 for rocky and alma linux

orchestra/opsys: release shift to 9.6 for rocky and alma linux

The release 9.5 is gone and all files removed on RockyLinux
and AlmaLinux sites, so just update defaults to the latest.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: install bind-utils for host command on centos

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

provision/downburst: install pip for centos

By some reason, ansible is not install pip3 for centos:

  Unable to find any of pip3 to use.  pip needs to be installed.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

teuthology/config: add teuthology_git_url option

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2074 from kshtsk/wip-fix-revive-openstack

setup.cfg: fix teuthology-openstack dependency

setup.cfg: fix teuthology-openstack dependency

The PR #2050 "Revive teuthology-openstack" introduced new dependency
after refactoring partly of teuthology.openstack instance classes
to use api instead of command line. However the openstacksdk
added to wrong section options.extras_require.openstack, that
intended to be used for provisioning openstack nodes via libcloud,
which is optional and is not installable by default.

I.e. `pip install -e ".[openstack]"` must be used in order to get it.

However the refactoring forced unconditional openstack requirement.

That's why this dependency should go to options.install_requires in order
to make `pip install -e .` work again.

Fixes: 252ab4f6a44c0d64b1e4cfb2b86e9bf58b5d9ffb
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2073 from kshtsk/wip-fedora-4x

Fedora 4x

requirements.txt: Update gevent

Signed-off-by: Zack Cerza <zack@cerza.org>

bootstrap: Fedora does not have python3.12 anymore

Latest Fedora, 41 and 42, has python3.13 installed
by default, just use python3 to allow use default
version.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

bootstrap: add -q for rpm --whatprovides

The rpm -q option lost during refactoring which is obviously
needed when requesting the package name.

Fixes: c290c6851a7c10aaa4a5e18e1c14d750c338e339
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

Merge pull request #2068 from batrick/jouranlctl-all

tasks/internal/syslog: gather all journald entries

Merge pull request #2070 from deepssin/sec_rule

[Openstack]allow all TCP ingress to prevent node communication failures

Merge pull request #2069 from deepssin/openstack_issues

Add centos9 stream user data file & add centos9 support to openstack

Merge pull request #2067 from kshtsk/wip-remote-resolve-ip

orchestra/remote: add resolve_ip method

[Openstack]allow all TCP ingress to prevent node communication failures

Signed-off-by: deepssin <deepssin@redhat.com>

Add centos9 stream user data file & add centos9 support to openstack

Signed-off-by: deepssin <deepssin@redhat.com>

containers/teuthology-dev/Dockerfile: fix running bootstrap

The `git config --unset` returns 5 if there is missing variable supposed
to be unset. Just always return true for this command so the bootstrap
can proceed.

Fixes: 7cc6ba4838150acd17649381313093185075ecb0
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

bootstrap: fix syntax error

This patch addresses the error in the logs, like this:

  Using python: python3.12
  /usr/bin/lsb_release
  Deleting existing virtual environment
  Creating new venv at ./virtualenv
  ./bootstrap: line 131: [: missing `]'
  ./bootstrap: line 131: 12: command not found

Fixes: 0b5a5bc629180cf5318b77a8ecb4c131abbd057d
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

workflows/dependencies: apt update

Update before bootstrap

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

orchestra/remote: add resolve_ip method

Add utility method to resolve a hostname from within remote host.
This is useful to resolve ip address of the remote host itself,
because getting ip address from transport object of ssh is not
suitable because it may have only bastion host and port, which
is not relevant for the purpose of configuring a cluster.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>

task/internal/syslog: parallelize and debugify gzip of syslogs

To match what we do for the ceph task.

Looks like:

    2025-07-21T17:04:25.354 INFO:teuthology.task.internal.syslog:Compressing syslogs...
    2025-07-21T17:04:25.354 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose --
    2025-07-21T17:04:25.357 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose --
    2025-07-21T17:04:25.359 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose --
    2025-07-21T17:04:25.380 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
    2025-07-21T17:04:25.380 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
    2025-07-21T17:04:25.380 INFO:teuthology.orchestra.run.smithi117.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:   gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log
    2025-07-21T17:04:25.381 INFO:teuthology.orchestra.run.smithi117.stderr:  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
    2025-07-21T17:04:25.381 INFO:teuthology.orchestra.run.smithi117.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:     0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
    2025-07-21T17:04:25.385 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
    2025-07-21T17:04:25.385 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
    2025-07-21T17:04:25.385 INFO:teuthology.orchestra.run.smithi142.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:   gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log
    2025-07-21T17:04:25.385 INFO:teuthology.orchestra.run.smithi142.stderr:  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
    2025-07-21T17:04:25.385 INFO:teuthology.orchestra.run.smithi142.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:     0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
    2025-07-21T17:04:25.386 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
    2025-07-21T17:04:25.387 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
    2025-07-21T17:04:25.387 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log
    2025-07-21T17:04:25.387 INFO:teuthology.orchestra.run.smithi186.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:     0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
    2025-07-21T17:04:25.387 INFO:teuthology.orchestra.run.smithi186.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:     0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
    2025-07-21T17:04:25.423 INFO:teuthology.orchestra.run.smithi142.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log:      88.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz
    2025-07-21T17:04:25.426 INFO:teuthology.orchestra.run.smithi186.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log:      88.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz
    2025-07-21T17:04:25.502 INFO:teuthology.orchestra.run.smithi117.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log:      90.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

task/internal/syslog: gzip journal along with everything else

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tasks/internal/syslog: gather all journald entries

Soemtimes a machine crashes during tests and log entries around that time are
not persisted because we're only looking at this boot.

(This choice seem odd since the lifetime of these operating system images is
the same as the teuthology run.)

Now looks like:

    2025-07-21T15:01:22.249 INFO:teuthology.task.internal.syslog:Gathering journactl...
    2025-07-21T15:01:22.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log
    2025-07-21T15:01:22.254 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log
    2025-07-21T15:01:22.256 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log

From: /teuthology/pdonnell-2025-07-21_14:01:58-fs-main-distro-default-smithi/8399938/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #2066 from deepssin/issue-72055

Add openstack dependencies in docs/requirements

Add openstack dependencies in docs/requirements

Signed-off-by: deepssin <deepssin@redhat.com>

Merge pull request #2065 from ceph/job-flavor

Add flavor to job yaml (at Job creation)

Merge pull request #2064 from ceph/run-dev-container-ci

ci: run dev_container build for PRs

ci: run dev_container build for PRs

Build the image for PRs but do not push.
For pushes to main branch, push to registery.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Add flavor to job yaml (at Job creation)

Currently, when creating jobs, paddles
does not get Job.flavor information.
This leads to "flavor" getting set only
after the job is completed.

This commit adds "flavor" to job yaml config
so Job.flavor is set when creating jobs.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #2063 from deepssin/issue-72055

Add jq and curl to teuthology-dev Dockerfile

Add jq and curl to teuthology-dev Dockerfile

Signed-off-by: deepssin <deepssin@redhat.com>

Merge pull request #2062 from VallariAg/fix-container-image

containers: set image as ceph.containers.image

containers: set image as ceph.containers.image

instead of ceph.image

As cephadm task overrides from here: https://github.com/ceph/ceph/blob/ef15ecb8821fd79e24b046d2812696bd290c8489/qa/tasks/cephadm.py#L2301

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #2061 from ceph/wip-rocky10

opsys.py: Add Rocky 10

Merge pull request #2060 from ceph/wip-base-config-fix

Fix wrong os-type issue

Merge pull request #2050 from deepssin/enable-teuthology-openstack

Revive teuthology-openstack

tests: add unit test test_update_key and test_sha1

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

suite: Add reproducer for issue 71911

See: https://tracker.ceph.com/issues/71911

Signed-off-by: Zack Cerza <zack@cerza.org>
(cherry picked from commit be1d157d5a27e5eaaa791f674ee34e25d718ff61)

suite/run.py: update sha1 from basic_config to parsed_yaml

This is because during backtracking to find
build when using --newest, the new build/suite
sha is updated on basic_config. And job's parsed_yaml
is never updated with new sha.

Fixes: https://tracker.ceph.com/issues/71612
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

opsys.py: Add Rocky 10

Signed-off-by: David Galloway <david.galloway@ibm.com>

Revert "teuthology/schedule.py: update parsed_yaml with base_config"

This reverts commit 6c356472c37fc4334749d744738ff1ed4ebb031c.

This commit caused a regression: https://tracker.ceph.com/issues/71911

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Add jq and curl in docker-compose Dockerfile

Signed-off-by: deepssin <deepssin@redhat.com>