systemd test

Revert "testnode: try centos9 again"

This reverts commit fc94a550e4c426faae9bba4631d476221f242630.

testnode: try centos9 again

testnode dockerfile sed format

testnode container changes

testnode call systemd explicitly

test_locking: Tolerate CentOS Stream's versioning

It's not a problem functionally, so explicitly allow it in the unit
test.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuth_suite none

containers/testnode: no_cgroups_split

testnode install lsb_release

crun

drop catatonit

testnode: try centos 8.stream

testnode kill init

testnode drop fuse, add catatonit

teuth.sh drop filter

containers/teuthology-dev: Parametrize suite

containers/testnode: osd_method

containers/testnode: Install pkgs for ceph tests

containers/testnode: Write /scratch_devs

containers/testnode: Manage a few items in /run/

bc of systemd

containers/testnode: Use systemd

containers/testnode: Support nesting

containers/testnode: Install podman

To get a recent version we have to switch OSes.

containers/teuthology: Pin base image to focal

containers/teuthology: Shorten watchdog_interval

TEMP? teuth-dev dockerfile rm ansible

TEMP cephadm image

containers: Use secrets for SSH keys, not env vars

FIXME docs/docker-compose: Move to new containers/ dir

the dir is not yet empty

dispatcher: Log the path to the supervisor logfile

Since we're no longer logging that output from the dispatcher, make it
easy to find the supervisor log.

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Keep job output out of stdout/err

If we want the dispatcher to run under systemd, let's keep its output a
bit more terse. The logfile will still be available.

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Exit gracefully on SIGTERM

Well, relatively soon after receiving it.

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Log why we're stopping

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1814 from NitzanMordhai/wip-nitzan-scrapy-deadjobs-nonetype-match

scrapy: comparing dead job skip typeNone

Merge pull request #1793 from cybozu/fix-docker-compose-postgres

Specify PostgreSQL v14 to avoid permission errors.

scrapy: comparing dead job skip typeNone
SequenceMatcher can't get NoneType when comparing 2 dead jobs,
adding check and compare for those jobs.

Fixes: https://tracker.ceph.com/issues/58264
Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>

Merge pull request #1811 from ceph/wip-58491

misc.get_system_type: Use Remote.os

misc.get_system_type: Use Remote.os

... instead of lsb_release, as newer distros don't ship it.

Fixes: #58491
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1810 from ceph/selinux-denial-fix

task/selinux: Fix regressed grepping of audit logs

task/selinux: Fix regressed grepping of audit logs

Fixes: https://tracker.ceph.com/issues/58610
Introduced: a106217
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1805 from ceph/el-py39

bootstrap: Correct python3 package names for EL8/9

Merge pull request #1807 from kshtsk/wip-update-leap

orchestra/opsys: add newer sle and leap versions

orchestra/opsys: add newer sle and leap versions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstrap: Correct python3 package names for EL8/9

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1806 from ceph/fix-pytest

tests: Fix warnings, deprecations, and broken tests

Merge pull request #1802 from ceph/faster-newest

suite: Speed up 'newest' feature

tox.ini: Update to reflect passenv change

See the second bullet point here:
https://tox.wiki/en/4.0.3/faq.html#tox-4-changed-ini-rules

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix bugs revealed by mock's recent change

Once the asserts were corrected, these tests started failing.
Fortunately, they didn't seem to be masking any actual bugs.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix broken Mock assert statements

This breaks (or, rather, reveals the broken state of) several unit
tests. A subsequent commit will fix them.

See https://github.com/testing-cabal/mock/commit/21787a948cf1186f3bd38462d5b912784fa81218

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Make sure to patch repo_utils ops

These tests were hitting the network.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Rename setup() and teardown() methods

These were nose methods, and pytest is dropping support for them.

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix 'invalid escape sequence' deprecation warnings

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1799 from dmick/wip-bootstrap

bootstrap: rpm systems: use --whatprovides to test for package presence

Merge pull request #1795 from dmick/wip-log-fog-exceptions

provision/fog.py: log exceptions while looping for ssh

bootstrap: rpm systems: use --whatprovides to test for package presence

the package name isn't necessarily the name of the dependency

Signed-off-by: Dan Mick <dmick@redhat.com>

provision/fog.py: log exceptions while looping for ssh

Make sure the user knows of potential configuration problems, etc.

Fixes: https://tracker.ceph.com/issues/58015
Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1797 from ceph/unique-dispatcher

dispatcher: Refuse to start extra dispatchers

suite: Speed up 'newest' feature

For a 'newest' value of N, we were calling find_git_parent N times.
find_git_parent was invoking githelper's 'refresh' endpoint each time,
which in my testing took 20-30 seconds. So, for a sha1 that needed 20
backtracks, I was seeing teuthology-suite take eight minutes to
complete. With this change, it took 30 seconds for the same
teuthology-suite invocation.

We were also not including the trailing slash on the 'refresh' endpoint,
resulting in a 308 redirect each time - doubling the request count.
That's fixed as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Refuse to start extra dispatchers

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1798 from ceph/wip-py38

Drop Python 3.6; Allow choosing Python version at runtime

Update packaging

"pyproject-api 1.2.1 has requirement packaging>=21.3, but you have
packaging 21.0."

Signed-off-by: Zack Cerza <zack@redhat.com>

Update tomli

This fixes a CI breakage caused by inconsistent dependencies,
specifically: "pyproject-api 1.2.1 has requirement tomli>=2.0.1;
python_version < "3.11", but you have tomli 1.2.1."

Signed-off-by: Zack Cerza <zack@redhat.com>

Update pip-tools

The previous version was incompatible with newer pip versions!

Signed-off-by: Zack Cerza <zack@redhat.com>

Default to python 3.8

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Pin postgres to v14

Starting with v15, paddles is broken out of the box. We'll need to fix
that, of course, but not here.

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop Python 3.6 as a supported version

It was EOLed a year ago.

Signed-off-by: Zack Cerza <zack@redhat.com>

.github: Test with more Python versions

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Rebuild wrong-python venvs

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Use --use-pep517 when installing reqs

See https://github.com/pypa/pip/issues/8559

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Support alternate python versions

And check for their usability before proceeding

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1794 from ceph/githelper

config: Update githelper URL

config: Update githelper URL

Signed-off-by: Zack Cerza <zack@redhat.com>

Specify PostgreSQL v14 to avoid permission errors.

PostgreSQL v15  has a notable change that revokes the CREATE permission
from users except a database owner. So we can not execute paddles using
it. As a workaround, I specify PostgreSQL version.

ref: https://www.postgresql.org/about/news/postgresql-15-released-2526/

Signed-off-by: Yuji Ito <llamerada.jp@gmail.com>

Merge pull request #1790 from ceph/no-devtmpfs-scratch-devs

misc.get_scratch_devices(): Ignore devtmpfs

Devices shared with a container by its host will appear in /proc/mounts
this way.

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.get_scratch_devices(): Split up checks

For better logging.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1786 from rishabh-d-dave/orch-mount-method

Merge pull request #1789 from ktdreyer/fog-error-suggestion

fog: suggest image names on error

Merge pull request #1788 from ktdreyer/lock-help

lock: remove --list --brief example

orchestra: add a method to check if given path is mounted

Add a method mount() that checks if the given path is mounted or
unmounted without using "mount" of "findmnt" commands.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

fog: suggest image names

Improve the error message when the user specifies a missing OS version
for a machine type.

Old error message:

  RuntimeError: Could not find an image for rhel 7.9

New error message:

  RuntimeError: Fog has no cali_rhel_7.9 image.
                Available cali images: ['cali_rhel_8.6', 'cali_rhel_9.0']

lock: remove --list --brief example

The --list and --brief arguments are exclusive.

  teuthology-lock: error: argument --brief: not allowed
  with argument --list

Merge pull request #1719 from kamoltat/wip-ksirivad-fix-53206

task/internal: open gzip file if core is compressed
Reviewed by: Ilya Dryomov <idryomov@redhat.com>
Reviewed by: Zack Cerza <zcerza@redhat.com>

Merge pull request #1787 from kamoltat/wip-ksirivad-boostrap-centosstream

boostrap: Add support for CentOSStream

boostrap: Add support for CentOSStream

currently we cannot do boostrap for CentOSStream
so we are adding support for that.

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

task/internal/__init__.py: improve core program extraction

Problem:
The previous technique for extracting the program name
of the coredump file doesn't deal with the case
where there is no white space after program name
e.g.,

``SVR4-style, from 'ceph_test_rados_api_io', real uid: 1000``

here we will get an error when we reach the part where we

``remote.sh(['which', dump_program]).rstrip()``.

Solution:

Use regex to extract first occurence of the word that
is in the format ``from '<word>'``.

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1785 from ceph/jammy

opsys.py: Make Ubuntu 22.04 default

test_default_distro_version: Update to 22.04

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1784 from ceph/wip-57092

misc._ssh_keyscan(): Drop '-t rsa'

opsys.py: Make Ubuntu 22.04 default

Signed-off-by: David Galloway <dgallowa@redhat.com>

misc._ssh_keyscan(): Drop '-t rsa'

It appears to be unnecessary, and causes Ubuntu 22.04 to respond with
zero keys.

Signed-off-by: Zack Cerza <zack@redhat.com>

task/tests: init pytest test_fetch_core_dumps

Create a unittest to test the function:
`fetch_binaries_for_coredumps()`.

Testing:

1. uncompressed coredump and correct format.
2. uncompressed coredump and incorrect format.
3. compressed coredump and correct format.
4. compressed coredump and incorrect format.

Fixes: https://tracker.ceph.com/issues/53206
Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1774 from ceph/nuke-typerror

nuke: Tolerate null node description
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

nuke: Tolerate null node description

Fixes a TypeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1783 from zdover23/wip-doc-2022-08-04-fragment-merging-typo

docs/frag...merging.rst - s/contructed/constructed
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

docs/frag...merging.rst - s/contructed/constructed

This PR corrects "contructed" to "constructed".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #1781 from Ngwind/patch-1

Update COMPONENTS.rst

Merge pull request #1782 from ceph/jammy

opsys.py: Add Ubuntu 22.04

opsys.py: Add Ubuntu 22.04

Signed-off-by: David Galloway <dgallowa@redhat.com>