task.ansible: Fix an IndexError w/ 0 remotes

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1816 from ceph/paddles-fault-tolerance

supervisor: Ignore some reporting failures

Specifically, the pings coming from the watchdog loop

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Do not time out when locking machines

Signed-off-by: Zack Cerza <zack@redhat.com>

safe_while: Retry forever when tries == -1

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1812 from ceph/orch-ssh-key

Merge pull request #1814 from NitzanMordhai/wip-nitzan-scrapy-deadjobs-nonetype-match

scrapy: comparing dead job skip typeNone

Merge pull request #1793 from cybozu/fix-docker-compose-postgres

Specify PostgreSQL v14 to avoid permission errors.

scrapy: comparing dead job skip typeNone
SequenceMatcher can't get NoneType when comparing 2 dead jobs,
adding check and compare for those jobs.

Fixes: https://tracker.ceph.com/issues/58264
Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>

orchestra.connection: Look for 'ssh_key' in config

On systems where SSH is configured using ~/.ssh/config.d/, we don't have
a great way of locating and processing the full configuration, so we
might not find the correct key to use. Allow the user to specify one key
to use for all test nodes if they wish.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1811 from ceph/wip-58491

misc.get_system_type: Use Remote.os

misc.get_system_type: Use Remote.os

... instead of lsb_release, as newer distros don't ship it.

Fixes: #58491
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1810 from ceph/selinux-denial-fix

task/selinux: Fix regressed grepping of audit logs

task/selinux: Fix regressed grepping of audit logs

Fixes: https://tracker.ceph.com/issues/58610
Introduced: a106217
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1805 from ceph/el-py39

bootstrap: Correct python3 package names for EL8/9

Merge pull request #1807 from kshtsk/wip-update-leap

orchestra/opsys: add newer sle and leap versions

orchestra/opsys: add newer sle and leap versions

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstrap: Correct python3 package names for EL8/9

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1806 from ceph/fix-pytest

tests: Fix warnings, deprecations, and broken tests

Merge pull request #1802 from ceph/faster-newest

suite: Speed up 'newest' feature

tox.ini: Update to reflect passenv change

See the second bullet point here:
https://tox.wiki/en/4.0.3/faq.html#tox-4-changed-ini-rules

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix bugs revealed by mock's recent change

Once the asserts were corrected, these tests started failing.
Fortunately, they didn't seem to be masking any actual bugs.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix broken Mock assert statements

This breaks (or, rather, reveals the broken state of) several unit
tests. A subsequent commit will fix them.

See https://github.com/testing-cabal/mock/commit/21787a948cf1186f3bd38462d5b912784fa81218

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Make sure to patch repo_utils ops

These tests were hitting the network.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Rename setup() and teardown() methods

These were nose methods, and pytest is dropping support for them.

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix 'invalid escape sequence' deprecation warnings

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1799 from dmick/wip-bootstrap

bootstrap: rpm systems: use --whatprovides to test for package presence

Merge pull request #1795 from dmick/wip-log-fog-exceptions

provision/fog.py: log exceptions while looping for ssh

bootstrap: rpm systems: use --whatprovides to test for package presence

the package name isn't necessarily the name of the dependency

Signed-off-by: Dan Mick <dmick@redhat.com>

provision/fog.py: log exceptions while looping for ssh

Make sure the user knows of potential configuration problems, etc.

Fixes: https://tracker.ceph.com/issues/58015
Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1797 from ceph/unique-dispatcher

dispatcher: Refuse to start extra dispatchers

suite: Speed up 'newest' feature

For a 'newest' value of N, we were calling find_git_parent N times.
find_git_parent was invoking githelper's 'refresh' endpoint each time,
which in my testing took 20-30 seconds. So, for a sha1 that needed 20
backtracks, I was seeing teuthology-suite take eight minutes to
complete. With this change, it took 30 seconds for the same
teuthology-suite invocation.

We were also not including the trailing slash on the 'refresh' endpoint,
resulting in a 308 redirect each time - doubling the request count.
That's fixed as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Refuse to start extra dispatchers

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1798 from ceph/wip-py38

Drop Python 3.6; Allow choosing Python version at runtime

Update packaging

"pyproject-api 1.2.1 has requirement packaging>=21.3, but you have
packaging 21.0."

Signed-off-by: Zack Cerza <zack@redhat.com>

Update tomli

This fixes a CI breakage caused by inconsistent dependencies,
specifically: "pyproject-api 1.2.1 has requirement tomli>=2.0.1;
python_version < "3.11", but you have tomli 1.2.1."

Signed-off-by: Zack Cerza <zack@redhat.com>

Update pip-tools

The previous version was incompatible with newer pip versions!

Signed-off-by: Zack Cerza <zack@redhat.com>

Default to python 3.8

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Pin postgres to v14

Starting with v15, paddles is broken out of the box. We'll need to fix
that, of course, but not here.

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop Python 3.6 as a supported version

It was EOLed a year ago.

Signed-off-by: Zack Cerza <zack@redhat.com>

.github: Test with more Python versions

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Rebuild wrong-python venvs

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Use --use-pep517 when installing reqs

See https://github.com/pypa/pip/issues/8559

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Support alternate python versions

And check for their usability before proceeding

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1794 from ceph/githelper

config: Update githelper URL

config: Update githelper URL

Signed-off-by: Zack Cerza <zack@redhat.com>

Specify PostgreSQL v14 to avoid permission errors.

PostgreSQL v15  has a notable change that revokes the CREATE permission
from users except a database owner. So we can not execute paddles using
it. As a workaround, I specify PostgreSQL version.

ref: https://www.postgresql.org/about/news/postgresql-15-released-2526/

Signed-off-by: Yuji Ito <llamerada.jp@gmail.com>

Merge pull request #1790 from ceph/no-devtmpfs-scratch-devs

misc.get_scratch_devices(): Ignore devtmpfs

Devices shared with a container by its host will appear in /proc/mounts
this way.

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.get_scratch_devices(): Split up checks

For better logging.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1786 from rishabh-d-dave/orch-mount-method

Merge pull request #1789 from ktdreyer/fog-error-suggestion

fog: suggest image names on error

Merge pull request #1788 from ktdreyer/lock-help

lock: remove --list --brief example

orchestra: add a method to check if given path is mounted

Add a method mount() that checks if the given path is mounted or
unmounted without using "mount" of "findmnt" commands.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

fog: suggest image names

Improve the error message when the user specifies a missing OS version
for a machine type.

Old error message:

  RuntimeError: Could not find an image for rhel 7.9

New error message:

  RuntimeError: Fog has no cali_rhel_7.9 image.
                Available cali images: ['cali_rhel_8.6', 'cali_rhel_9.0']

lock: remove --list --brief example

The --list and --brief arguments are exclusive.

  teuthology-lock: error: argument --brief: not allowed
  with argument --list

Merge pull request #1719 from kamoltat/wip-ksirivad-fix-53206

task/internal: open gzip file if core is compressed
Reviewed by: Ilya Dryomov <idryomov@redhat.com>
Reviewed by: Zack Cerza <zcerza@redhat.com>

Merge pull request #1787 from kamoltat/wip-ksirivad-boostrap-centosstream

boostrap: Add support for CentOSStream

boostrap: Add support for CentOSStream

currently we cannot do boostrap for CentOSStream
so we are adding support for that.

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

task/internal/__init__.py: improve core program extraction

Problem:
The previous technique for extracting the program name
of the coredump file doesn't deal with the case
where there is no white space after program name
e.g.,

``SVR4-style, from 'ceph_test_rados_api_io', real uid: 1000``

here we will get an error when we reach the part where we

``remote.sh(['which', dump_program]).rstrip()``.

Solution:

Use regex to extract first occurence of the word that
is in the format ``from '<word>'``.

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1785 from ceph/jammy

opsys.py: Make Ubuntu 22.04 default

test_default_distro_version: Update to 22.04

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1784 from ceph/wip-57092

misc._ssh_keyscan(): Drop '-t rsa'

opsys.py: Make Ubuntu 22.04 default

Signed-off-by: David Galloway <dgallowa@redhat.com>

misc._ssh_keyscan(): Drop '-t rsa'

It appears to be unnecessary, and causes Ubuntu 22.04 to respond with
zero keys.

Signed-off-by: Zack Cerza <zack@redhat.com>

task/tests: init pytest test_fetch_core_dumps

Create a unittest to test the function:
`fetch_binaries_for_coredumps()`.

Testing:

1. uncompressed coredump and correct format.
2. uncompressed coredump and incorrect format.
3. compressed coredump and correct format.
4. compressed coredump and incorrect format.

Fixes: https://tracker.ceph.com/issues/53206
Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1774 from ceph/nuke-typerror

nuke: Tolerate null node description
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

nuke: Tolerate null node description

Fixes a TypeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1783 from zdover23/wip-doc-2022-08-04-fragment-merging-typo

docs/frag...merging.rst - s/contructed/constructed
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

docs/frag...merging.rst - s/contructed/constructed

This PR corrects "contructed" to "constructed".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #1781 from Ngwind/patch-1

Update COMPONENTS.rst

Merge pull request #1782 from ceph/jammy

opsys.py: Add Ubuntu 22.04

opsys.py: Add Ubuntu 22.04

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1779 from ceph/dc-suite-fix

Fix scheduling regression

docs/docker-compose: Use Ubuntu Focal for testnode

I'm seeing ceph-cm-ansible failures w/ Jammy because of python-apt vs.
python3-apt. We'll revisit this soon.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1778 from ceph/no-mails-with-dryrun

suite: Do not send emails with --dry-run at schedule_fail
Reviewed by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Update COMPONENTS.rst

Fix `components.rst` file not being rendered

suite: add pytests for schedule_fail

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

choose_teuthology_branch: Choose main later on

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/docker-compose: Tolerate empty TEUTH_BRANCH

Signed-off-by: Zack Cerza <zack@redhat.com>

schedule_suite(): Make 'configs' a list

The var being left as a generator led to all jobs being filtered out of
some suites, rendering them broken.

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/docker-compose: Ensure we actually schedule

This should have failed: https://github.com/ceph/teuthology/pull/1776/checks

Signed-off-by: Zack Cerza <zack@redhat.com>

beanstalk: Print tube stats in JSON format

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1776 from batrick/yaml-preprocessor

teuthology: add pre/post fragment merge hooks

task/internal: open gzip file if core is compressed

Addded a case where if core is compressed, then
we will try to open it as gzip and read the content
inside.

We also add try and except when we are parsing
through the content to prevent the program from
crashing.

Fixes: https://tracker.ceph.com/issues/53206
Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

docs: document fragment merging

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite/test: add tests for fragment merging

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology: add lua based fragment merge scripting

As part of this change, there is a new generator design for producing
job configs. YAML fragments are memoized and merged manually to avoid
expensive and unnecessary parsing of the merged fragments. This provides
for a dramatic speedup in processing matrices with large numbers of
jobs. For rados suite with --subset 1/1000, this branch is 5x faster
(77s vs. 15s). (Note: the difference shrinks when there are fewer or
jobs or larger subsets are used due to cycling and the matrix generation
dominating runtime.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: support MappingProxyType in deep_merge

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: avoid copying references of b when list or dict

Otherwise, if b persists in memory (in a cache), it may wrongly
accumulate future merges.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1777 from VallariAg/log-total-jobs

suite/run: Log total number of jobs scheduled

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

suite: Do not send emails with --dry-run at schedule_fail

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #1769 from zmc/docs-missing-commands

docs/commands: Add missing commands
Reviewed by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1775 from ceph/update-reqs

Update pip-tools and sentry-sdk

Update sentry-sdk

We've started seeing what appear to be duplicate issues:
https://tracker.ceph.com/issues/56410
Hoping a simple update will resolve this...

Signed-off-by: Zack Cerza <zack@redhat.com>

Update pip-tools

Fixes https://github.com/jazzband/pip-tools/issues/1617

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1772 into master

* refs/pull/1772/head:
Revert "Merge pull request #1624 from batrick/get-valgrind-args"

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Revert "Merge pull request #1624 from batrick/get-valgrind-args"

This reverts commit 40fff4b2911949847a302adfe2f4714cfaccb58e, reversing
changes made to 93d2b653facc0e5a7a532b39d9ff91a044f2c4a9.

Merge pull request #1770 from ceph/plow