bootstrap: Correct python3 package names for EL8/9

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1806 from ceph/fix-pytest

tests: Fix warnings, deprecations, and broken tests

Merge pull request #1802 from ceph/faster-newest

suite: Speed up 'newest' feature

tox.ini: Update to reflect passenv change

See the second bullet point here:
https://tox.wiki/en/4.0.3/faq.html#tox-4-changed-ini-rules

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix bugs revealed by mock's recent change

Once the asserts were corrected, these tests started failing.
Fortunately, they didn't seem to be masking any actual bugs.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Fix broken Mock assert statements

This breaks (or, rather, reveals the broken state of) several unit
tests. A subsequent commit will fix them.

See https://github.com/testing-cabal/mock/commit/21787a948cf1186f3bd38462d5b912784fa81218

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Make sure to patch repo_utils ops

These tests were hitting the network.

Signed-off-by: Zack Cerza <zack@redhat.com>

tests: Rename setup() and teardown() methods

These were nose methods, and pytest is dropping support for them.

Signed-off-by: Zack Cerza <zack@redhat.com>

Fix 'invalid escape sequence' deprecation warnings

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1799 from dmick/wip-bootstrap

bootstrap: rpm systems: use --whatprovides to test for package presence

Merge pull request #1795 from dmick/wip-log-fog-exceptions

provision/fog.py: log exceptions while looping for ssh

bootstrap: rpm systems: use --whatprovides to test for package presence

the package name isn't necessarily the name of the dependency

Signed-off-by: Dan Mick <dmick@redhat.com>

provision/fog.py: log exceptions while looping for ssh

Make sure the user knows of potential configuration problems, etc.

Fixes: https://tracker.ceph.com/issues/58015
Signed-off-by: Dan Mick <dmick@redhat.com>

Merge pull request #1797 from ceph/unique-dispatcher

dispatcher: Refuse to start extra dispatchers

suite: Speed up 'newest' feature

For a 'newest' value of N, we were calling find_git_parent N times.
find_git_parent was invoking githelper's 'refresh' endpoint each time,
which in my testing took 20-30 seconds. So, for a sha1 that needed 20
backtracks, I was seeing teuthology-suite take eight minutes to
complete. With this change, it took 30 seconds for the same
teuthology-suite invocation.

We were also not including the trailing slash on the 'refresh' endpoint,
resulting in a 308 redirect each time - doubling the request count.
That's fixed as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Refuse to start extra dispatchers

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1798 from ceph/wip-py38

Drop Python 3.6; Allow choosing Python version at runtime

Update packaging

"pyproject-api 1.2.1 has requirement packaging>=21.3, but you have
packaging 21.0."

Signed-off-by: Zack Cerza <zack@redhat.com>

Update tomli

This fixes a CI breakage caused by inconsistent dependencies,
specifically: "pyproject-api 1.2.1 has requirement tomli>=2.0.1;
python_version < "3.11", but you have tomli 1.2.1."

Signed-off-by: Zack Cerza <zack@redhat.com>

Update pip-tools

The previous version was incompatible with newer pip versions!

Signed-off-by: Zack Cerza <zack@redhat.com>

Default to python 3.8

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Pin postgres to v14

Starting with v15, paddles is broken out of the box. We'll need to fix
that, of course, but not here.

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop Python 3.6 as a supported version

It was EOLed a year ago.

Signed-off-by: Zack Cerza <zack@redhat.com>

.github: Test with more Python versions

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Rebuild wrong-python venvs

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Use --use-pep517 when installing reqs

See https://github.com/pypa/pip/issues/8559

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Support alternate python versions

And check for their usability before proceeding

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1794 from ceph/githelper

config: Update githelper URL

config: Update githelper URL

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1790 from ceph/no-devtmpfs-scratch-devs

misc.get_scratch_devices(): Ignore devtmpfs

Devices shared with a container by its host will appear in /proc/mounts
this way.

Signed-off-by: Zack Cerza <zack@redhat.com>

misc.get_scratch_devices(): Split up checks

For better logging.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1786 from rishabh-d-dave/orch-mount-method

Merge pull request #1789 from ktdreyer/fog-error-suggestion

fog: suggest image names on error

Merge pull request #1788 from ktdreyer/lock-help

lock: remove --list --brief example

orchestra: add a method to check if given path is mounted

Add a method mount() that checks if the given path is mounted or
unmounted without using "mount" of "findmnt" commands.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

fog: suggest image names

Improve the error message when the user specifies a missing OS version
for a machine type.

Old error message:

  RuntimeError: Could not find an image for rhel 7.9

New error message:

  RuntimeError: Fog has no cali_rhel_7.9 image.
                Available cali images: ['cali_rhel_8.6', 'cali_rhel_9.0']

lock: remove --list --brief example

The --list and --brief arguments are exclusive.

  teuthology-lock: error: argument --brief: not allowed
  with argument --list

Merge pull request #1719 from kamoltat/wip-ksirivad-fix-53206

task/internal: open gzip file if core is compressed
Reviewed by: Ilya Dryomov <idryomov@redhat.com>
Reviewed by: Zack Cerza <zcerza@redhat.com>

Merge pull request #1787 from kamoltat/wip-ksirivad-boostrap-centosstream

boostrap: Add support for CentOSStream

boostrap: Add support for CentOSStream

currently we cannot do boostrap for CentOSStream
so we are adding support for that.

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

task/internal/__init__.py: improve core program extraction

Problem:
The previous technique for extracting the program name
of the coredump file doesn't deal with the case
where there is no white space after program name
e.g.,

``SVR4-style, from 'ceph_test_rados_api_io', real uid: 1000``

here we will get an error when we reach the part where we

``remote.sh(['which', dump_program]).rstrip()``.

Solution:

Use regex to extract first occurence of the word that
is in the format ``from '<word>'``.

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1785 from ceph/jammy

opsys.py: Make Ubuntu 22.04 default

test_default_distro_version: Update to 22.04

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1784 from ceph/wip-57092

misc._ssh_keyscan(): Drop '-t rsa'

opsys.py: Make Ubuntu 22.04 default

Signed-off-by: David Galloway <dgallowa@redhat.com>

misc._ssh_keyscan(): Drop '-t rsa'

It appears to be unnecessary, and causes Ubuntu 22.04 to respond with
zero keys.

Signed-off-by: Zack Cerza <zack@redhat.com>

task/tests: init pytest test_fetch_core_dumps

Create a unittest to test the function:
`fetch_binaries_for_coredumps()`.

Testing:

1. uncompressed coredump and correct format.
2. uncompressed coredump and incorrect format.
3. compressed coredump and correct format.
4. compressed coredump and incorrect format.

Fixes: https://tracker.ceph.com/issues/53206
Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1774 from ceph/nuke-typerror

nuke: Tolerate null node description
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

nuke: Tolerate null node description

Fixes a TypeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1783 from zdover23/wip-doc-2022-08-04-fragment-merging-typo

docs/frag...merging.rst - s/contructed/constructed
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

docs/frag...merging.rst - s/contructed/constructed

This PR corrects "contructed" to "constructed".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #1781 from Ngwind/patch-1

Update COMPONENTS.rst

Merge pull request #1782 from ceph/jammy

opsys.py: Add Ubuntu 22.04

opsys.py: Add Ubuntu 22.04

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1779 from ceph/dc-suite-fix

Fix scheduling regression

docs/docker-compose: Use Ubuntu Focal for testnode

I'm seeing ceph-cm-ansible failures w/ Jammy because of python-apt vs.
python3-apt. We'll revisit this soon.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1778 from ceph/no-mails-with-dryrun

suite: Do not send emails with --dry-run at schedule_fail
Reviewed by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Update COMPONENTS.rst

Fix `components.rst` file not being rendered

suite: add pytests for schedule_fail

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

choose_teuthology_branch: Choose main later on

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/docker-compose: Tolerate empty TEUTH_BRANCH

Signed-off-by: Zack Cerza <zack@redhat.com>

schedule_suite(): Make 'configs' a list

The var being left as a generator led to all jobs being filtered out of
some suites, rendering them broken.

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/docker-compose: Ensure we actually schedule

This should have failed: https://github.com/ceph/teuthology/pull/1776/checks

Signed-off-by: Zack Cerza <zack@redhat.com>

beanstalk: Print tube stats in JSON format

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1776 from batrick/yaml-preprocessor

teuthology: add pre/post fragment merge hooks

task/internal: open gzip file if core is compressed

Addded a case where if core is compressed, then
we will try to open it as gzip and read the content
inside.

We also add try and except when we are parsing
through the content to prevent the program from
crashing.

Fixes: https://tracker.ceph.com/issues/53206
Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

docs: document fragment merging

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite/test: add tests for fragment merging

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology: add lua based fragment merge scripting

As part of this change, there is a new generator design for producing
job configs. YAML fragments are memoized and merged manually to avoid
expensive and unnecessary parsing of the merged fragments. This provides
for a dramatic speedup in processing matrices with large numbers of
jobs. For rados suite with --subset 1/1000, this branch is 5x faster
(77s vs. 15s). (Note: the difference shrinks when there are fewer or
jobs or larger subsets are used due to cycling and the matrix generation
dominating runtime.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: support MappingProxyType in deep_merge

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: avoid copying references of b when list or dict

Otherwise, if b persists in memory (in a cache), it may wrongly
accumulate future merges.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1777 from VallariAg/log-total-jobs

suite/run: Log total number of jobs scheduled

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

suite: Do not send emails with --dry-run at schedule_fail

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #1769 from zmc/docs-missing-commands

docs/commands: Add missing commands
Reviewed by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1775 from ceph/update-reqs

Update pip-tools and sentry-sdk

Update sentry-sdk

We've started seeing what appear to be duplicate issues:
https://tracker.ceph.com/issues/56410
Hoping a simple update will resolve this...

Signed-off-by: Zack Cerza <zack@redhat.com>

Update pip-tools

Fixes https://github.com/jazzband/pip-tools/issues/1617

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1772 into master

* refs/pull/1772/head:
Revert "Merge pull request #1624 from batrick/get-valgrind-args"

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Revert "Merge pull request #1624 from batrick/get-valgrind-args"

This reverts commit 40fff4b2911949847a302adfe2f4714cfaccb58e, reversing
changes made to 93d2b653facc0e5a7a532b39d9ff91a044f2c4a9.

Merge pull request #1770 from ceph/plow

opsys.py: Add RHEL9

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1765 from ceph/wip-45512

docs/commands: Add missing commands

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1764 from kshtsk/wip-watch-fix-typeerror

Merge pull request #1766 from ceph/rhel86

Merge pull request #1767 from ceph/bs-missing-pip

bootstrap: If pip is missing, rebuild virtualenv

Signed-off-by: Zack Cerza <zack@redhat.com>

opsys.py: Default to RHEL8.6

Signed-off-by: David Galloway <dgallowa@redhat.com>

nuke.nuke: Rework lock-checking logic

Previously, we would call list_locks(), then iterate over the response,
each time iterating over the list of targets. If list_locks()
encountered an error and returned an empty response, we'd never actually
verify what we intended to. Instead, we should specifically query for
each target. This is far safer and faster.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: fix type error when description is none

teuthology-watch fails when run is complete and jobs' description
gets None value

Example:

2022-06-07 16:41:22,538.538 INFO:teuthology.suite:waiting for the run runner-2022-06-07_16:41:04-suse:tier0-ses7p-none-default-ecp to complete
2022-06-07 16:41:22,539.539 DEBUG:teuthology.suite:the list of unfinished jobs will be displayed every 5.0 minutes
2022-06-07 16:46:22,599.599 DEBUG:teuthology.suite:wait for jobs ['654']
2022-06-07 16:51:22,633.633 DEBUG:teuthology.suite:wait for jobs ['654']
2022-06-07 16:51:22,686.686 INFO:teuthology.suite:wait is done
Traceback (most recent call last):
  File "/home/runner/src/teuthology_master/virtualenv/bin/teuthology-wait", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-wait')())
  File "/home/runner/src/teuthology_master/scripts/wait.py", line 30, in main
    return teuthology.suite.wait(name, config.max_job_time, None)
  File "/home/runner/src/teuthology_master/teuthology/suite/__init__.py", line 234, in wait
    log.info(job['status'] + " " + url + " " + job['description'])
TypeError: must be str, not NoneType

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1624 from batrick/get-valgrind-args

Merge pull request #1762 from ceph/nested-subset-fix

qa: correct line processing logic

qa: correct line processing logic

Most runs don't use --no-nested-subset and for those that used --subset,
the if conditions would correctly pickup "seed" (when it mattered).
However, when --subset was not specified in the original run, the "seed"
was not correctly picked up. Therefore, the result of inserting the "if
no_nested_subset is None:" before the "elif seed is None:" caused it to
never read the seed for most folks teuthology runs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1761 from ceph/git-clone-errors

repo_utils: Always log git clone failure output

repo_utils: Always log git clone failure output

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1753 from ceph/dc-venv

Merge pull request #1760 from ceph/wip-private-branch

suite: Allow 'git@' repo URLs

suite: Allow 'git@' repo URLs

We were accidentally mangling them.

Signed-off-by: Zack Cerza <zack@redhat.com>