suite/run: Log total number of jobs scheduled

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #1775 from ceph/update-reqs

Update pip-tools and sentry-sdk

Update sentry-sdk

We've started seeing what appear to be duplicate issues:
https://tracker.ceph.com/issues/56410
Hoping a simple update will resolve this...

Signed-off-by: Zack Cerza <zack@redhat.com>

Update pip-tools

Fixes https://github.com/jazzband/pip-tools/issues/1617

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1772 into master

* refs/pull/1772/head:
Revert "Merge pull request #1624 from batrick/get-valgrind-args"

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Revert "Merge pull request #1624 from batrick/get-valgrind-args"

This reverts commit 40fff4b2911949847a302adfe2f4714cfaccb58e, reversing
changes made to 93d2b653facc0e5a7a532b39d9ff91a044f2c4a9.

Merge pull request #1770 from ceph/plow

opsys.py: Add RHEL9

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1765 from ceph/wip-45512

Merge pull request #1764 from kshtsk/wip-watch-fix-typeerror

Merge pull request #1766 from ceph/rhel86

Merge pull request #1767 from ceph/bs-missing-pip

bootstrap: If pip is missing, rebuild virtualenv

Signed-off-by: Zack Cerza <zack@redhat.com>

opsys.py: Default to RHEL8.6

Signed-off-by: David Galloway <dgallowa@redhat.com>

nuke.nuke: Rework lock-checking logic

Previously, we would call list_locks(), then iterate over the response,
each time iterating over the list of targets. If list_locks()
encountered an error and returned an empty response, we'd never actually
verify what we intended to. Instead, we should specifically query for
each target. This is far safer and faster.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: fix type error when description is none

teuthology-watch fails when run is complete and jobs' description
gets None value

Example:

2022-06-07 16:41:22,538.538 INFO:teuthology.suite:waiting for the run runner-2022-06-07_16:41:04-suse:tier0-ses7p-none-default-ecp to complete
2022-06-07 16:41:22,539.539 DEBUG:teuthology.suite:the list of unfinished jobs will be displayed every 5.0 minutes
2022-06-07 16:46:22,599.599 DEBUG:teuthology.suite:wait for jobs ['654']
2022-06-07 16:51:22,633.633 DEBUG:teuthology.suite:wait for jobs ['654']
2022-06-07 16:51:22,686.686 INFO:teuthology.suite:wait is done
Traceback (most recent call last):
  File "/home/runner/src/teuthology_master/virtualenv/bin/teuthology-wait", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-wait')())
  File "/home/runner/src/teuthology_master/scripts/wait.py", line 30, in main
    return teuthology.suite.wait(name, config.max_job_time, None)
  File "/home/runner/src/teuthology_master/teuthology/suite/__init__.py", line 234, in wait
    log.info(job['status'] + " " + url + " " + job['description'])
TypeError: must be str, not NoneType

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1624 from batrick/get-valgrind-args

Merge pull request #1762 from ceph/nested-subset-fix

qa: correct line processing logic

qa: correct line processing logic

Most runs don't use --no-nested-subset and for those that used --subset,
the if conditions would correctly pickup "seed" (when it mattered).
However, when --subset was not specified in the original run, the "seed"
was not correctly picked up. Therefore, the result of inserting the "if
no_nested_subset is None:" before the "elif seed is None:" caused it to
never read the seed for most folks teuthology runs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1761 from ceph/git-clone-errors

repo_utils: Always log git clone failure output

repo_utils: Always log git clone failure output

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1753 from ceph/dc-venv

Merge pull request #1760 from ceph/wip-private-branch

suite: Allow 'git@' repo URLs

suite: Allow 'git@' repo URLs

We were accidentally mangling them.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1757 from ceph/master-to-main

Inclusive Language Update

test_util.git_repository: Create main branch

(if necessary)

Signed-off-by: Zack Cerza <zack@redhat.com>

test_repo_utils: Don't break with main branch

More generally, if git was configured to use a default branch other
than 'master', this test would fail.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 9ab6d0a50e4821dccaeef024485986792e60a57b)

Replace non-branch occurrences of 'master'

Signed-off-by: Zack Cerza <zack@cerza.org>

Replace 'master' branch references with 'main'

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1704 into master

* refs/pull/1704/head:
teuthology/suite/test: test nested subsets
teuthology: add option to disable nested subsets
teuthology/suite: create nested matrix subsets
teuthology/suite: patch builtin open method
teuthology/test: use correct exception type
teuthology/suite/test: make sure patchers are cleaned up on exception
teuthology/suite/test: clarify variable name

Reviewed-by: Zack Cerza <zmc@redhat.com>

teuthology/suite/test: test nested subsets

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology: add option to disable nested subsets

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: create nested matrix subsets

The general idea is to allow the `%` convolution operator to also subset
the resulting matrix. This is done by specifying a number of divisions
for the subset in the `%` file. Such as:

dir/%:

    8

This commit maps a matrix index range of `[0, Subset.size())` to the
matrix it is taking a subset of, `[0, Matrix.size())`. To get full
coverage, a random number is used to specify "which" subset to use.
Contrast with the `--subset` argument to `teuthology-suite` which lets
you specify which subset.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: patch builtin open method

teuthology.suite.open does not patch properly.

Fixes: 0b822b8455a8b2903759776ff3c4ea05c416b44e
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/test: use correct exception type

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite/test: make sure patchers are cleaned up on exception

Otherwise, test infrastructure blows up on use of the fake file system
methods.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite/test: clarify variable name

"den" for denominator is not that intuitive.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1756 from ljflores/wip-teuthology-suite-priority-table

scripts: add priority table to teuthology-suite --help

scripts: add priority table to teuthology-suite --help

Table copied from https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro/.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #1754 from ceph/deps-upgrade-test

Add new GitHub Action to test deps on upgrade

bootstrap: Do ansible version check inside venv

We need access to the packaging module to do so.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add new GitHub Action to test deps on upgrade

We've seen a couple regressions that are only visible when upgrading
and existing virtualenv.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1752 from tchaikov/repo-utils-cleanup

teuthology/repo_util: pass universal_newlines=True

test_repo_utils: Don't break with main branch

More generally, if git was configured to use a default branch other
than 'master', this test would fail.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Look for teuthology branch mismatches

If config.teuthology_path is set, we don't fetch teuthology at runtime.
This can lead to confusing situations if --teuthology-branch is passed
to teuthology-suite, as the job config will claim a branch is used when
it is not. We can avoid this problem by refusing to schedule if there
is a mismatch.

Signed-off-by: Zack Cerza <zack@redhat.com>

exceptions: Add BranchMismatchError

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils: Add current_branch()

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Move TEUTH_BRANCH definition

So that we can access it from within the teuthology container.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop pyc removal

We stopped writing these files years ago.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Optimize pip installs

With this change, we can avoid rerunning bootstrap on build if
requirements.txt hasn't changed. Build times after non-requirements
changes go from 3-5min to ~2s.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Optionally skip teuthology install

The docker-compose dev setup was the inspiration for this feature; it
allows us to split the installation of dependencies and the project
itself into two phases. This in turn allows us to avoid rerunning
bootstrap if requirements.txt didn't change.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/repo_util: pass universal_newlines=True

for better readability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1751 from ceph/wip-bs-err

bootstrap_teuthology: Call decode() on stderr

bootstrap_teuthology: Call decode() on stderr

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1748 from ceph/ansible-2.10

Update ansible to 2.10

bootstrap: Drop virtualenv in favor of venv

We don't need it, and if it is present, setuptools will use
easy_install instead of building wheels.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update ansible to 2.10

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: If ansible<2.10 is present, remove it

... before proceeding. This is because in 2.10, ansible split into two
packages, a scenario that pip apparently isn't able to handle properly.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1750 from tchaikov/pass-extra

update-requirements.sh: pass --extra=test to pip-compile

Merge pull request #1698 from kshtsk/wip-raise-if-not-reconnect

orchestra/remote: throw exception on reconnect failed

update-requirements.sh: pass --extra=test to pip-compile

otherwise the requirements used by test would be missed

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1749 from tchaikov/setup-add-nose

setup.cfg: add nose to "options.extras_require/test"

setup.py: dedup the requires

* move the dependencies from options.install_requires to
  options.extras_require if those dependencies are only used by tests.
  as per
  https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
  > install_requires is a setuptools setup.py keyword that should be
  > used to specify what a project minimally needs to run correctly.
* remove the duplicated dependencies from options.extras_require
  if they are also listed in options.install_requires.
* update requirements.txt using
  "pip-compile --extra=test pyproject.toml", so requirements.txt
  can include the dependencies required by tests.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

scrape: drop dependency of "six"

since we don't need to be compatible with python2, let's stop using
the six package.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

setup.cfg: add nose to "options.extras_require/test"

some tests are using nose python package. since teuthology is
responsible for preparing the deps to run the tests. let's add
nose back to setup.cfg.

the "nose" package was included in requirements.txt before
21372fd9b2b87bb6417929f10097fba7db9cdca9. in this change, it,
it is brought back.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1743 from ceph/packaging

Revised packaging updates

Merge pull request #1747 from ceph/ca-test-fix

task/ceph_ansible: Fix attribute handling

Merge pull request #1746 from rishabh-d-dave/py3104-MutableMapping

config: import MuableMapping based on python version

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

task/ceph_ansible: Fix attribute handling

... in get_host_vars(). This had broken unit tests when a top-level
"ceph_ansible" key is present in the teuthology config.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Drop basepython lines

They're redundant.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Install 'test' extras for py.test envs

Signed-off-by: Zack Cerza <zack@redhat.com>

Modernized versioning

Using setuptools_scm; we can automatically generate version numbers
instead of using this odd combination of hardcoding and running git
commands. This method is faster as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Install 'test' extras

So that we can use the 'tests' task.

Signed-off-by: Zack Cerza <zack@redhat.com>

run_tasks: Do not mask missing task dependencies

While doing packaging work, I noticed that teuthology began claiming it
couldn't find the 'tests' task. After some slightly painful debugging I
realized the issue was that the task was trying to import pytest, which
wasn't installed. The ModuleNotFoundError that was being raised was
being confused with the exception that would be raised if the task
couldn't be found at all. With this change, we see the root cause.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Update pytest_runtest_makereport

And fix a typo

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop python2 support

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Set isolated_build

This also requires us to re-add mock to the py3 env.

Signed-off-by: Zack Cerza <zack@redhat.com>

TestOpenStackBase.setup: Remove arguments

In newer py.test versions, the test method itself was being passed as
the conf arg, causing errors. We don't even use it, so remove it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Tolerate a missing manhole package

The vast majority of users won't ever need this.

Signed-off-by: Zack Cerza <zack@redhat.com>

MANIFEST.in: Add tox.ini and pytest.ini

Signed-off-by: Zack Cerza <zack@redhat.com>

Replace setup.py with setup.cfg and pyproject.toml

Signed-off-by: Zack Cerza <zack@redhat.com>

config: import MuableMapping based on python version

MutableMapping has been moved from collections to collections.abc.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1739 from ceph/docker-compose

docker-compose: "wait mode"

docker-compose.yml: Start beanstalkd sooner

There was a bit of a race condition happening in wait mode, where
teuthology would start up before beanstalkd and fail immediately.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Default to RSA keys

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add wait mode

Instead of scheduling the teuthology:no-ceph suite and then exiting
when it finishes, schedule nothing and don't exit automatically. This
allows running whatever - and however many - tests are desired.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Document containerized nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

.dockerignore: Simplify paths

The we we'd used wildcards, we were only ignoring directory contents -
not the directories themselves. Also add the tox temporary directory.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Use repo root as build context

Rather than add more logic to support things like cloning different
forks of the repo, let's just not clone at all.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Make sure to run docker-compose down

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1741 from ceph/dc-fixes

teuthology.sh: Run tests using current branch

Merge pull request #1742 from ceph/repo_logging

repo_utils: Fix logging in bootstrap_teuthology

repo_utils: Fix logging in bootstrap_teuthology

Signed-off-by: Zack Cerza <zack@redhat.com>

worker: Don't log incorrect teuthology sha1

If teuthology_path is set, we don't fetch a separate copy of the repo.
In that case, the sha1 logged here may be incorrect.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Run tests using current branch

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1740 from ceph/wip-revert4333

Revert "Merge pull request #1737 from ceph/build"

Revert "Merge pull request #1737 from ceph/build"

This reverts commit 43327234bc2c8066c173e4c6d602c8a0a05db642, reversing
changes made to a347b2ac5400a43ba76a678399b1deda7dae1a6c.

Merge pull request #1737 from ceph/build

Packaging overhaul

bootstrap: Drop pyc removal

We stopped writing these files years ago.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Optionally skip teuthology install

The docker-compose dev setup was the inspiration for this feature; it
allows us to split the installation of dependencies and the project
itself into two phases. This in turn allows us to avoid rerunning
bootstrap if requirements.txt didn't change.

Signed-off-by: Zack Cerza <zack@redhat.com>