Merge pull request #1774 from ceph/nuke-typerror

nuke: Tolerate null node description
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

nuke: Tolerate null node description

Fixes a TypeError

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1783 from zdover23/wip-doc-2022-08-04-fragment-merging-typo

docs/frag...merging.rst - s/contructed/constructed
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

docs/frag...merging.rst - s/contructed/constructed

This PR corrects "contructed" to "constructed".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #1781 from Ngwind/patch-1

Update COMPONENTS.rst

Merge pull request #1782 from ceph/jammy

opsys.py: Add Ubuntu 22.04

opsys.py: Add Ubuntu 22.04

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1779 from ceph/dc-suite-fix

Fix scheduling regression

docs/docker-compose: Use Ubuntu Focal for testnode

I'm seeing ceph-cm-ansible failures w/ Jammy because of python-apt vs.
python3-apt. We'll revisit this soon.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1778 from ceph/no-mails-with-dryrun

suite: Do not send emails with --dry-run at schedule_fail
Reviewed by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Update COMPONENTS.rst

Fix `components.rst` file not being rendered

suite: add pytests for schedule_fail

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

choose_teuthology_branch: Choose main later on

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/docker-compose: Tolerate empty TEUTH_BRANCH

Signed-off-by: Zack Cerza <zack@redhat.com>

schedule_suite(): Make 'configs' a list

The var being left as a generator led to all jobs being filtered out of
some suites, rendering them broken.

Signed-off-by: Zack Cerza <zack@redhat.com>

docs/docker-compose: Ensure we actually schedule

This should have failed: https://github.com/ceph/teuthology/pull/1776/checks

Signed-off-by: Zack Cerza <zack@redhat.com>

beanstalk: Print tube stats in JSON format

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1776 from batrick/yaml-preprocessor

teuthology: add pre/post fragment merge hooks

docs: document fragment merging

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite/test: add tests for fragment merging

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology: add lua based fragment merge scripting

As part of this change, there is a new generator design for producing
job configs. YAML fragments are memoized and merged manually to avoid
expensive and unnecessary parsing of the merged fragments. This provides
for a dramatic speedup in processing matrices with large numbers of
jobs. For rados suite with --subset 1/1000, this branch is 5x faster
(77s vs. 15s). (Note: the difference shrinks when there are fewer or
jobs or larger subsets are used due to cycling and the matrix generation
dominating runtime.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: support MappingProxyType in deep_merge

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/misc: avoid copying references of b when list or dict

Otherwise, if b persists in memory (in a cache), it may wrongly
accumulate future merges.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1777 from VallariAg/log-total-jobs

suite/run: Log total number of jobs scheduled

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

suite: Do not send emails with --dry-run at schedule_fail

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>

Merge pull request #1769 from zmc/docs-missing-commands

docs/commands: Add missing commands
Reviewed by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1775 from ceph/update-reqs

Update pip-tools and sentry-sdk

Update sentry-sdk

We've started seeing what appear to be duplicate issues:
https://tracker.ceph.com/issues/56410
Hoping a simple update will resolve this...

Signed-off-by: Zack Cerza <zack@redhat.com>

Update pip-tools

Fixes https://github.com/jazzband/pip-tools/issues/1617

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1772 into master

* refs/pull/1772/head:
Revert "Merge pull request #1624 from batrick/get-valgrind-args"

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Revert "Merge pull request #1624 from batrick/get-valgrind-args"

This reverts commit 40fff4b2911949847a302adfe2f4714cfaccb58e, reversing
changes made to 93d2b653facc0e5a7a532b39d9ff91a044f2c4a9.

Merge pull request #1770 from ceph/plow

opsys.py: Add RHEL9

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1765 from ceph/wip-45512

docs/commands: Add missing commands

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1764 from kshtsk/wip-watch-fix-typeerror

Merge pull request #1766 from ceph/rhel86

Merge pull request #1767 from ceph/bs-missing-pip

bootstrap: If pip is missing, rebuild virtualenv

Signed-off-by: Zack Cerza <zack@redhat.com>

opsys.py: Default to RHEL8.6

Signed-off-by: David Galloway <dgallowa@redhat.com>

nuke.nuke: Rework lock-checking logic

Previously, we would call list_locks(), then iterate over the response,
each time iterating over the list of targets. If list_locks()
encountered an error and returned an empty response, we'd never actually
verify what we intended to. Instead, we should specifically query for
each target. This is far safer and faster.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: fix type error when description is none

teuthology-watch fails when run is complete and jobs' description
gets None value

Example:

2022-06-07 16:41:22,538.538 INFO:teuthology.suite:waiting for the run runner-2022-06-07_16:41:04-suse:tier0-ses7p-none-default-ecp to complete
2022-06-07 16:41:22,539.539 DEBUG:teuthology.suite:the list of unfinished jobs will be displayed every 5.0 minutes
2022-06-07 16:46:22,599.599 DEBUG:teuthology.suite:wait for jobs ['654']
2022-06-07 16:51:22,633.633 DEBUG:teuthology.suite:wait for jobs ['654']
2022-06-07 16:51:22,686.686 INFO:teuthology.suite:wait is done
Traceback (most recent call last):
  File "/home/runner/src/teuthology_master/virtualenv/bin/teuthology-wait", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-wait')())
  File "/home/runner/src/teuthology_master/scripts/wait.py", line 30, in main
    return teuthology.suite.wait(name, config.max_job_time, None)
  File "/home/runner/src/teuthology_master/teuthology/suite/__init__.py", line 234, in wait
    log.info(job['status'] + " " + url + " " + job['description'])
TypeError: must be str, not NoneType

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1624 from batrick/get-valgrind-args

Merge pull request #1762 from ceph/nested-subset-fix

qa: correct line processing logic

qa: correct line processing logic

Most runs don't use --no-nested-subset and for those that used --subset,
the if conditions would correctly pickup "seed" (when it mattered).
However, when --subset was not specified in the original run, the "seed"
was not correctly picked up. Therefore, the result of inserting the "if
no_nested_subset is None:" before the "elif seed is None:" caused it to
never read the seed for most folks teuthology runs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1761 from ceph/git-clone-errors

repo_utils: Always log git clone failure output

repo_utils: Always log git clone failure output

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1753 from ceph/dc-venv

Merge pull request #1760 from ceph/wip-private-branch

suite: Allow 'git@' repo URLs

suite: Allow 'git@' repo URLs

We were accidentally mangling them.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1757 from ceph/master-to-main

Inclusive Language Update

test_util.git_repository: Create main branch

(if necessary)

Signed-off-by: Zack Cerza <zack@redhat.com>

test_repo_utils: Don't break with main branch

More generally, if git was configured to use a default branch other
than 'master', this test would fail.

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit 9ab6d0a50e4821dccaeef024485986792e60a57b)

Replace non-branch occurrences of 'master'

Signed-off-by: Zack Cerza <zack@cerza.org>

Replace 'master' branch references with 'main'

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge PR #1704 into master

* refs/pull/1704/head:
teuthology/suite/test: test nested subsets
teuthology: add option to disable nested subsets
teuthology/suite: create nested matrix subsets
teuthology/suite: patch builtin open method
teuthology/test: use correct exception type
teuthology/suite/test: make sure patchers are cleaned up on exception
teuthology/suite/test: clarify variable name

Reviewed-by: Zack Cerza <zmc@redhat.com>

teuthology/suite/test: test nested subsets

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology: add option to disable nested subsets

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: create nested matrix subsets

The general idea is to allow the `%` convolution operator to also subset
the resulting matrix. This is done by specifying a number of divisions
for the subset in the `%` file. Such as:

dir/%:

    8

This commit maps a matrix index range of `[0, Subset.size())` to the
matrix it is taking a subset of, `[0, Matrix.size())`. To get full
coverage, a random number is used to specify "which" subset to use.
Contrast with the `--subset` argument to `teuthology-suite` which lets
you specify which subset.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite: patch builtin open method

teuthology.suite.open does not patch properly.

Fixes: 0b822b8455a8b2903759776ff3c4ea05c416b44e
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/test: use correct exception type

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite/test: make sure patchers are cleaned up on exception

Otherwise, test infrastructure blows up on use of the fake file system
methods.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

teuthology/suite/test: clarify variable name

"den" for denominator is not that intuitive.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1756 from ljflores/wip-teuthology-suite-priority-table

scripts: add priority table to teuthology-suite --help

scripts: add priority table to teuthology-suite --help

Table copied from https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro/.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #1754 from ceph/deps-upgrade-test

Add new GitHub Action to test deps on upgrade

bootstrap: Do ansible version check inside venv

We need access to the packaging module to do so.

Signed-off-by: Zack Cerza <zack@redhat.com>

Add new GitHub Action to test deps on upgrade

We've seen a couple regressions that are only visible when upgrading
and existing virtualenv.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1752 from tchaikov/repo-utils-cleanup

teuthology/repo_util: pass universal_newlines=True

test_repo_utils: Don't break with main branch

More generally, if git was configured to use a default branch other
than 'master', this test would fail.

Signed-off-by: Zack Cerza <zack@redhat.com>

suite: Look for teuthology branch mismatches

If config.teuthology_path is set, we don't fetch teuthology at runtime.
This can lead to confusing situations if --teuthology-branch is passed
to teuthology-suite, as the job config will claim a branch is used when
it is not. We can avoid this problem by refusing to schedule if there
is a mismatch.

Signed-off-by: Zack Cerza <zack@redhat.com>

exceptions: Add BranchMismatchError

Signed-off-by: Zack Cerza <zack@redhat.com>

repo_utils: Add current_branch()

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Move TEUTH_BRANCH definition

So that we can access it from within the teuthology container.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop pyc removal

We stopped writing these files years ago.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Optimize pip installs

With this change, we can avoid rerunning bootstrap on build if
requirements.txt hasn't changed. Build times after non-requirements
changes go from 3-5min to ~2s.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Optionally skip teuthology install

The docker-compose dev setup was the inspiration for this feature; it
allows us to split the installation of dependencies and the project
itself into two phases. This in turn allows us to avoid rerunning
bootstrap if requirements.txt didn't change.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology/repo_util: pass universal_newlines=True

for better readability

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1751 from ceph/wip-bs-err

bootstrap_teuthology: Call decode() on stderr

bootstrap_teuthology: Call decode() on stderr

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1748 from ceph/ansible-2.10

Update ansible to 2.10

bootstrap: Drop virtualenv in favor of venv

We don't need it, and if it is present, setuptools will use
easy_install instead of building wheels.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Update ansible to 2.10

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: If ansible<2.10 is present, remove it

... before proceeding. This is because in 2.10, ansible split into two
packages, a scenario that pip apparently isn't able to handle properly.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1750 from tchaikov/pass-extra

update-requirements.sh: pass --extra=test to pip-compile

Merge pull request #1698 from kshtsk/wip-raise-if-not-reconnect

orchestra/remote: throw exception on reconnect failed

update-requirements.sh: pass --extra=test to pip-compile

otherwise the requirements used by test would be missed

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1749 from tchaikov/setup-add-nose

setup.cfg: add nose to "options.extras_require/test"

setup.py: dedup the requires

* move the dependencies from options.install_requires to
  options.extras_require if those dependencies are only used by tests.
  as per
  https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
  > install_requires is a setuptools setup.py keyword that should be
  > used to specify what a project minimally needs to run correctly.
* remove the duplicated dependencies from options.extras_require
  if they are also listed in options.install_requires.
* update requirements.txt using
  "pip-compile --extra=test pyproject.toml", so requirements.txt
  can include the dependencies required by tests.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

scrape: drop dependency of "six"

since we don't need to be compatible with python2, let's stop using
the six package.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

setup.cfg: add nose to "options.extras_require/test"

some tests are using nose python package. since teuthology is
responsible for preparing the deps to run the tests. let's add
nose back to setup.cfg.

the "nose" package was included in requirements.txt before
21372fd9b2b87bb6417929f10097fba7db9cdca9. in this change, it,
it is brought back.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1743 from ceph/packaging

Revised packaging updates

Merge pull request #1747 from ceph/ca-test-fix

task/ceph_ansible: Fix attribute handling

Merge pull request #1746 from rishabh-d-dave/py3104-MutableMapping

config: import MuableMapping based on python version

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

task/ceph_ansible: Fix attribute handling

... in get_host_vars(). This had broken unit tests when a top-level
"ceph_ansible" key is present in the teuthology config.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Drop basepython lines

They're redundant.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Install 'test' extras for py.test envs

Signed-off-by: Zack Cerza <zack@redhat.com>

Modernized versioning

Using setuptools_scm; we can automatically generate version numbers
instead of using this odd combination of hardcoding and running git
commands. This method is faster as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Install 'test' extras

So that we can use the 'tests' task.

Signed-off-by: Zack Cerza <zack@redhat.com>