Modernized versioning

Using setuptools_scm; we can automatically generate version numbers
instead of using this odd combination of hardcoding and running git
commands. This method is faster as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove references to teuth_bin_path

This effectively hardcoded the path to the virtualenv we'll use, which
aside from being bad generally, also prevented us from supporting
absolute-pathed virtualenvs.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Allow absolute-path $VENV

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop virtualenv in favor of venv

We don't need it, and if it is present, setuptools will use
easy_install instead of building wheels.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Install 'test' extras

So that we can use the 'tests' task.

Signed-off-by: Zack Cerza <zack@redhat.com>

run_tasks: Do not mask missing task dependencies

While doing packaging work, I noticed that teuthology began claiming it
couldn't find the 'tests' task. After some slightly painful debugging I
realized the issue was that the task was trying to import pytest, which
wasn't installed. The ModuleNotFoundError that was being raised was
being confused with the exception that would be raised if the task
couldn't be found at all. With this change, we see the root cause.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Update pytest_runtest_makereport

And fix a typo

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop python2 support

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Set isolated_build

This also requires us to re-add mock to the py3 env.

Signed-off-by: Zack Cerza <zack@redhat.com>

TestOpenStackBase.setup: Remove arguments

In newer py.test versions, the test method itself was being passed as
the conf arg, causing errors. We don't even use it, so remove it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Tolerate a missing manhole package

The vast majority of users won't ever need this.

Signed-off-by: Zack Cerza <zack@redhat.com>

MANIFEST.in: Add tox.ini and pytest.ini

Signed-off-by: Zack Cerza <zack@redhat.com>

Replace setup.py with setup.cfg and pyproject.toml

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1734 from ceph/dependabot/pip/paramiko-2.10.1

build(deps): bump paramiko from 2.7.2 to 2.10.1

build(deps): bump paramiko from 2.7.2 to 2.10.1

Bumps [paramiko](https://github.com/paramiko/paramiko) from 2.7.2 to 2.10.1.
- [Release notes](https://github.com/paramiko/paramiko/releases)
- [Changelog](https://github.com/paramiko/paramiko/blob/main/NEWS)
- [Commits](https://github.com/paramiko/paramiko/compare/2.7.2...2.10.1)

---
updated-dependencies:
- dependency-name: paramiko
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1726 from ceph/bs-mac-noauto

bootstrap: Don't auto-install on Mac

Merge pull request #1727 from ceph/docker-compose

docker-compose: Use containerized testnodes to run the teuthology suite

docker-compose: Update readme

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Increase paddles' check time

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Separate build contexts

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Allow different custom config

When using "real" testnodes, we should not use the custom config we've
been using for containerized nodes. When using "real" testnodes, default
 to not using one, but allow specifying a path in $CUSTOM_CONF.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-update-inventory: Add machine type flag

With this change, the tool can be trivially used to add new nodes to
paddles.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Write SSH key stubs later in build

We'd rather duplicate this layer than the package-install layer if a
user is going to use both containerized testnodes and "real" ones.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Optionally use an inventory repo

This is being added to support "real" testnodes without having to
manually copy inventory fragments over.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Don't double-clone teuthology

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Optionally use an existing SSH key

This will only work with "real" testnodes; with containerized testnodes
there's no reason not to use a generated key.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Populate "real" testnodes

... if we're using them, that is.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Fix old typo

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Copy config files more consistently

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Exit after jobs are done

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Optionally exit when queue is empty

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Return job's return code

Signed-off-by: Zack Cerza <zack@redhat.com>

Add 'integration' GitHub Action

This will use our recent docker-compose work.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Use current branch by default

Since the main use-case for this work at the moment is self-testing
teuthology, let's make it easy to test the same branch that's being
worked on.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Use the teuthology:no-ceph suite

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode: Install apache2 and nfs

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose.yml: Force testnode arch

I initially worked on this on an M1 Mac, and that caused problems with
scheduling due to a lack of aarch64 packages. The M1 can run amd64
images, though.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode: Add stop script

So that each container marks itself down before stopping.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode.sh: Retry submitting to paddles

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile-testnode: Switch to Ubuntu

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: In containers, simply kill sshd

At least with the testnode containers I'm working on at the moment,
this appears to be the best way to shut them down from inside.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Copy conf files into container

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose.yml: Use three testnodes

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile-testnode: Install Python packages

These are needed by ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Use absolute paths

Signed-off-by: Zack Cerza <zack@redhat.com>

.gitignore: Ignore more built docs

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add .dockerignore

Signed-off-by: Zack Cerza <zack@redhat.com>

.teuthology.yaml: Use absolute paths

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add custom_conf.yaml

This will be merged into generated job configs.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Split out entrypoint to teuthology.sh

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add testnode_sudoers

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Perform teuth setup later in process

Should greatly reduce the frequency of package installs during image
rebuilds.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Set up SSH client incl. generated key

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add .teuthology.yaml

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Generate an SSH keypair

We'll use a fresh keypair for each invocation.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add containerized testnode

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Run bootstrap during build

This saves quite a bit of time

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Reformat for better readability

Signed-off-by: Zack Cerza <zack@redhat.com>

lock_many: Include username in target keys

So that we can actually support different usernames per node.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1731 from kamoltat/wip-sphinx

docs/requirements.txt: bump sphinx to 4.4.0

docs/requirements.txt: bump sphinx to 4.4.0

bump sphinx to latest stable

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1728 from kamoltat/wip-ksirivad-fix-docker-compose

docs/docker-compose: paddles use quay.io image
Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

docs/docker-compose: paddles use quay.io image

paddles use quay.io

docker-compose.yaml: teuthology depends on healthy
paddles rather pulpito

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

bootstrap: Don't auto-install on Mac

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1725 from ceph/fix-clock

task.clock: Fix call to filter_out_containers()

task.clock: Fix call to filter_out_containers()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1722 from ceph/task-skips-container

Skip various operations on containerized testnodes

nuke: Don't attempt to nuke downed nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal.syslog: Skip container nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

task.clock: Skip containerized testnodes

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1724 from ceph/update-reqs

Update some requirements

Merge pull request #1720 from ceph/teuth-suite

Revive the 'test' task, used by the 'teuthology' suite

task.tests Run unit tests

In addition to the six tests we were running, let's add the ~six
hundred unit tests we've added over the years.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_run: Skip a test if running in teuth process

Signed-off-by: Zack Cerza <zack@redhat.com>

Add skipif_teuthology_process

There are a couple unit tests that seem to be fundamentally unable to
be used properly if they are run as part of a teuthology process, as
opposed to directly inside py.test. This fixture makes it easy to skip
those tests in such an environment without skipping them elsewhere.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Use job_status module

Previously this task had an embarrassing "status=fail, success=true"
bug.

Signed-off-by: Zack Cerza <zack@redhat.com>

TeuthologyContextPlugin: Parametrize conditionally

So that this plugin can be compatible with our unit tests.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Add mock; update py.test

While it'd be technically more correct to split these out into dev
requirements, I think a little bit of an overhaul of our bootstrap
process overdue. We can save that work for then. For now, this smooths
a kink in the docker-compose work we're doing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Update ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

Update requests

Signed-off-by: Zack Cerza <zack@redhat.com>

Update s3transfer

Signed-off-by: Zack Cerza <zack@redhat.com>

Update boto3

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1703 from ceph/wip-53853

kill: Check if sudo works before using it

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

kill: Check if sudo works before using it

Also, use sudo's -n flag to avoid the password prompt and resulting
delay.

Fixes: https://tracker.ceph.com/issues/53853
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1723 from ceph/wip-standalone

fixes for running non-scheduled jobs

task/ansible: check for archive before using it

Running a non-scheduled job you may not have archiving configured.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

run: make fetching tasks work for non-scheduled jobs

The qa path should be the default (just like it's set by
teuthology-suite), and we actually need to do the import to access the
tasks.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1721 from ceph/more-tox-gha

github actions: Run docs build during CI
Reviewed-by: Zack Cerza zack@redhat.com

task.selinux: Exclude container nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal: Skip coredump on containers

Signed-off-by: Zack Cerza <zack@redhat.com>

task/kernel: Skip if running in a container

Signed-off-by: Zack Cerza <zack@redhat.com>

Skip reimaging on unsupported machine types

Signed-off-by: Zack Cerza <zack@redhat.com>

github actions: Run docs build during CI

If we do this, we can drop the Jenkins job. The openstack tests are
currently all skipped, so we can just leave them out for now.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Fix reference to python binary

It was trying to use `python` as opposed to `python3`.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Fix StringIO import

It's in the io module now.

Signed-off-by: Zack Cerza <zack@redhat.com>

TeuthologyContextPlugin: Remove pytest_configure()

The reason for this method's behavior here is no longer valid. Also, it
prevents any output from being logged.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1715 from ceph/pynacl

Update pynacl

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1714 from ceph/ui-connect

teuthology-update-inventory: Call Remote.connect()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1712 from ceph/remote-is-container

Remote: Add is_container property

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1718 from kamoltat/wip-ksirivad-dispatcher-notes-doc

docs: add dispatcher trouble-shooting notes
Reviewed by: Aishwarya Mathuria <amathuri@redhat.com>