start.sh: Copy config files more consistently

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Exit after jobs are done

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Optionally exit when queue is empty

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Return job's return code

Signed-off-by: Zack Cerza <zack@redhat.com>

Add 'integration' GitHub Action

This will use our recent docker-compose work.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Use current branch by default

Since the main use-case for this work at the moment is self-testing
teuthology, let's make it easy to test the same branch that's being
worked on.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Use the teuthology:no-ceph suite

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode: Install apache2 and nfs

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose.yml: Force testnode arch

I initially worked on this on an M1 Mac, and that caused problems with
scheduling due to a lack of aarch64 packages. The M1 can run amd64
images, though.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode: Add stop script

So that each container marks itself down before stopping.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode.sh: Retry submitting to paddles

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile-testnode: Switch to Ubuntu

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: In containers, simply kill sshd

At least with the testnode containers I'm working on at the moment,
this appears to be the best way to shut them down from inside.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Copy conf files into container

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose.yml: Use three testnodes

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile-testnode: Install Python packages

These are needed by ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Use absolute paths

Signed-off-by: Zack Cerza <zack@redhat.com>

.gitignore: Ignore more built docs

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add .dockerignore

Signed-off-by: Zack Cerza <zack@redhat.com>

.teuthology.yaml: Use absolute paths

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add custom_conf.yaml

This will be merged into generated job configs.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Split out entrypoint to teuthology.sh

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add testnode_sudoers

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Perform teuth setup later in process

Should greatly reduce the frequency of package installs during image
rebuilds.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Set up SSH client incl. generated key

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add .teuthology.yaml

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Generate an SSH keypair

We'll use a fresh keypair for each invocation.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add containerized testnode

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Run bootstrap during build

This saves quite a bit of time

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Reformat for better readability

Signed-off-by: Zack Cerza <zack@redhat.com>

lock_many: Include username in target keys

So that we can actually support different usernames per node.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1731 from kamoltat/wip-sphinx

docs/requirements.txt: bump sphinx to 4.4.0

docs/requirements.txt: bump sphinx to 4.4.0

bump sphinx to latest stable

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1728 from kamoltat/wip-ksirivad-fix-docker-compose

docs/docker-compose: paddles use quay.io image
Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

docs/docker-compose: paddles use quay.io image

paddles use quay.io

docker-compose.yaml: teuthology depends on healthy
paddles rather pulpito

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1725 from ceph/fix-clock

task.clock: Fix call to filter_out_containers()

task.clock: Fix call to filter_out_containers()

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1722 from ceph/task-skips-container

Skip various operations on containerized testnodes

nuke: Don't attempt to nuke downed nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal.syslog: Skip container nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

task.clock: Skip containerized testnodes

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1724 from ceph/update-reqs

Update some requirements

Merge pull request #1720 from ceph/teuth-suite

Revive the 'test' task, used by the 'teuthology' suite

task.tests Run unit tests

In addition to the six tests we were running, let's add the ~six
hundred unit tests we've added over the years.

Signed-off-by: Zack Cerza <zack@redhat.com>

test_run: Skip a test if running in teuth process

Signed-off-by: Zack Cerza <zack@redhat.com>

Add skipif_teuthology_process

There are a couple unit tests that seem to be fundamentally unable to
be used properly if they are run as part of a teuthology process, as
opposed to directly inside py.test. This fixture makes it easy to skip
those tests in such an environment without skipping them elsewhere.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Use job_status module

Previously this task had an embarrassing "status=fail, success=true"
bug.

Signed-off-by: Zack Cerza <zack@redhat.com>

TeuthologyContextPlugin: Parametrize conditionally

So that this plugin can be compatible with our unit tests.

Signed-off-by: Zack Cerza <zack@redhat.com>

requirements: Add mock; update py.test

While it'd be technically more correct to split these out into dev
requirements, I think a little bit of an overhaul of our bootstrap
process overdue. We can save that work for then. For now, this smooths
a kink in the docker-compose work we're doing.

Signed-off-by: Zack Cerza <zack@redhat.com>

Update ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

Update requests

Signed-off-by: Zack Cerza <zack@redhat.com>

Update s3transfer

Signed-off-by: Zack Cerza <zack@redhat.com>

Update boto3

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1703 from ceph/wip-53853

kill: Check if sudo works before using it

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

kill: Check if sudo works before using it

Also, use sudo's -n flag to avoid the password prompt and resulting
delay.

Fixes: https://tracker.ceph.com/issues/53853
Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1723 from ceph/wip-standalone

fixes for running non-scheduled jobs

task/ansible: check for archive before using it

Running a non-scheduled job you may not have archiving configured.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

run: make fetching tasks work for non-scheduled jobs

The qa path should be the default (just like it's set by
teuthology-suite), and we actually need to do the import to access the
tasks.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1721 from ceph/more-tox-gha

github actions: Run docs build during CI
Reviewed-by: Zack Cerza zack@redhat.com

task.selinux: Exclude container nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

task.internal: Skip coredump on containers

Signed-off-by: Zack Cerza <zack@redhat.com>

task/kernel: Skip if running in a container

Signed-off-by: Zack Cerza <zack@redhat.com>

Skip reimaging on unsupported machine types

Signed-off-by: Zack Cerza <zack@redhat.com>

github actions: Run docs build during CI

If we do this, we can drop the Jenkins job. The openstack tests are
currently all skipped, so we can just leave them out for now.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Fix reference to python binary

It was trying to use `python` as opposed to `python3`.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Fix StringIO import

It's in the io module now.

Signed-off-by: Zack Cerza <zack@redhat.com>

TeuthologyContextPlugin: Remove pytest_configure()

The reason for this method's behavior here is no longer valid. Also, it
prevents any output from being logged.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1715 from ceph/pynacl

Update pynacl

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1714 from ceph/ui-connect

teuthology-update-inventory: Call Remote.connect()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1712 from ceph/remote-is-container

Remote: Add is_container property

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #1718 from kamoltat/wip-ksirivad-dispatcher-notes-doc

docs: add dispatcher trouble-shooting notes
Reviewed by: Aishwarya Mathuria <amathuri@redhat.com>

docs: add dispatcher trouble-shooting notes

documenting https://tracker.ceph.com/issues/54366

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1717 from ceph/1716b

selinux.py: Ignore both contexts for NetworkManager_dispatcher_t

selinux.py: Ignore both contexts for NetworkManager_dispatcher_t

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1716 from ceph/54337

selinux.py: Ignore new NetworkManager denials

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Venky Shankar vshankar@redhat.com

selinux.py: Ignore new NetworkManager denials

Fixes: https://tracker.ceph.com/issues/54337
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1713 from ceph/ansible-log

task.ansible: Don't direct task's logging to file

Merge pull request #1711 from ceph/cluster-filter

Cluster: Add filter() function

requirements: Update pynacl

This avoids having to compile it from source on
every image build.

Signed-off-by: Zack Cerza <zack@redhat.com>

update-requirements.sh: Write intermediate file

The previous method was only writing the .txt file; when debugging
dependency issues you'll often want to see the intermediate. I don't
see a reason to not always write it.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-update-inventory: Call Remote.connect()

Signed-off-by: Zack Cerza <zack@redhat.com>

Remote: Add is_container property

Signed-off-by: Zack Cerza <zack@redhat.com>

task.ansible: Don't direct task's logging to file

We only want output generated by ansible itself in ansible.log

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1695 from kshtsk/wip-cluster-run-sync

orchestra/cluster: make run parallel and wait

Cluster: Add filter() function

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1707 from kamoltat/wip-ksirivad-add-ansible-inventory-docker-compose

docs/docker-compose: add ansible inventory to README
Reviewed by: Josh Durgin <jdurgin@redhat.com>

Merge PR #1710 into master

* refs/pull/1710/head:
teuthology/task/selinux: ignore sss_cache

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

teuthology/task/selinux: ignore sss_cache

Fixes: https://tracker.ceph.com/issues/54125
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #1708 into master

* refs/pull/1708/head:
opsys.py: Change default CentOS version to 8 Stream

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #1709 from kshtsk/wip-auto-get-mt-for-kill

kill: get machine type from paddles for the run

kill: get machine type from paddles for the run

When calling kill for a run which is still in beanstalkd
queue there is no job directories created in archive
and there is no way to find out which machine type to use
the teuthology-kill reports that you must manually provide
machine type for the run, which is often borring and quite
inconvenient.

Instead of torturing user to recall a tube for the run name
we just ask paddles if it has anything logged and use the
machine type from what we receive from it.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

opsys.py: Change default CentOS version to 8 Stream

8.X is no more

Signed-off-by: David Galloway <dgallowa@redhat.com>

docs/docker-compose: add ansible inventory to README

Added instructions that will help users
set up ansible inventory files in after they
built their local teuthology container. This
part is needed when ansible tasks are executed
when jobs are running.

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #1706 from batrick/log-warning

teuthology: convert deprecated method name

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology: convert deprecated method name

Avoiding this warning:

    /home/runner/work/teuthology/teuthology/teuthology/task/install/__init__.py:285: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1678 from ceph/wip-ksirivad-move-ansible-log

task/ansible: move ansible logs to ansible.log

task/ansible: move ansible logs to ansible.log

Currently ansible tasks are logging to
teuthology.log. We want to separate this to
it's own log files for better readability.

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #1665 from ceph/wip-ksirivad-docker-compose-script

Added dockerfile + docker-compose script dev setup

Removed beanstalkd in teuthology image

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Moved Dockerfile

Moved docker file to:

docs/docker-compose/Dockerfile

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>