Dockerfile: Optimize pip installs

With this change, we can avoid rerunning bootstrap on build if
requirements.txt hasn't changed. Build times after non-requirements
changes go from 3-5min to ~2s.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Optionally skip teuthology install

The docker-compose dev setup was the inspiration for this feature; it
allows us to split the installation of dependencies and the project
itself into two phases. This in turn allows us to avoid rerunning
bootstrap if requirements.txt didn't change.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop virtualenv in favor of venv

We don't need it, and if it is present, setuptools will use
easy_install instead of building wheels.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1750 from tchaikov/pass-extra

update-requirements.sh: pass --extra=test to pip-compile

Merge pull request #1698 from kshtsk/wip-raise-if-not-reconnect

orchestra/remote: throw exception on reconnect failed

update-requirements.sh: pass --extra=test to pip-compile

otherwise the requirements used by test would be missed

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1749 from tchaikov/setup-add-nose

setup.cfg: add nose to "options.extras_require/test"

setup.py: dedup the requires

* move the dependencies from options.install_requires to
  options.extras_require if those dependencies are only used by tests.
  as per
  https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/#id5
  > install_requires is a setuptools setup.py keyword that should be
  > used to specify what a project minimally needs to run correctly.
* remove the duplicated dependencies from options.extras_require
  if they are also listed in options.install_requires.
* update requirements.txt using
  "pip-compile --extra=test pyproject.toml", so requirements.txt
  can include the dependencies required by tests.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

scrape: drop dependency of "six"

since we don't need to be compatible with python2, let's stop using
the six package.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

setup.cfg: add nose to "options.extras_require/test"

some tests are using nose python package. since teuthology is
responsible for preparing the deps to run the tests. let's add
nose back to setup.cfg.

the "nose" package was included in requirements.txt before
21372fd9b2b87bb6417929f10097fba7db9cdca9. in this change, it,
it is brought back.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #1743 from ceph/packaging

Revised packaging updates

Merge pull request #1747 from ceph/ca-test-fix

task/ceph_ansible: Fix attribute handling

Merge pull request #1746 from rishabh-d-dave/py3104-MutableMapping

config: import MuableMapping based on python version

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

task/ceph_ansible: Fix attribute handling

... in get_host_vars(). This had broken unit tests when a top-level
"ceph_ansible" key is present in the teuthology config.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Drop basepython lines

They're redundant.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Install 'test' extras for py.test envs

Signed-off-by: Zack Cerza <zack@redhat.com>

Modernized versioning

Using setuptools_scm; we can automatically generate version numbers
instead of using this odd combination of hardcoding and running git
commands. This method is faster as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Install 'test' extras

So that we can use the 'tests' task.

Signed-off-by: Zack Cerza <zack@redhat.com>

run_tasks: Do not mask missing task dependencies

While doing packaging work, I noticed that teuthology began claiming it
couldn't find the 'tests' task. After some slightly painful debugging I
realized the issue was that the task was trying to import pytest, which
wasn't installed. The ModuleNotFoundError that was being raised was
being confused with the exception that would be raised if the task
couldn't be found at all. With this change, we see the root cause.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Update pytest_runtest_makereport

And fix a typo

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop python2 support

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Set isolated_build

This also requires us to re-add mock to the py3 env.

Signed-off-by: Zack Cerza <zack@redhat.com>

TestOpenStackBase.setup: Remove arguments

In newer py.test versions, the test method itself was being passed as
the conf arg, causing errors. We don't even use it, so remove it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Tolerate a missing manhole package

The vast majority of users won't ever need this.

Signed-off-by: Zack Cerza <zack@redhat.com>

MANIFEST.in: Add tox.ini and pytest.ini

Signed-off-by: Zack Cerza <zack@redhat.com>

Replace setup.py with setup.cfg and pyproject.toml

Signed-off-by: Zack Cerza <zack@redhat.com>

config: import MuableMapping based on python version

MutableMapping has been moved from collections to collections.abc.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #1739 from ceph/docker-compose

docker-compose: "wait mode"

docker-compose.yml: Start beanstalkd sooner

There was a bit of a race condition happening in wait mode, where
teuthology would start up before beanstalkd and fail immediately.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Default to RSA keys

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add wait mode

Instead of scheduling the teuthology:no-ceph suite and then exiting
when it finishes, schedule nothing and don't exit automatically. This
allows running whatever - and however many - tests are desired.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Document containerized nodes

Signed-off-by: Zack Cerza <zack@redhat.com>

.dockerignore: Simplify paths

The we we'd used wildcards, we were only ignoring directory contents -
not the directories themselves. Also add the tox temporary directory.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Use repo root as build context

Rather than add more logic to support things like cloning different
forks of the repo, let's just not clone at all.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Make sure to run docker-compose down

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1741 from ceph/dc-fixes

teuthology.sh: Run tests using current branch

Merge pull request #1742 from ceph/repo_logging

repo_utils: Fix logging in bootstrap_teuthology

repo_utils: Fix logging in bootstrap_teuthology

Signed-off-by: Zack Cerza <zack@redhat.com>

worker: Don't log incorrect teuthology sha1

If teuthology_path is set, we don't fetch a separate copy of the repo.
In that case, the sha1 logged here may be incorrect.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Run tests using current branch

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1740 from ceph/wip-revert4333

Revert "Merge pull request #1737 from ceph/build"

Revert "Merge pull request #1737 from ceph/build"

This reverts commit 43327234bc2c8066c173e4c6d602c8a0a05db642, reversing
changes made to a347b2ac5400a43ba76a678399b1deda7dae1a6c.

Merge pull request #1737 from ceph/build

Packaging overhaul

bootstrap: Drop pyc removal

We stopped writing these files years ago.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Optionally skip teuthology install

The docker-compose dev setup was the inspiration for this feature; it
allows us to split the installation of dependencies and the project
itself into two phases. This in turn allows us to avoid rerunning
bootstrap if requirements.txt didn't change.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Drop basepython lines

They're redundant.

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Install 'test' extras for py.test envs

Signed-off-by: Zack Cerza <zack@redhat.com>

Modernized versioning

Using setuptools_scm; we can automatically generate version numbers
instead of using this odd combination of hardcoding and running git
commands. This method is faster as well.

Signed-off-by: Zack Cerza <zack@redhat.com>

Remove references to teuth_bin_path

This effectively hardcoded the path to the virtualenv we'll use, which
aside from being bad generally, also prevented us from supporting
absolute-pathed virtualenvs.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Allow absolute-path $VENV

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Drop virtualenv in favor of venv

We don't need it, and if it is present, setuptools will use
easy_install instead of building wheels.

Signed-off-by: Zack Cerza <zack@redhat.com>

bootstrap: Install 'test' extras

So that we can use the 'tests' task.

Signed-off-by: Zack Cerza <zack@redhat.com>

run_tasks: Do not mask missing task dependencies

While doing packaging work, I noticed that teuthology began claiming it
couldn't find the 'tests' task. After some slightly painful debugging I
realized the issue was that the task was trying to import pytest, which
wasn't installed. The ModuleNotFoundError that was being raised was
being confused with the exception that would be raised if the task
couldn't be found at all. With this change, we see the root cause.

Signed-off-by: Zack Cerza <zack@redhat.com>

task.tests: Update pytest_runtest_makereport

And fix a typo

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1735 from ceph/rhel85

opsys.py: RHEL8.5 is latest

bootstrap: Drop python2 support

Signed-off-by: Zack Cerza <zack@redhat.com>

tox.ini: Set isolated_build

This also requires us to re-add mock to the py3 env.

Signed-off-by: Zack Cerza <zack@redhat.com>

TestOpenStackBase.setup: Remove arguments

In newer py.test versions, the test method itself was being passed as
the conf arg, causing errors. We don't even use it, so remove it.

Signed-off-by: Zack Cerza <zack@redhat.com>

Tolerate a missing manhole package

The vast majority of users won't ever need this.

Signed-off-by: Zack Cerza <zack@redhat.com>

MANIFEST.in: Add tox.ini and pytest.ini

Signed-off-by: Zack Cerza <zack@redhat.com>

Replace setup.py with setup.cfg and pyproject.toml

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #1734 from ceph/dependabot/pip/paramiko-2.10.1

build(deps): bump paramiko from 2.7.2 to 2.10.1

opsys.py: RHEL8.5 is latest

Signed-off-by: David Galloway <dgallowa@redhat.com>

build(deps): bump paramiko from 2.7.2 to 2.10.1

Bumps [paramiko](https://github.com/paramiko/paramiko) from 2.7.2 to 2.10.1.
- [Release notes](https://github.com/paramiko/paramiko/releases)
- [Changelog](https://github.com/paramiko/paramiko/blob/main/NEWS)
- [Commits](https://github.com/paramiko/paramiko/compare/2.7.2...2.10.1)

---
updated-dependencies:
- dependency-name: paramiko
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1726 from ceph/bs-mac-noauto

bootstrap: Don't auto-install on Mac

Merge pull request #1727 from ceph/docker-compose

docker-compose: Use containerized testnodes to run the teuthology suite

docker-compose: Update readme

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Increase paddles' check time

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Separate build contexts

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Allow different custom config

When using "real" testnodes, we should not use the custom config we've
been using for containerized nodes. When using "real" testnodes, default
 to not using one, but allow specifying a path in $CUSTOM_CONF.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-update-inventory: Add machine type flag

With this change, the tool can be trivially used to add new nodes to
paddles.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Write SSH key stubs later in build

We'd rather duplicate this layer than the package-install layer if a
user is going to use both containerized testnodes and "real" ones.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Optionally use an inventory repo

This is being added to support "real" testnodes without having to
manually copy inventory fragments over.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Don't double-clone teuthology

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Optionally use an existing SSH key

This will only work with "real" testnodes; with containerized testnodes
there's no reason not to use a generated key.

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Populate "real" testnodes

... if we're using them, that is.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Fix old typo

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Copy config files more consistently

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Exit after jobs are done

Signed-off-by: Zack Cerza <zack@redhat.com>

dispatcher: Optionally exit when queue is empty

Signed-off-by: Zack Cerza <zack@redhat.com>

supervisor: Return job's return code

Signed-off-by: Zack Cerza <zack@redhat.com>

Add 'integration' GitHub Action

This will use our recent docker-compose work.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Use current branch by default

Since the main use-case for this work at the moment is self-testing
teuthology, let's make it easy to test the same branch that's being
worked on.

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Use the teuthology:no-ceph suite

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode: Install apache2 and nfs

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose.yml: Force testnode arch

I initially worked on this on an M1 Mac, and that caused problems with
scheduling due to a lack of aarch64 packages. The M1 can run amd64
images, though.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode: Add stop script

So that each container marks itself down before stopping.

Signed-off-by: Zack Cerza <zack@redhat.com>

testnode.sh: Retry submitting to paddles

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile-testnode: Switch to Ubuntu

Signed-off-by: Zack Cerza <zack@redhat.com>

nuke: In containers, simply kill sshd

At least with the testnode containers I'm working on at the moment,
this appears to be the best way to shut them down from inside.

Signed-off-by: Zack Cerza <zack@redhat.com>

start.sh: Copy conf files into container

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose.yml: Use three testnodes

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile-testnode: Install Python packages

These are needed by ansible

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology.sh: Use absolute paths

Signed-off-by: Zack Cerza <zack@redhat.com>

.gitignore: Ignore more built docs

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add .dockerignore

Signed-off-by: Zack Cerza <zack@redhat.com>

.teuthology.yaml: Use absolute paths

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add custom_conf.yaml

This will be merged into generated job configs.

Signed-off-by: Zack Cerza <zack@redhat.com>

Dockerfile: Split out entrypoint to teuthology.sh

Signed-off-by: Zack Cerza <zack@redhat.com>

docker-compose: Add testnode_sudoers

Signed-off-by: Zack Cerza <zack@redhat.com>