From: Zack Cerza <zack@redhat.com>
Date: Tue, 2 Aug 2022 17:06:47 +0000 (-0600)
Subject: docs/docker-compose: Move to new containers/ dir
X-Git-Url: http://git.apps.os.sepia.ceph.com/?a=commitdiff_plain;h=efead9f5b298937075e1aa84d8946ab9a997e105;p=teuthology.git

docs/docker-compose: Move to new containers/ dir

This work is being mostly replaced by
https://github.com/zmc/ceph-devstack. However, since that tool has
fairly strict requirements, let's try to preserve the docker-compose
functionality. It will still be useful for tests that don't require
OSDs.

Signed-off-by: Zack Cerza <zack@redhat.com>
---

diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
index 69a55b8773..6761c04f1f 100644
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -9,4 +9,4 @@ jobs:
       - uses: actions/checkout@v2
       - name: Test using docker-compose
         run: ./start.sh
-        working-directory: ./docs/docker-compose
+        working-directory: ./containers/docker-compose
diff --git a/beanstalk/alpine/Dockerfile b/beanstalk/alpine/Dockerfile
deleted file mode 100644
index 7afb0005b8..0000000000
--- a/beanstalk/alpine/Dockerfile
+++ /dev/null
@@ -1,13 +0,0 @@
-# For beanstalkd 1.12 use edge branch
-#FROM alpine:edge
-
-FROM alpine:3.12.3
-
-MAINTAINER Kyrylo Shatskyy <kyrylo.shatskyy@suse.com>
-
-RUN apk update && apk add beanstalkd beanstalkd-doc
-
-ENV BEANSTALK_ADDR "0.0.0.0"
-ENV BEANSTALK_PORT "11300"
-
-CMD /usr/bin/beanstalkd -V -l $BEANSTALK_ADDR -p $BEANSTALK_PORT
diff --git a/containers/beanstalk/alpine/Dockerfile b/containers/beanstalk/alpine/Dockerfile
new file mode 100644
index 0000000000..7afb0005b8
--- /dev/null
+++ b/containers/beanstalk/alpine/Dockerfile
@@ -0,0 +1,13 @@
+# For beanstalkd 1.12 use edge branch
+#FROM alpine:edge
+
+FROM alpine:3.12.3
+
+MAINTAINER Kyrylo Shatskyy <kyrylo.shatskyy@suse.com>
+
+RUN apk update && apk add beanstalkd beanstalkd-doc
+
+ENV BEANSTALK_ADDR "0.0.0.0"
+ENV BEANSTALK_PORT "11300"
+
+CMD /usr/bin/beanstalkd -V -l $BEANSTALK_ADDR -p $BEANSTALK_PORT
diff --git a/containers/docker-compose/README.md b/containers/docker-compose/README.md
new file mode 100644
index 0000000000..e69d1bf7f1
--- /dev/null
+++ b/containers/docker-compose/README.md
@@ -0,0 +1,93 @@
+# Teuthology Development Environment Instruction
+
+The purpose of this guide is to help developers set
+up a development environment for Teuthology. We will be using 
+Docker to set up all the containers for
+Postgres, Paddles, Pulpito, Beanstalk, and Teuthology.
+
+Currently, it's possible to execute against two classes of test nodes: 
+
+* Using containerized test nodes
+  * Advantage: No need for a lab at all!
+  * Disadvantage: Cannot run all Ceph tests; best for exercising the framework itself
+* Using nodes from an existing lab (e.g. the Sepia lab)
+  * Advantage: Can run all Ceph tests
+  * Disadvantage: Requires lab access
+
+
+Additionally, there are two modes of execution:
+* One-shot (the default): Containers start up, schedule and run the `teuthology:no-ceph` suite, and shut down. Success or failure is indicated by the `start.sh` exit code.
+* Wait: Containers start up, and `teuthology-dispatcher` is started, but no jobs are scheduled. Runs until the user presses Ctrl-C or `docker-compose down` is run.
+  
+The teuthology container will be built with code from the repository clone that's currently in use.
+
+## Prerequisites
+
+### Installing and Running Docker
+
+For Docker installation see: 
+https://docs.docker.com/get-docker/
+
+### Using Containerized Nodes
+
+There's nothing special to do; see the Running Tests section below.
+
+### Using an Existing Lab
+
+This document assumes you have access to the lab that you intend to use, and that you're already familiar with its VPN and SSH infrastructure.
+
+Depending on your local operating system, it may be necessary to connect to the VPN before starting Docker.
+
+#### Using your SSH private key
+
+In your local shell, simply:
+```bash
+export SSH_PRIVKEY_PATH=$HOME/.ssh/id_rsa
+```
+The teuthology container will write it to a file at runtime.
+
+#### Reserving Machines in the Lab
+
+Taking the Sepia lab as an example once again, most users will want to do something like:
+
+```bash
+ssh teuthology.front.sepia.ceph.com
+~/teuthology/virtualenv/bin/teuthology-lock \
+  --lock-many 1 \
+  --machine-type smithi \
+  --desc "teuthology dev testing"
+```
+
+When you are done, don't forget to unlock!
+
+#### Using Lab Machines
+
+Once you have your machines locked, you need to provide a list of their hostnames and their machine type:
+
+```bash
+export TESTNODES="smithi999.front.sepia.ceph.com,smithi123.front.sepia.ceph.com"
+export MACHINE_TYPE="smithi"
+```
+
+If the lab uses a "secrets" or "inventory" repository for [ceph-cm-ansible](https://github.com/ceph/ceph-cm-ansible), you'll need to provide a URL for that. In Sepia:
+```bash
+export ANSIBLE_INVENTORY_REPO="https://github.com/ceph/ceph-sepia-secrets"
+```
+This repo will be cloned locally, using your existing `git` configuration, and copied into the teuthology container at build time.
+
+## Running Tests
+
+To run the default `teuthology:no-ceph` suite in one-shot mode:
+```bash
+./start.sh
+```
+
+To run in wait mode:
+```bash
+TEUTHOLOGY_WAIT=1 ./start.sh
+```
+
+To schedule tests in wait mode:
+```bash
+docker exec docker-compose_teuthology_1 /venv/bin/teuthology-suite ...
+```
\ No newline at end of file
diff --git a/containers/docker-compose/docker-compose.yml b/containers/docker-compose/docker-compose.yml
new file mode 100644
index 0000000000..30cc9fc328
--- /dev/null
+++ b/containers/docker-compose/docker-compose.yml
@@ -0,0 +1,92 @@
+version: '3.8'
+
+services:
+  postgres:
+    image: postgres:14
+    healthcheck:
+      test: [ "CMD", "pg_isready", "-q", "-d", "paddles", "-U", "admin" ]
+      timeout: 5s
+      interval: 10s
+      retries: 2
+    environment:
+      - POSTGRES_USER=root
+      - POSTGRES_PASSWORD=password
+      - APP_DB_USER=admin
+      - APP_DB_PASS=password
+      - APP_DB_NAME=paddles
+    volumes:
+      - ../postgres/db:/docker-entrypoint-initdb.d/:Z
+    ports:
+      - 5432:5432
+  paddles:
+    image: quay.io/ceph-infra/paddles
+    environment: 
+      PADDLES_SERVER_HOST: 0.0.0.0
+      PADDLES_SQLALCHEMY_URL: postgresql+psycopg2://admin:password@postgres:5432/paddles
+    depends_on:
+        postgres:
+            condition: service_healthy
+    links:
+        - postgres
+    healthcheck:
+        test: ["CMD", "curl", "-f", "http://0.0.0.0:8080"]
+        timeout: 5s
+        interval: 30s
+        retries: 2
+    ports: 
+        - 8080:8080
+  pulpito:
+    image: quay.io/ceph-infra/pulpito
+    environment:
+      PULPITO_PADDLES_ADDRESS: http://paddles:8080
+    depends_on:
+        paddles:
+            condition: service_healthy
+    links:
+        - paddles
+    healthcheck:
+        test: ["CMD", "curl", "-f", "http://0.0.0.0:8081"]
+        timeout: 5s
+        interval: 10s
+        retries: 2
+    ports:
+        - 8081:8081
+  beanstalk:
+        build: ../beanstalk/alpine
+        ports:
+            - "11300:11300"
+  teuthology:
+    build:
+      context: ../../
+      dockerfile: ./containers/teuthology-dev/Dockerfile
+      args:
+        SSH_PRIVKEY_FILE: $SSH_PRIVKEY_FILE
+    depends_on:
+        paddles:
+            condition: service_healthy
+    links:
+        - paddles
+        - beanstalk
+    environment:
+      SSH_PRIVKEY:
+      SSH_PRIVKEY_FILE:
+      MACHINE_TYPE:
+      TESTNODES:
+      TEUTHOLOGY_WAIT:
+      TEUTH_BRANCH:
+  testnode:
+    build:
+      context: ../testnode-docker-compose
+      dockerfile: ./Dockerfile
+    deploy:
+      replicas: 3
+    depends_on:
+        paddles:
+            condition: service_healthy
+    links:
+      - paddles
+    ports:
+      - "22"
+    environment:
+      SSH_PUBKEY:
+    platform: linux/amd64
diff --git a/containers/docker-compose/start.sh b/containers/docker-compose/start.sh
new file mode 100755
index 0000000000..871258c5be
--- /dev/null
+++ b/containers/docker-compose/start.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+set -e
+export TEUTHOLOGY_BRANCH=${TEUTHOLOGY_BRANCH:-$(git branch --show-current)}
+export TEUTH_BRANCH=${TEUTHOLOGY_BRANCH}
+if [ -n "$ANSIBLE_INVENTORY_REPO" ]; then
+    basename=$(basename $ANSIBLE_INVENTORY_REPO | cut -d. -f1)
+    if [ ! -d "$basename" ]; then
+        git clone \
+            --depth 1 \
+            $ANSIBLE_INVENTORY_REPO
+    fi
+    mkdir -p teuthology/ansible_inventory
+    cp -rf $basename/ansible/ teuthology/ansible_inventory
+    if [ ! -d teuthology/ansible_inventory/hosts ]; then
+        mv -f teuthology/ansible_inventory/inventory teuthology/ansible_inventory/hosts
+    fi
+fi
+# Make the hosts and secrets directories, so that the COPY instruction in the 
+# Dockerfile does not cause a build failure when not using this feature.
+mkdir -p teuthology/ansible_inventory/hosts teuthology/ansible_inventory/secrets
+
+if [ -n "$CUSTOM_CONF" ]; then
+    cp "$CUSTOM_CONF" teuthology/
+fi
+
+# Generate an SSH keypair to use if necessary
+if [ -z "$SSH_PRIVKEY_PATH" ]; then
+    SSH_PRIVKEY_PATH=$(mktemp -u /tmp/teuthology-ssh-key-XXXXXX)
+    ssh-keygen -t rsa -N '' -f $SSH_PRIVKEY_PATH
+    export SSH_PRIVKEY=$(cat $SSH_PRIVKEY_PATH)
+    export SSH_PUBKEY=$(cat $SSH_PRIVKEY_PATH.pub)
+    export SSH_PRIVKEY_FILE=id_rsa
+else
+    export SSH_PRIVKEY=$(cat $SSH_PRIVKEY_PATH)
+    export SSH_PRIVKEY_FILE=$(basename $SSH_PRIVKEY_PATH | cut -d. -f1)
+fi
+
+if [ -z "$TEUTHOLOGY_WAIT" ]; then
+    DC_EXIT_FLAG='--abort-on-container-exit --exit-code-from teuthology'
+    DC_AUTO_DOWN_CMD='docker-compose down'
+fi
+export TEUTHOLOGY_WAIT
+
+trap "docker-compose down" SIGINT
+docker-compose up \
+    --build \
+    $DC_EXIT_FLAG
+$DC_AUTO_DOWN_CMD
diff --git a/containers/postgres/db/01-init.sh b/containers/postgres/db/01-init.sh
new file mode 100755
index 0000000000..b9e5adc2ff
--- /dev/null
+++ b/containers/postgres/db/01-init.sh
@@ -0,0 +1,8 @@
+set -e
+export PGPASSWORD=$POSTGRES_PASSWORD;
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
+  CREATE USER $APP_DB_USER WITH PASSWORD '$APP_DB_PASS';
+  CREATE DATABASE $APP_DB_NAME;
+  GRANT ALL PRIVILEGES ON DATABASE $APP_DB_NAME TO $APP_DB_USER;
+  \connect $APP_DB_NAME $APP_DB_USER
+EOSQL
\ No newline at end of file
diff --git a/containers/testnode-docker-compose/Dockerfile b/containers/testnode-docker-compose/Dockerfile
new file mode 100644
index 0000000000..016d32117a
--- /dev/null
+++ b/containers/testnode-docker-compose/Dockerfile
@@ -0,0 +1,26 @@
+FROM ubuntu:focal
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt update && \
+    apt -y install \
+        sudo \
+        openssh-server \
+        hostname \
+        curl \
+        python3-pip \
+        apache2 \
+        nfs-kernel-server && \
+    apt clean all
+COPY testnode_start.sh /
+COPY testnode_stop.sh /
+COPY testnode_sudoers /etc/sudoers.d/teuthology
+RUN \
+    ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N '' && \
+    sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config && \
+    mkdir -p /root/.ssh && \
+    chmod 700 /root/.ssh && \
+    useradd -g sudo ubuntu && \
+    mkdir -p /home/ubuntu/.ssh && \
+    chmod 700 /home/ubuntu/.ssh && \
+    chown -R ubuntu /home/ubuntu
+EXPOSE 22
+ENTRYPOINT /testnode_start.sh
diff --git a/containers/testnode-docker-compose/testnode_start.sh b/containers/testnode-docker-compose/testnode_start.sh
new file mode 100755
index 0000000000..d29c3b6d0a
--- /dev/null
+++ b/containers/testnode-docker-compose/testnode_start.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/bash
+set -x
+echo "$SSH_PUBKEY" > /root/.ssh/authorized_keys
+echo "$SSH_PUBKEY" > /home/ubuntu/.ssh/authorized_keys
+chown ubuntu /home/ubuntu/.ssh/authorized_keys
+payload="{\"name\": \"$(hostname)\", \"machine_type\": \"testnode\", \"up\": true, \"locked\": false, \"os_type\": \"ubuntu\", \"os_version\": \"20.04\"}"
+for i in $(seq 1 5); do
+    echo "attempt $i"
+    curl -v -f -d "$payload" http://paddles:8080/nodes/ && break
+    sleep 1
+done
+mkdir -p /run/sshd
+exec /usr/sbin/sshd -D
diff --git a/containers/testnode-docker-compose/testnode_stop.sh b/containers/testnode-docker-compose/testnode_stop.sh
new file mode 100755
index 0000000000..2e1044d807
--- /dev/null
+++ b/containers/testnode-docker-compose/testnode_stop.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/bash
+set -x
+hostname=$(hostname)
+payload="{\"name\": \"$hostname\", \"machine_type\": \"testnode\", \"up\": false}"
+for i in $(seq 1 5); do
+    echo "attempt $i"
+    curl -s -f -X PUT -d "$payload" http://paddles:8080/nodes/$hostname/ && break
+    sleep 1
+done
+pkill sshd
\ No newline at end of file
diff --git a/containers/testnode-docker-compose/testnode_sudoers b/containers/testnode-docker-compose/testnode_sudoers
new file mode 100644
index 0000000000..35828ad9be
--- /dev/null
+++ b/containers/testnode-docker-compose/testnode_sudoers
@@ -0,0 +1,4 @@
+%sudo ALL=(ALL) NOPASSWD: ALL
+# For ansible pipelining
+Defaults !requiretty
+Defaults visiblepw
diff --git a/containers/testnode/Dockerfile b/containers/testnode/Dockerfile
new file mode 100644
index 0000000000..016d32117a
--- /dev/null
+++ b/containers/testnode/Dockerfile
@@ -0,0 +1,26 @@
+FROM ubuntu:focal
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt update && \
+    apt -y install \
+        sudo \
+        openssh-server \
+        hostname \
+        curl \
+        python3-pip \
+        apache2 \
+        nfs-kernel-server && \
+    apt clean all
+COPY testnode_start.sh /
+COPY testnode_stop.sh /
+COPY testnode_sudoers /etc/sudoers.d/teuthology
+RUN \
+    ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N '' && \
+    sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config && \
+    mkdir -p /root/.ssh && \
+    chmod 700 /root/.ssh && \
+    useradd -g sudo ubuntu && \
+    mkdir -p /home/ubuntu/.ssh && \
+    chmod 700 /home/ubuntu/.ssh && \
+    chown -R ubuntu /home/ubuntu
+EXPOSE 22
+ENTRYPOINT /testnode_start.sh
diff --git a/containers/testnode/testnode_start.sh b/containers/testnode/testnode_start.sh
new file mode 100755
index 0000000000..d29c3b6d0a
--- /dev/null
+++ b/containers/testnode/testnode_start.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/bash
+set -x
+echo "$SSH_PUBKEY" > /root/.ssh/authorized_keys
+echo "$SSH_PUBKEY" > /home/ubuntu/.ssh/authorized_keys
+chown ubuntu /home/ubuntu/.ssh/authorized_keys
+payload="{\"name\": \"$(hostname)\", \"machine_type\": \"testnode\", \"up\": true, \"locked\": false, \"os_type\": \"ubuntu\", \"os_version\": \"20.04\"}"
+for i in $(seq 1 5); do
+    echo "attempt $i"
+    curl -v -f -d "$payload" http://paddles:8080/nodes/ && break
+    sleep 1
+done
+mkdir -p /run/sshd
+exec /usr/sbin/sshd -D
diff --git a/containers/testnode/testnode_stop.sh b/containers/testnode/testnode_stop.sh
new file mode 100755
index 0000000000..2e1044d807
--- /dev/null
+++ b/containers/testnode/testnode_stop.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/bash
+set -x
+hostname=$(hostname)
+payload="{\"name\": \"$hostname\", \"machine_type\": \"testnode\", \"up\": false}"
+for i in $(seq 1 5); do
+    echo "attempt $i"
+    curl -s -f -X PUT -d "$payload" http://paddles:8080/nodes/$hostname/ && break
+    sleep 1
+done
+pkill sshd
\ No newline at end of file
diff --git a/containers/testnode/testnode_sudoers b/containers/testnode/testnode_sudoers
new file mode 100644
index 0000000000..35828ad9be
--- /dev/null
+++ b/containers/testnode/testnode_sudoers
@@ -0,0 +1,4 @@
+%sudo ALL=(ALL) NOPASSWD: ALL
+# For ansible pipelining
+Defaults !requiretty
+Defaults visiblepw
diff --git a/containers/teuthology-dev/.teuthology.yaml b/containers/teuthology-dev/.teuthology.yaml
new file mode 100644
index 0000000000..bac8ec1aaf
--- /dev/null
+++ b/containers/teuthology-dev/.teuthology.yaml
@@ -0,0 +1,9 @@
+queue_host: beanstalk
+queue_port: 11300
+lock_server: http://paddles:8080
+results_server: http://paddles:8080
+results_ui_server: http://pulpito:8081/
+teuthology_path: /teuthology
+archive_base: /archive_dir
+reserve_machines: 0
+lab_domain: ''
\ No newline at end of file
diff --git a/containers/teuthology-dev/Dockerfile b/containers/teuthology-dev/Dockerfile
new file mode 100644
index 0000000000..f350b31dbd
--- /dev/null
+++ b/containers/teuthology-dev/Dockerfile
@@ -0,0 +1,43 @@
+FROM ubuntu:latest
+ARG SSH_PRIVKEY_FILE=id_ed25519
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update && \
+    apt-get install -y \
+    git \
+    qemu-utils \
+    python3-dev \
+    libssl-dev \
+    ipmitool \
+    python3-pip \
+    python3-venv \
+    vim \
+    libev-dev \
+    libvirt-dev \
+    libffi-dev \
+    libyaml-dev \
+    lsb-release && \
+    apt-get clean all
+WORKDIR /teuthology
+COPY requirements.txt bootstrap /teuthology/
+RUN \
+    cd /teuthology && \
+    mkdir ../archive_dir && \
+    mkdir log && \
+    chmod +x /teuthology/bootstrap && \
+    PIP_INSTALL_FLAGS="-r requirements.txt" ./bootstrap
+COPY . /teuthology
+RUN \
+    ./bootstrap
+COPY containers/teuthology-dev/containerized_node.yaml /teuthology
+COPY containers/teuthology-dev/.teuthology.yaml /root
+COPY containers/teuthology-dev/teuthology.sh /
+RUN mkdir -p /etc/ansible
+COPY containers/teuthology-dev/ansible_inventory/hosts /etc/ansible/
+COPY containers/teuthology-dev/ansible_inventory/secrets /etc/ansible/
+RUN \
+    mkdir $HOME/.ssh && \
+    touch $HOME/.ssh/${SSH_PRIVKEY_FILE} && \
+    chmod 600 $HOME/.ssh/${SSH_PRIVKEY_FILE} && \
+    echo "StrictHostKeyChecking=no" > $HOME/.ssh/config && \
+    echo "UserKnownHostsFile=/dev/null" >> $HOME/.ssh/config
+ENTRYPOINT /teuthology.sh
\ No newline at end of file
diff --git a/containers/teuthology-dev/containerized_node.yaml b/containers/teuthology-dev/containerized_node.yaml
new file mode 100644
index 0000000000..02304886c4
--- /dev/null
+++ b/containers/teuthology-dev/containerized_node.yaml
@@ -0,0 +1,8 @@
+overrides:
+  ansible.cephlab:
+    skip_tags: "timezone,nagios,monitoring-scripts,ssh,hostname,pubkeys,zap,sudoers,kerberos,selinux,lvm,ntp-client,resolvconf,packages,cpan,nfs"
+    vars:
+      containerized_node: true
+      ansible_user: root
+      cm_user: root
+      start_rpcbind: false
diff --git a/containers/teuthology-dev/teuthology.sh b/containers/teuthology-dev/teuthology.sh
new file mode 100755
index 0000000000..0378f93d44
--- /dev/null
+++ b/containers/teuthology-dev/teuthology.sh
@@ -0,0 +1,46 @@
+#!/usr/bin/bash
+set -e
+# We don't want -x yet, in case the private key is sensitive
+if [ -n "$SSH_PRIVKEY_FILE" ]; then
+    echo "$SSH_PRIVKEY" > $HOME/.ssh/$SSH_PRIVKEY_FILE
+fi
+source /teuthology/virtualenv/bin/activate
+set -x
+if [ -n "$TESTNODES" ]; then
+    for node in $(echo $TESTNODES | tr , ' '); do
+        teuthology-update-inventory -m $MACHINE_TYPE $node
+    done
+    CUSTOM_CONF=${CUSTOM_CONF:-}
+else
+    CUSTOM_CONF=/teuthology/containerized_node.yaml
+fi
+export MACHINE_TYPE=${MACHINE_TYPE:-testnode}
+if [ -z "$TEUTHOLOGY_WAIT" ]; then
+    if [ -n "$TEUTH_BRANCH" ]; then
+      TEUTH_BRANCH_FLAG="--teuthology-branch $TEUTH_BRANCH"
+    fi
+    teuthology-suite -v \
+        $TEUTH_BRANCH_FLAG \
+        --ceph-repo https://github.com/ceph/ceph.git \
+        --suite-repo https://github.com/ceph/ceph.git \
+        -c main \
+        -m $MACHINE_TYPE \
+        --limit 1 \
+        -n 100 \
+        --suite teuthology:no-ceph \
+        --filter-out "libcephfs,kclient,stream,centos,rhel" \
+        -d ubuntu -D 20.04 \
+        --suite-branch main \
+        --subset 9000/100000 \
+        -p 75 \
+        --seed 349 \
+        --force-priority \
+        $CUSTOM_CONF
+    DISPATCHER_EXIT_FLAG='--exit-on-empty-queue'
+    teuthology-queue -m $MACHINE_TYPE -s | \
+      python3 -c "import sys, json; assert json.loads(sys.stdin.read())['count'] > 0, 'queue is empty!'"
+fi
+teuthology-dispatcher -v \
+    --log-dir /teuthology/log \
+    --tube $MACHINE_TYPE \
+    $DISPATCHER_EXIT_FLAG
diff --git a/docs/docker-compose/README.md b/docs/docker-compose/README.md
deleted file mode 100644
index e69d1bf7f1..0000000000
--- a/docs/docker-compose/README.md
+++ /dev/null
@@ -1,93 +0,0 @@
-# Teuthology Development Environment Instruction
-
-The purpose of this guide is to help developers set
-up a development environment for Teuthology. We will be using 
-Docker to set up all the containers for
-Postgres, Paddles, Pulpito, Beanstalk, and Teuthology.
-
-Currently, it's possible to execute against two classes of test nodes: 
-
-* Using containerized test nodes
-  * Advantage: No need for a lab at all!
-  * Disadvantage: Cannot run all Ceph tests; best for exercising the framework itself
-* Using nodes from an existing lab (e.g. the Sepia lab)
-  * Advantage: Can run all Ceph tests
-  * Disadvantage: Requires lab access
-
-
-Additionally, there are two modes of execution:
-* One-shot (the default): Containers start up, schedule and run the `teuthology:no-ceph` suite, and shut down. Success or failure is indicated by the `start.sh` exit code.
-* Wait: Containers start up, and `teuthology-dispatcher` is started, but no jobs are scheduled. Runs until the user presses Ctrl-C or `docker-compose down` is run.
-  
-The teuthology container will be built with code from the repository clone that's currently in use.
-
-## Prerequisites
-
-### Installing and Running Docker
-
-For Docker installation see: 
-https://docs.docker.com/get-docker/
-
-### Using Containerized Nodes
-
-There's nothing special to do; see the Running Tests section below.
-
-### Using an Existing Lab
-
-This document assumes you have access to the lab that you intend to use, and that you're already familiar with its VPN and SSH infrastructure.
-
-Depending on your local operating system, it may be necessary to connect to the VPN before starting Docker.
-
-#### Using your SSH private key
-
-In your local shell, simply:
-```bash
-export SSH_PRIVKEY_PATH=$HOME/.ssh/id_rsa
-```
-The teuthology container will write it to a file at runtime.
-
-#### Reserving Machines in the Lab
-
-Taking the Sepia lab as an example once again, most users will want to do something like:
-
-```bash
-ssh teuthology.front.sepia.ceph.com
-~/teuthology/virtualenv/bin/teuthology-lock \
-  --lock-many 1 \
-  --machine-type smithi \
-  --desc "teuthology dev testing"
-```
-
-When you are done, don't forget to unlock!
-
-#### Using Lab Machines
-
-Once you have your machines locked, you need to provide a list of their hostnames and their machine type:
-
-```bash
-export TESTNODES="smithi999.front.sepia.ceph.com,smithi123.front.sepia.ceph.com"
-export MACHINE_TYPE="smithi"
-```
-
-If the lab uses a "secrets" or "inventory" repository for [ceph-cm-ansible](https://github.com/ceph/ceph-cm-ansible), you'll need to provide a URL for that. In Sepia:
-```bash
-export ANSIBLE_INVENTORY_REPO="https://github.com/ceph/ceph-sepia-secrets"
-```
-This repo will be cloned locally, using your existing `git` configuration, and copied into the teuthology container at build time.
-
-## Running Tests
-
-To run the default `teuthology:no-ceph` suite in one-shot mode:
-```bash
-./start.sh
-```
-
-To run in wait mode:
-```bash
-TEUTHOLOGY_WAIT=1 ./start.sh
-```
-
-To schedule tests in wait mode:
-```bash
-docker exec docker-compose_teuthology_1 /venv/bin/teuthology-suite ...
-```
\ No newline at end of file
diff --git a/docs/docker-compose/db/01-init.sh b/docs/docker-compose/db/01-init.sh
deleted file mode 100755
index b9e5adc2ff..0000000000
--- a/docs/docker-compose/db/01-init.sh
+++ /dev/null
@@ -1,8 +0,0 @@
-set -e
-export PGPASSWORD=$POSTGRES_PASSWORD;
-psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
-  CREATE USER $APP_DB_USER WITH PASSWORD '$APP_DB_PASS';
-  CREATE DATABASE $APP_DB_NAME;
-  GRANT ALL PRIVILEGES ON DATABASE $APP_DB_NAME TO $APP_DB_USER;
-  \connect $APP_DB_NAME $APP_DB_USER
-EOSQL
\ No newline at end of file
diff --git a/docs/docker-compose/docker-compose.yml b/docs/docker-compose/docker-compose.yml
deleted file mode 100644
index f64d17a546..0000000000
--- a/docs/docker-compose/docker-compose.yml
+++ /dev/null
@@ -1,92 +0,0 @@
-version: '3.8'
-
-services:
-  postgres:
-    image: postgres:14
-    healthcheck:
-      test: [ "CMD", "pg_isready", "-q", "-d", "paddles", "-U", "admin" ]
-      timeout: 5s
-      interval: 10s
-      retries: 2
-    environment:
-      - POSTGRES_USER=root
-      - POSTGRES_PASSWORD=password
-      - APP_DB_USER=admin
-      - APP_DB_PASS=password
-      - APP_DB_NAME=paddles
-    volumes:
-      - ./db:/docker-entrypoint-initdb.d/
-    ports:
-      - 5432:5432
-  paddles:
-    image: quay.io/ceph-infra/paddles
-    environment: 
-      PADDLES_SERVER_HOST: 0.0.0.0
-      PADDLES_SQLALCHEMY_URL: postgresql+psycopg2://admin:password@postgres:5432/paddles
-    depends_on:
-        postgres:
-            condition: service_healthy
-    links:
-        - postgres
-    healthcheck:
-        test: ["CMD", "curl", "-f", "http://0.0.0.0:8080"]
-        timeout: 5s
-        interval: 30s
-        retries: 2
-    ports: 
-        - 8080:8080
-  pulpito:
-    image: quay.io/ceph-infra/pulpito
-    environment:
-      PULPITO_PADDLES_ADDRESS: http://paddles:8080
-    depends_on:
-        paddles:
-            condition: service_healthy
-    links:
-        - paddles
-    healthcheck:
-        test: ["CMD", "curl", "-f", "http://0.0.0.0:8081"]
-        timeout: 5s
-        interval: 10s
-        retries: 2
-    ports:
-        - 8081:8081
-  beanstalk:
-        build: ../../beanstalk/alpine
-        ports:
-            - "11300:11300"
-  teuthology:
-    build:
-      context: ../../
-      dockerfile: ./docs/docker-compose/teuthology/Dockerfile
-      args:
-        SSH_PRIVKEY_FILE: $SSH_PRIVKEY_FILE
-    depends_on:
-        paddles:
-            condition: service_healthy
-    links:
-        - paddles
-        - beanstalk
-    environment:
-      SSH_PRIVKEY:
-      SSH_PRIVKEY_FILE:
-      MACHINE_TYPE:
-      TESTNODES:
-      TEUTHOLOGY_WAIT:
-      TEUTH_BRANCH:
-  testnode:
-    build:
-      context: ./testnode
-      dockerfile: ./Dockerfile
-    deploy:
-      replicas: 3
-    depends_on:
-        paddles:
-            condition: service_healthy
-    links:
-      - paddles
-    ports:
-      - "22"
-    environment:
-      SSH_PUBKEY:
-    platform: linux/amd64
diff --git a/docs/docker-compose/start.sh b/docs/docker-compose/start.sh
deleted file mode 100755
index 871258c5be..0000000000
--- a/docs/docker-compose/start.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/bash
-set -e
-export TEUTHOLOGY_BRANCH=${TEUTHOLOGY_BRANCH:-$(git branch --show-current)}
-export TEUTH_BRANCH=${TEUTHOLOGY_BRANCH}
-if [ -n "$ANSIBLE_INVENTORY_REPO" ]; then
-    basename=$(basename $ANSIBLE_INVENTORY_REPO | cut -d. -f1)
-    if [ ! -d "$basename" ]; then
-        git clone \
-            --depth 1 \
-            $ANSIBLE_INVENTORY_REPO
-    fi
-    mkdir -p teuthology/ansible_inventory
-    cp -rf $basename/ansible/ teuthology/ansible_inventory
-    if [ ! -d teuthology/ansible_inventory/hosts ]; then
-        mv -f teuthology/ansible_inventory/inventory teuthology/ansible_inventory/hosts
-    fi
-fi
-# Make the hosts and secrets directories, so that the COPY instruction in the 
-# Dockerfile does not cause a build failure when not using this feature.
-mkdir -p teuthology/ansible_inventory/hosts teuthology/ansible_inventory/secrets
-
-if [ -n "$CUSTOM_CONF" ]; then
-    cp "$CUSTOM_CONF" teuthology/
-fi
-
-# Generate an SSH keypair to use if necessary
-if [ -z "$SSH_PRIVKEY_PATH" ]; then
-    SSH_PRIVKEY_PATH=$(mktemp -u /tmp/teuthology-ssh-key-XXXXXX)
-    ssh-keygen -t rsa -N '' -f $SSH_PRIVKEY_PATH
-    export SSH_PRIVKEY=$(cat $SSH_PRIVKEY_PATH)
-    export SSH_PUBKEY=$(cat $SSH_PRIVKEY_PATH.pub)
-    export SSH_PRIVKEY_FILE=id_rsa
-else
-    export SSH_PRIVKEY=$(cat $SSH_PRIVKEY_PATH)
-    export SSH_PRIVKEY_FILE=$(basename $SSH_PRIVKEY_PATH | cut -d. -f1)
-fi
-
-if [ -z "$TEUTHOLOGY_WAIT" ]; then
-    DC_EXIT_FLAG='--abort-on-container-exit --exit-code-from teuthology'
-    DC_AUTO_DOWN_CMD='docker-compose down'
-fi
-export TEUTHOLOGY_WAIT
-
-trap "docker-compose down" SIGINT
-docker-compose up \
-    --build \
-    $DC_EXIT_FLAG
-$DC_AUTO_DOWN_CMD
diff --git a/docs/docker-compose/testnode/Dockerfile b/docs/docker-compose/testnode/Dockerfile
deleted file mode 100644
index 016d32117a..0000000000
--- a/docs/docker-compose/testnode/Dockerfile
+++ /dev/null
@@ -1,26 +0,0 @@
-FROM ubuntu:focal
-ENV DEBIAN_FRONTEND=noninteractive
-RUN apt update && \
-    apt -y install \
-        sudo \
-        openssh-server \
-        hostname \
-        curl \
-        python3-pip \
-        apache2 \
-        nfs-kernel-server && \
-    apt clean all
-COPY testnode_start.sh /
-COPY testnode_stop.sh /
-COPY testnode_sudoers /etc/sudoers.d/teuthology
-RUN \
-    ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N '' && \
-    sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/' /etc/ssh/sshd_config && \
-    mkdir -p /root/.ssh && \
-    chmod 700 /root/.ssh && \
-    useradd -g sudo ubuntu && \
-    mkdir -p /home/ubuntu/.ssh && \
-    chmod 700 /home/ubuntu/.ssh && \
-    chown -R ubuntu /home/ubuntu
-EXPOSE 22
-ENTRYPOINT /testnode_start.sh
diff --git a/docs/docker-compose/testnode/testnode_start.sh b/docs/docker-compose/testnode/testnode_start.sh
deleted file mode 100755
index d29c3b6d0a..0000000000
--- a/docs/docker-compose/testnode/testnode_start.sh
+++ /dev/null
@@ -1,13 +0,0 @@
-#!/usr/bin/bash
-set -x
-echo "$SSH_PUBKEY" > /root/.ssh/authorized_keys
-echo "$SSH_PUBKEY" > /home/ubuntu/.ssh/authorized_keys
-chown ubuntu /home/ubuntu/.ssh/authorized_keys
-payload="{\"name\": \"$(hostname)\", \"machine_type\": \"testnode\", \"up\": true, \"locked\": false, \"os_type\": \"ubuntu\", \"os_version\": \"20.04\"}"
-for i in $(seq 1 5); do
-    echo "attempt $i"
-    curl -v -f -d "$payload" http://paddles:8080/nodes/ && break
-    sleep 1
-done
-mkdir -p /run/sshd
-exec /usr/sbin/sshd -D
diff --git a/docs/docker-compose/testnode/testnode_stop.sh b/docs/docker-compose/testnode/testnode_stop.sh
deleted file mode 100755
index 2e1044d807..0000000000
--- a/docs/docker-compose/testnode/testnode_stop.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/usr/bin/bash
-set -x
-hostname=$(hostname)
-payload="{\"name\": \"$hostname\", \"machine_type\": \"testnode\", \"up\": false}"
-for i in $(seq 1 5); do
-    echo "attempt $i"
-    curl -s -f -X PUT -d "$payload" http://paddles:8080/nodes/$hostname/ && break
-    sleep 1
-done
-pkill sshd
\ No newline at end of file
diff --git a/docs/docker-compose/testnode/testnode_sudoers b/docs/docker-compose/testnode/testnode_sudoers
deleted file mode 100644
index 35828ad9be..0000000000
--- a/docs/docker-compose/testnode/testnode_sudoers
+++ /dev/null
@@ -1,4 +0,0 @@
-%sudo ALL=(ALL) NOPASSWD: ALL
-# For ansible pipelining
-Defaults !requiretty
-Defaults visiblepw
diff --git a/docs/docker-compose/teuthology/.teuthology.yaml b/docs/docker-compose/teuthology/.teuthology.yaml
deleted file mode 100644
index bac8ec1aaf..0000000000
--- a/docs/docker-compose/teuthology/.teuthology.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-queue_host: beanstalk
-queue_port: 11300
-lock_server: http://paddles:8080
-results_server: http://paddles:8080
-results_ui_server: http://pulpito:8081/
-teuthology_path: /teuthology
-archive_base: /archive_dir
-reserve_machines: 0
-lab_domain: ''
\ No newline at end of file
diff --git a/docs/docker-compose/teuthology/Dockerfile b/docs/docker-compose/teuthology/Dockerfile
deleted file mode 100644
index 5587489d17..0000000000
--- a/docs/docker-compose/teuthology/Dockerfile
+++ /dev/null
@@ -1,43 +0,0 @@
-FROM ubuntu:latest
-ARG SSH_PRIVKEY_FILE=id_ed25519
-ENV DEBIAN_FRONTEND=noninteractive
-RUN apt-get update && \
-    apt-get install -y \
-    git \
-    qemu-utils \
-    python3-dev \
-    libssl-dev \
-    ipmitool \
-    python3-pip \
-    python3-venv \
-    vim \
-    libev-dev \
-    libvirt-dev \
-    libffi-dev \
-    libyaml-dev \
-    lsb-release && \
-    apt-get clean all
-WORKDIR /teuthology
-COPY requirements.txt bootstrap /teuthology/
-RUN \
-    cd /teuthology && \
-    mkdir ../archive_dir && \
-    mkdir log && \
-    chmod +x /teuthology/bootstrap && \
-    PIP_INSTALL_FLAGS="-r requirements.txt" ./bootstrap
-COPY . /teuthology
-RUN \
-    ./bootstrap
-COPY docs/docker-compose/teuthology/containerized_node.yaml /teuthology
-COPY docs/docker-compose/teuthology/.teuthology.yaml /root
-COPY docs/docker-compose/teuthology/teuthology.sh /
-RUN mkdir -p /etc/ansible
-COPY docs/docker-compose/teuthology/ansible_inventory/hosts /etc/ansible/
-COPY docs/docker-compose/teuthology/ansible_inventory/secrets /etc/ansible/
-RUN \
-    mkdir $HOME/.ssh && \
-    touch $HOME/.ssh/${SSH_PRIVKEY_FILE} && \
-    chmod 600 $HOME/.ssh/${SSH_PRIVKEY_FILE} && \
-    echo "StrictHostKeyChecking=no" > $HOME/.ssh/config && \
-    echo "UserKnownHostsFile=/dev/null" >> $HOME/.ssh/config
-ENTRYPOINT /teuthology.sh
\ No newline at end of file
diff --git a/docs/docker-compose/teuthology/containerized_node.yaml b/docs/docker-compose/teuthology/containerized_node.yaml
deleted file mode 100644
index 02304886c4..0000000000
--- a/docs/docker-compose/teuthology/containerized_node.yaml
+++ /dev/null
@@ -1,8 +0,0 @@
-overrides:
-  ansible.cephlab:
-    skip_tags: "timezone,nagios,monitoring-scripts,ssh,hostname,pubkeys,zap,sudoers,kerberos,selinux,lvm,ntp-client,resolvconf,packages,cpan,nfs"
-    vars:
-      containerized_node: true
-      ansible_user: root
-      cm_user: root
-      start_rpcbind: false
diff --git a/docs/docker-compose/teuthology/teuthology.sh b/docs/docker-compose/teuthology/teuthology.sh
deleted file mode 100755
index 0378f93d44..0000000000
--- a/docs/docker-compose/teuthology/teuthology.sh
+++ /dev/null
@@ -1,46 +0,0 @@
-#!/usr/bin/bash
-set -e
-# We don't want -x yet, in case the private key is sensitive
-if [ -n "$SSH_PRIVKEY_FILE" ]; then
-    echo "$SSH_PRIVKEY" > $HOME/.ssh/$SSH_PRIVKEY_FILE
-fi
-source /teuthology/virtualenv/bin/activate
-set -x
-if [ -n "$TESTNODES" ]; then
-    for node in $(echo $TESTNODES | tr , ' '); do
-        teuthology-update-inventory -m $MACHINE_TYPE $node
-    done
-    CUSTOM_CONF=${CUSTOM_CONF:-}
-else
-    CUSTOM_CONF=/teuthology/containerized_node.yaml
-fi
-export MACHINE_TYPE=${MACHINE_TYPE:-testnode}
-if [ -z "$TEUTHOLOGY_WAIT" ]; then
-    if [ -n "$TEUTH_BRANCH" ]; then
-      TEUTH_BRANCH_FLAG="--teuthology-branch $TEUTH_BRANCH"
-    fi
-    teuthology-suite -v \
-        $TEUTH_BRANCH_FLAG \
-        --ceph-repo https://github.com/ceph/ceph.git \
-        --suite-repo https://github.com/ceph/ceph.git \
-        -c main \
-        -m $MACHINE_TYPE \
-        --limit 1 \
-        -n 100 \
-        --suite teuthology:no-ceph \
-        --filter-out "libcephfs,kclient,stream,centos,rhel" \
-        -d ubuntu -D 20.04 \
-        --suite-branch main \
-        --subset 9000/100000 \
-        -p 75 \
-        --seed 349 \
-        --force-priority \
-        $CUSTOM_CONF
-    DISPATCHER_EXIT_FLAG='--exit-on-empty-queue'
-    teuthology-queue -m $MACHINE_TYPE -s | \
-      python3 -c "import sys, json; assert json.loads(sys.stdin.read())['count'] > 0, 'queue is empty!'"
-fi
-teuthology-dispatcher -v \
-    --log-dir /teuthology/log \
-    --tube $MACHINE_TYPE \
-    $DISPATCHER_EXIT_FLAG