Merge pull request #305 from ceph/wip-8640

implement `--version` and improve `setup.py`

create a --version flag for teuthology

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

pull the version and description in setup.py

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

add an initial version to teuthology

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Tweak usage statement

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #304 from ceph/wip-fix-build-matrix

suite: fix build_matrix when non-yaml present in % dir

suite: fix build_matrix when non-yaml present in % dir

First, if we get a non-yaml file, return an empty list.

Second, if we are in a % mode, exclude an empty raw result from the
combination.  Otherwise, we get no resulting jobs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #303 from ceph/wip-8930

Tasks are failing since using "data" pool no longer part of default install

Reviewed-by: Sage Weil <sage@redhat.com>

Tasks are failing since using "data" pool no longer part of default install

Create a pool specifically for each task

Fixes: 8930
Signed-off-by: David Zafman <david.zafman@inktank.com>

Expand '~' in ssh key paths

... since paramiko doesn't bother to do this

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #301 from ceph/wip-nuke-vpm

nuke: allow nuking vpm hosts

ceph_manager: increase osd revival timeout from 75 -> 150 seconds

Saw this fail on a firefly run just bc of valgrind slowness.  Make the
timeout higher!

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind: ignore ec plugin factory leaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #300 from ceph/wip-mpi-vers

task/mpi: Explicit check for version mismatch

Merge pull request #302 from ceph/wip-8850

tell us if you are really not running any more

add another call to use "ps aux"

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

tell us if you are really not running any more

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

nuke: allow nuking vpm hosts

Skip the console check, though.

Signed-off-by: Sage Weil <sage@redhat.com>

task/mpi: Explicit check for version mismatch

Instead of proceeding to have mpiexec fail or hang,
do an up-front check that the MPI version is the same
on all of the nodes in the test.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix package_version_for_hash() call

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/mds_journal_migration: fix cleanup

Previously this was remounting in the background
which meant that if a workunit was executed immediately
afterwards it could put files in the mount dir before
the FS was mounted, causing the mount to fail.

Signed-off-by: John Spray <john.spray@redhat.com>

task/workunit: add sudo to dir delete

This failure to delete used to be hidden in
a catch-all exception handler that was
removed in 396dc073.

This was causing failures in kcephfs suite
like:
Command failed on plana96 with status 1: 'rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0'

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #296 from ceph/wip-standby-mds

Wip standby mds

task/osd_recovery: use rbd instead of metadata pool

Signed-off-by: Sage Weil <sage@redhat.com>

task/osd_backfill: use 'rbd' instead of 'data' pool

Signed-off-by: Sage Weil <sage@redhat.com>

task/workunit: general tidy

* PEP8-ize whitespace
* specialize a catch-all except:
* don't alias 'misc' to 'teuthology'

Signed-off-by: John Spray <john.spray@redhat.com>

task/workunit: clean up dir deletion

This was always trying to remove the mountpoint,
and then swallowing the exception and printing
it to the log.

Instead:
 * Ensure it's only trying to delete mnt if it created it
 * Don't swallow those exceptions: if deleting the dirs
   fails then something has gone wrong and it should bubble
   up.

Signed-off-by: John Spray <john.spray@redhat.com>

task/cephfs: generalise Filesystem for multi-MDS

This enables tasks like mds_journal_migration to be
run in an environment with standby-replay MDSs present.

Signed-off-by: John Spray <john.spray@redhat.com>

Remove oddly-placed, broken import

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Rename CephState to DaemonGroup and move to orchestra

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move DaemonState to teuthology.orchestra

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fail sooner if an invalid config_yaml is passed

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove kcon_most

It was unused.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #293 from ceph/wip-ship-utilities

move ship_utilities to install task (from ceph test)

ceph_deploy: use new ship_utilities location

Signed-off-by: Sage Weil <sage@redhat.com>

task/mds_journal_migration: use existing clients

Since refactoring ceph_fuse and kclient tasks
to store Mount objects on ctx, the mds_journal_migration
task can use those instead of explicitly instantiating its own.

Signed-off-by: John Spray <john.spray@redhat.com>

move ship_utilities to install task (from ceph test)

No need to install ceph just to get daemon-helper, adjust-ulimits, etc..
These utilities are used by lots of tasks, not just ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

Fix get_hash() arg ordering

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #287 from ceph/wip-mds-sessions

Wip mds sessions

Merge pull request #288 from ceph/wip-upgrade-fix

task/ceph: Make cephfs_setup cuttlefish-compatible

Merge pull request #291 from ceph/wip-rgw-pools

rgw: set data extra pool correctly

Backport: next, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: set data extra pool correctly

When configuring multiregion zone, set the data extra pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Add and use new fetch() method

The old fetch_branch() wasn't behaving properly with the ceph.com git
mirror. This method works with github.com and ceph.com. Add a couple
unit tests, and leave the old fetch_branch() in place for now, even
though nothing uses it.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #290 from ceph/wip-deep-scrub

task/ceph.py: deep-scrub instead of scrub

Fix a bug where results_email was being ignored

Referring to the setting in ~/.teuthology.yaml

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/ceph.py: deep-scrub instead of scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

README: teuthology-suite

Signed-off-by: Sage Weil <sage@redhat.com>

task/kclient: remove redundant double loop

Signed-off-by: John Spray <john.spray@redhat.com>

task/cephfs: use dedent for embedded python

Signed-off-by: John Spray <john.spray@redhat.com>

task: fix admin_socket for non-int ids

This was working for OSDs and for my config
where MDS happened to have numeric ID, but
in general service IDs are strings, not ints.

Signed-off-by: John Spray <john.spray@redhat.com>

task: add mds_client_recovery

This task exercises the CephFS session recovery
behavior.

Signed-off-by: John Spray <john.spray@redhat.com>

task/mds_journal_migration: only mount my client

Pass explicit client ID list to ceph_fuse context so
that it doesn't try to mount all clients in the config,
in case there are other clients.

Also allow configuration to specify which client
this task should use.

Signed-off-by: John Spray <john.spray@redhat.com>

task: refactor kclient into KernelMount

Additionally make both kclient and ceph_fuse
tasks yield a collection of CephFSMount
objects so that subsequent tasks can
retrieve them.

Signed-off-by: John Spray <john.spray@redhat.com>

task: create FuseMount class from ceph_fuse

This is for use by other tasks that need filesystem
mounts under their own control.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix the incomplete substitute_placeholders() fix

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a debug statement to reset_repo()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Be smarter about choosing branches

Part of this is also about failing sooner, and not allowing invalid
configurations to enter the queue. This commit also fixes an obscure bug
in substitute_placeholders(). Finally, it adds unit tests for all bugs
fixed.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task: move Filesystem class to shared location

...in preparation for tests other than mds_journal_migration
to use it.

Signed-off-by: John Spray <john.spray@redhat.com>

Fall back to master for suite_branch

The reason for this is that last-in-suite jobs don't have suite_branch
or branch set. We just want to use master for those.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a flag to dump the entire job body.

... with a warning.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add missing placeholder

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a --dry-run flag to teuthology-schedule

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't drop the rest of os.environ

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use os.path.expanduser() instead of os.environ

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix typo

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix lock paths

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use the ceph_branch if suite_branch isn't found

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #289 from ceph/split_prep

Split prep

Discourage modifying defaults in instances

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't use double underscores

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Run tests for base class in subclass tests

Also remove test_defaults() since it was a bad idea.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Tweak fetch_teuthology_branch()

Specifically, make it match fetch_qa_suite()'s behavior.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Clone ceph-qa-suite and add it to PYTHONPATH

So that teuthology runs can find their tasks

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use config.src_base_path

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add src_base_path to defaults.

Defaults to ~/src

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Store the suite_branch in the job config

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use the new JobConfig object

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make teuthology.config *far* more robust

In addition to parsing its configuration from a yaml file, it can now be
created from a dict. It can also be dumped as a dict, or as a yaml
stream. It is also now split into multiple classes so that the same
implementation can be used for job configs, and not just as a proxy for
~/.teuthology.yaml.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update existing unit test, and add another.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use a dict for a job template instead of a string.

This makes later modifications far easier.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add test for config substitution

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove unused import

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix install_except_hook()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't execute network-using tests by default

Set TEST_ONLINE to use them.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't use master as the default teuthology branch

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/ceph: Make cephfs_setup cuttlefish-compatible

Signed-off-by: John Spray <john.spray@redhat.com>
Fixes: #8711

watch-suite.sh: new syntax

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #276 from dachary/wip-typo

documentation typo

Merge pull request #282 from ceph/wip-jcsp-misc

Wip jcsp misc

Merge pull request #285 from ceph/wip-8750-jcsp

task/ceph: move set_max_mds into cephfs_setup

Reviewed-by: Sage Weil <sage@redhat.com>

Make the archive dir a positional argument

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #279 from ceph/wip-ls

ls: make --archive-dir default to .

Merge pull request #286 from ceph/wip-results-timeout

suite: change default results_timeout to 9h (from 6h)

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

suite: default to 9h results_timeout

6h isn't enough for xfstests.

Signed-off-by: Sage Weil <sage@inktank.com>

task/ceph: move set_max_mds into cephfs_setup

Move the calculation of active MDS count out
of the generic service setup and into the
filesystem setup.

Fixes: #8750
Signed-off-by: John Spray <john.spray@redhat.com>

ceph.conf: drop min pg per osd warning

With only the rbd pool getting created this needs to be dropped

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #281 from dachary/wip-lost-unfound

fix create_pool_with_unique_name call to use profile name

Reviewed-by: Sage Weil <sage@redhat.com>

Don't mess with ~/src/teuthology by default

Setting 'automated_scheduling: True' in ~/.teuthology.yaml enables it.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Relocate teuthology checkouts to ~/src/

Also use an underscore instead of a dash between 'teuthology' and the
branch name.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/ceph: add option to avoid doing scrub

When iterating on test in development, it's useful
to be able to skip this step.

Signed-off-by: John Spray <john.spray@redhat.com>