Remove oddly-placed, broken import

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Rename CephState to DaemonGroup and move to orchestra

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move DaemonState to teuthology.orchestra

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fail sooner if an invalid config_yaml is passed

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove kcon_most

It was unused.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #293 from ceph/wip-ship-utilities

move ship_utilities to install task (from ceph test)

ceph_deploy: use new ship_utilities location

Signed-off-by: Sage Weil <sage@redhat.com>

move ship_utilities to install task (from ceph test)

No need to install ceph just to get daemon-helper, adjust-ulimits, etc..
These utilities are used by lots of tasks, not just ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

Fix get_hash() arg ordering

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #287 from ceph/wip-mds-sessions

Wip mds sessions

Merge pull request #288 from ceph/wip-upgrade-fix

task/ceph: Make cephfs_setup cuttlefish-compatible

Merge pull request #291 from ceph/wip-rgw-pools

rgw: set data extra pool correctly

Backport: next, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: set data extra pool correctly

When configuring multiregion zone, set the data extra pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Add and use new fetch() method

The old fetch_branch() wasn't behaving properly with the ceph.com git
mirror. This method works with github.com and ceph.com. Add a couple
unit tests, and leave the old fetch_branch() in place for now, even
though nothing uses it.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #290 from ceph/wip-deep-scrub

task/ceph.py: deep-scrub instead of scrub

Fix a bug where results_email was being ignored

Referring to the setting in ~/.teuthology.yaml

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/ceph.py: deep-scrub instead of scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

README: teuthology-suite

Signed-off-by: Sage Weil <sage@redhat.com>

task/kclient: remove redundant double loop

Signed-off-by: John Spray <john.spray@redhat.com>

task/cephfs: use dedent for embedded python

Signed-off-by: John Spray <john.spray@redhat.com>

task: fix admin_socket for non-int ids

This was working for OSDs and for my config
where MDS happened to have numeric ID, but
in general service IDs are strings, not ints.

Signed-off-by: John Spray <john.spray@redhat.com>

task: add mds_client_recovery

This task exercises the CephFS session recovery
behavior.

Signed-off-by: John Spray <john.spray@redhat.com>

task/mds_journal_migration: only mount my client

Pass explicit client ID list to ceph_fuse context so
that it doesn't try to mount all clients in the config,
in case there are other clients.

Also allow configuration to specify which client
this task should use.

Signed-off-by: John Spray <john.spray@redhat.com>

task: refactor kclient into KernelMount

Additionally make both kclient and ceph_fuse
tasks yield a collection of CephFSMount
objects so that subsequent tasks can
retrieve them.

Signed-off-by: John Spray <john.spray@redhat.com>

task: create FuseMount class from ceph_fuse

This is for use by other tasks that need filesystem
mounts under their own control.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix the incomplete substitute_placeholders() fix

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a debug statement to reset_repo()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Be smarter about choosing branches

Part of this is also about failing sooner, and not allowing invalid
configurations to enter the queue. This commit also fixes an obscure bug
in substitute_placeholders(). Finally, it adds unit tests for all bugs
fixed.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task: move Filesystem class to shared location

...in preparation for tests other than mds_journal_migration
to use it.

Signed-off-by: John Spray <john.spray@redhat.com>

Fall back to master for suite_branch

The reason for this is that last-in-suite jobs don't have suite_branch
or branch set. We just want to use master for those.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a flag to dump the entire job body.

... with a warning.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add missing placeholder

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a --dry-run flag to teuthology-schedule

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't drop the rest of os.environ

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use os.path.expanduser() instead of os.environ

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix typo

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix lock paths

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use the ceph_branch if suite_branch isn't found

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #289 from ceph/split_prep

Split prep

Discourage modifying defaults in instances

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't use double underscores

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Run tests for base class in subclass tests

Also remove test_defaults() since it was a bad idea.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Tweak fetch_teuthology_branch()

Specifically, make it match fetch_qa_suite()'s behavior.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Clone ceph-qa-suite and add it to PYTHONPATH

So that teuthology runs can find their tasks

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use config.src_base_path

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add src_base_path to defaults.

Defaults to ~/src

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Store the suite_branch in the job config

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use the new JobConfig object

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make teuthology.config *far* more robust

In addition to parsing its configuration from a yaml file, it can now be
created from a dict. It can also be dumped as a dict, or as a yaml
stream. It is also now split into multiple classes so that the same
implementation can be used for job configs, and not just as a proxy for
~/.teuthology.yaml.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update existing unit test, and add another.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use a dict for a job template instead of a string.

This makes later modifications far easier.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add test for config substitution

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove unused import

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix install_except_hook()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't execute network-using tests by default

Set TEST_ONLINE to use them.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't use master as the default teuthology branch

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/ceph: Make cephfs_setup cuttlefish-compatible

Signed-off-by: John Spray <john.spray@redhat.com>
Fixes: #8711

watch-suite.sh: new syntax

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #276 from dachary/wip-typo

documentation typo

Merge pull request #282 from ceph/wip-jcsp-misc

Wip jcsp misc

Merge pull request #285 from ceph/wip-8750-jcsp

task/ceph: move set_max_mds into cephfs_setup

Reviewed-by: Sage Weil <sage@redhat.com>

Make the archive dir a positional argument

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #279 from ceph/wip-ls

ls: make --archive-dir default to .

Merge pull request #286 from ceph/wip-results-timeout

suite: change default results_timeout to 9h (from 6h)

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

suite: default to 9h results_timeout

6h isn't enough for xfstests.

Signed-off-by: Sage Weil <sage@inktank.com>

task/ceph: move set_max_mds into cephfs_setup

Move the calculation of active MDS count out
of the generic service setup and into the
filesystem setup.

Fixes: #8750
Signed-off-by: John Spray <john.spray@redhat.com>

ceph.conf: drop min pg per osd warning

With only the rbd pool getting created this needs to be dropped

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #281 from dachary/wip-lost-unfound

fix create_pool_with_unique_name call to use profile name

Reviewed-by: Sage Weil <sage@redhat.com>

Don't mess with ~/src/teuthology by default

Setting 'automated_scheduling: True' in ~/.teuthology.yaml enables it.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Relocate teuthology checkouts to ~/src/

Also use an underscore instead of a dash between 'teuthology' and the
branch name.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/ceph: add option to avoid doing scrub

When iterating on test in development, it's useful
to be able to skip this step.

Signed-off-by: John Spray <john.spray@redhat.com>

nuke: fix unmounted multiple fuse mount points

It was trying to pass multiple paths to fusermount, but
it should instead invoke fusermount for each path.

Signed-off-by: John Spray <john.spray@redhat.com>

run_tasks: catch malformed task case

If you do something like this:

- foo:
- bar:
- baz

Then catching AttributeError is necessary for giving a somewhat
friendlier error on the 'baz' where we have a string instead
of a dict.

Signed-off-by: John Spray <john.spray@redhat.com>

task/ceph_manager: generalize admin_socket calls

This is so that tasks can access the admin sockets
of other service types such as the MDS.

Signed-off-by: John Spray <john.spray@redhat.com>

Remove default results_server

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Allow setting results_email in teuthology.yaml

(Instead of passing --email to teuthology-suite)

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't remove the teuthology repo if errors occur

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Optionally, don't remove a repo when errors occur

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add another unit test

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

fix create_pool_with_unique_name call to use profile name

The hardcoded arguments are deprecated for the profile name. Create a
profile that matches the previous default arguments. Read the config to
allow the caller to override the defaults from the yaml file.

Signed-off-by: Loic Dachary <loic@dachary.org>

document create_pool erasure_code_profile_name argument

Signed-off-by: Loic Dachary <loic@dachary.org>

Allow testing with online repos

It is not the default.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a unit test simulating simultaneous checkouts

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a couple unit tests for teuthology.parallel

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #280 from dachary/wip-erasure-code-profile

add rgw.create_ec_pool default profile

add rgw.create_ec_pool default profile

the former arguments had defaults, the caller do not set them

Signed-off-by: Loic Dachary <loic@dachary.org>

ls: make --archive-dir default to .

This is how I use it 99% of the time.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #275 from ceph/wip-replace-update-sh

Replace update.sh functionality

Rename suite_base to suite_dir

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add --suite-branch and --suite-base

The former lets you specify a specific ceph-qa-suite branch to use for
testing. The latter lets you specify an as-is directory to use, for
example if you want ro run tests that you don't want to commit yet.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use a test-specific name/email for the git calls

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add more unit tests

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #277 from dachary/wip-erasure-code-profile

erasure code profile: fix bugs found while testing manually

Merge pull request #278 from ceph/wip-nullfs

Support the new fs syntax.

Reviewed-by: Greg Farnum <greg@inktank.com>

erasure-code: fix typo in create_pool argument name

missing _name at the end

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: do not prepend 'ceph' to the erasure-code-profile command

When used in the radosgw context, the 'ceph' string must be present but
when used in ceph_manager, the 'ceph' string is already included.

Signed-off-by: Loic Dachary <loic@dachary.org>

erasure-code: str() to avoid + conversion errors

if the value from the yaml file is an int, it must be converted into a
string bevore being concatenated with '='

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #271 from dachary/wip-erasure-code-profile

erasure-code: separate profile from pool creation

Fix linter errors

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a basic validation of the branch value

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>