Add a new subtask: smoke()

Currently it creates a volume, then verifies that the volume exists in
Ceph - as opposed to some other place. This is intended to verify that
we've configured DevStack at least somewhat correctly.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Forget about saving the exercise.sh log, for now

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Allow using different devstack branches

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add devstack.exercise subtask

The subtask runs devstack/exercise.sh

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make reconnect() use Remote.reconnect()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move reboot() to teuthology.misc

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add documentation to reboot()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add note about memory requirements

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Simplify create_if_vm's downburst config logic

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move restart_apache() to after the reboot

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make rejoin-stack.sh start screen in detached mode

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Actually call Remote.reconnect()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Tweaks to is_online and reconnect()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add logging calls to each helper function

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Set apache ServerName, reboot, restart devstack

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add Remote.reconnect()

Also rename Remote._connect() to Remote.connect() and change its
semantics to match Remote.reconnect()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use new init parameters for Remote

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make Remote.is_online more accurate

To really know if the connection is active, we need to read something
from the remote host.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remote instances can now establish SSH connections

Previously, they required an active ssh connection object to be passed
to them. That behavior is still functional.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add Remote.is_online property

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add Remote.hostname property

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

get_file() returns a str, not a StringIO...

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use ConfigParser.set() instead of update()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Work around a bug in ConfigParser

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Call seek(0) on StringIO objects before reading

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix typoed virsh call

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't pass owner to copy_key()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use get_file() and sudo_write_file()

... instead of copy_file()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Split out key generation

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add update_devstack_config_files()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add set_libvirt_secret() and split out other steps

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Split out devstack-ceph configuration

This is starting to get long, so create configure_devstack_and_ceph()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Copy ceph auth keys over to devstack node

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add optional 'owner' arg to sudo_write_file()

This gets passed directly to chown as 'sudo chown $owner $file'

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix linter errors.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add skeleton devstack task

Currently only installs devstack on one node, and creates some pools on
the ceph cluster.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add copy_file() to scp files between remotes

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #201 from ceph/wip-6531-wusui

Fix formatting and add docstrings to ceph_manager.py

Add back in print in ceph_manager that the last checkin accidentally removed

Fix formatting and add docstrings to ceph_manager.py

Fixes: 6531
Signed-off-by: Warren Usui <warren.usui@inktank.com>

jenkins: add debugging output to shell script

When we migrated the Jenkins commands into Git, the exact commands are
no longer visible unless we run with "-x".

Add "set -x" to the script to enable debugging.

Signed-off-by: Ken Dreyer <ken.dreyer@inktank.com>

Merge pull request #195 from ceph/wip-7003-wusui

Add docstrings to the orchestra code.  Also fix minor formatting issues

Merge pull request #186 from ceph/wip-7369

Fix #7369: "sed expression must be raw string"

Fix syntax error in first docstring added.

Merge pull request #190 from kdreyer-inktank/jenkins

add jenkins-pull-requests-build script

Merge pull request #199 from ceph/wip-6540-wusui

Add docstrings.  Fix formatting issues for rgw.py

Merge pull request #196 from ceph/wip-6541-wusui

Fix docstrings in radosgw-admin-rest.py

Merge pull request #197 from ceph/wip-6535-wusui

Fix docstrings for task/blktrace.py, peering_speed_test.py, proc_thrashe...

Merge pull request #198 from ceph/wip-6539-wusui

Fix docstrings and minor formatting in task/ssh_keys.py

Merge pull request #200 from ceph/wip-6532-wusui

Fixed task/args.py docstrings.

Revert "Show hostname instead of IP in errors"

This reverts commit eafeece243607eef8209eaa8c08eecc7d1098eb1.

I was having problems with my newly-created devstack task hanging
inexplicably. It turns out this commit was responsible.

add jenkins-pull-requests-build script

Move our Jenkins pull requests build script out of Jenkins' database and
into Git so that we can have greater visibility on future changes.

Going forward, we will modify the teuthology-pull-requests Jenkins task
to only run this script.

Signed-off-by: Ken Dreyer <ken.dreyer@inktank.com>

radosbench: simplify pool creation and add ec_pool support

Signed-off-by: Samuel Just <sam.just@inktank.com>

task/: add ec_pool and append to rados.py

Signed-off-by: Samuel Just <sam.just@inktank.com>

Revert "Capture stderr and include in CommandFailedError"

This reverts commit d0a434923100dd16548e5f76eaffa5462c4b91d4.

filestore_idempotent: filestore -> objectstore path change

Signed-off-by: Sage Weil <sage@inktank.com>

thrashosds: allow primary-affinity thrashing to be disabled

Signed-off-by: Sage Weil <sage@inktank.com>

thrashosds: randomly adjust primary_affinity

Signed-off-by: Sage Weil <sage@inktank.com>

ceph.conf: mon osd allow primary affinity = true

Signed-off-by: Sage Weil <sage@inktank.com>

Fixed task/args.py docstrings.

Add docstrings.  Fix formatting issues for rgw.py
Fix s3tests calls for rgw-logsocket.py.

Fixes: 6540
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Fix docstrings and minor formatting in task/ssh_keys.py

Fixes: 6539
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Fix docstrings for task/blktrace.py, peering_speed_test.py, proc_thrasher.py and
recovery_bench.py.

Fixes: 6535
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Fix docstrings in radosgw-admin-rest.py

Fixes: 6541
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #188 from ceph/wip-calamari-onefile

Add Calamari test tasks, test script

Add docstrings to the orchestra code.  Also fix minor formatting issues
(mostly column start locations).

Fixes: 7003
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Review: switch to selecting remotes by roles in the 'normal' way

This buys us, also, the ability to select remotes by rule, i.e.
"any osd or mon gets calamari-agent installed"

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Review: refactor packaging routines to common file

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #192 from ceph/wip-6537-wusui

Readjust the indentation of mon_clock_skew_check.py and mon_thrash.py.

Merge pull request #194 from ceph/wip-6534-wusui

Add docstrings to internal.py

Merge pull request #193 from ceph/wip-6538-wusui

Add doc strings to Swift tests

Merge pull request #191 from ceph/wip-7002-wusui

Added some docstrings.

Merge pull request #187 from ceph/wip-better-debug

Debug output improvements

Add docstrings to internal.py

Fixes: 6534
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Add doc strings to Swift tests

Fixes: 6538
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Readjust the indentation of mon_clock_skew_check.py and mon_thrash.py.
Added docstrings.

Fixes: 6537
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Review: Rename calamari_testdir/ to calamari/

Added some docstrings.

Fixes: 7002
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Fix flake8 failures

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Spelling

Signed-off-by: John Spray <john.spray@inktank.com>

Log correct name of context during errors

(instead of GeneratorContextManager)

Signed-off-by: John Spray <john.spray@inktank.com>

Catch a case where we weren't getting tracebacks

Signed-off-by: John Spray <john.spray@inktank.com>

Timestamp stdout log output

Signed-off-by: John Spray <john.spray@inktank.com>

Remove spurious print

This was causing junk output like <type 'exceptions.StopIteration'>

Signed-off-by: John Spray <john.spray@inktank.com>

Capture stderr and include in CommandFailedError

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #182 from ceph/wip-imports

Make libvirt import optional (OS X)

Show hostname instead of IP in errors

Signed-off-by: John Spray <john.spray@inktank.com>

Merge pull request #189 from ceph/wip-6536-wusui

Add docstrings to s3 related tasks.

Add docstrings to s3 related tasks.

Fixes: 6536
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Add calamari setup/test tasks

Signed-off-by: Warren Usui <warren.usui@inktank.com>
Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #184 from ceph/wip-pg-scrub-from-ceph-wusui

Scrub osds when ceph task finishes

Fix #7369: "sed expression must be raw string"

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make sure osd_scrub_pgs gets called if healthy call fails.

Fixes: 7198
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Move osd_scrub_pgs call to finally block of ceph main task.

Fixes: 7198
Signed-off-bye: Warren Usui <warren.usui@inktank.com>

schedule_suite: be slightly more verbose

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #185 from ceph/wip-7342

Enable killing runs that haven't even started yet

Enable killing runs that haven't even started yet

This requires passing --machine-type.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

valgrind: fix msync suppression

Signed-off-by: Sage Weil <sage@inktank.com>

Implement the rest of the checks for automatic pg scrubbing.
osd_scrub_pgs now:
1.) Insures that all pgs are clean and active.
2.) Performs the scrub of all OSDs.
3.) Waits until all pgs have been scrubbed or no progress in scrubbing has
    been made for two minutes.

These changes have been made on top of a previous check-in that was incomplete

Fixes: 7198
Signed-off-bye: Warren Usui <warren.usui@inktank.com>

misc.py: helper roles_to_remotes for generic use

roles_to_remotes will take a config key, 'roles' by default,
and search cluster.remotes for those roles, returning a list
of remotes that hold it.  The list is uniquified by default,
but that can be overridden.  So you can have a task that
should run on specific roles, configure it with
"roles: [mon.0 server.0 osd.2]", and call this to get a list
of remotes that contain those roles.

Signed-off-by: Dan Mick <dan.mick@inktank.com>