Make (de)canonicalize_hostname() work without user

Add tests proving they still work *with* usernames as well.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add unit tests for (de)canonicalize_hostname()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update/rewrite several functions to use paddles

Functions updated: lock.lock_one(), lock.unlock_one(),
lock.destroy_if_vm(), lockstatus.get_status()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

ceph.conf: mon reweight min pgs per osd = 4

So teh cephtool reweight-by-pg test will pass with mini clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

Add --suite-path

Slightly less ugly than setting PYTHONPATH

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #316 from ceph/wip-9183-wusui

Fix return from schedule_suite.

Fix return from schedule_suite.

Schedule_suite returns the number of jobs scheduled (accounting for
filters).  Information displayed also indicates how many jobs were
run and how many were filtered out.

Fixes: 9183
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #309 from ceph/wip-8954-wusui

Add filter and filter-out options to suites.

valgrind: make trusty tcmalloc supporession more general

To capture this:

  <kind>SyscallParam</kind>
  <what>Syscall param msync(start) points to unaddressable byte(s)</what>
  <stack>
    <frame>
      <ip>0x5081B3D</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.19.so</obj>
      <dir>/build/buildd/eglibc-2.19/nptl/../sysdeps/unix</dir>
      <file>syscall-template.S</file>
      <line>81</line>
    </frame>
    <frame>
      <ip>0x76BA123</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BCEF6</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BE151</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BE4E8</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BAA30</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
      <fn>_ULx86_64_step</fn>
    </frame>
    <frame>
      <ip>0x5C42052</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>GetStackTrace(void**, int, int)</fn>
    </frame>
    <frame>
      <ip>0x5C349F4</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>tcmalloc::PageHeap::GrowHeap(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x5C34D02</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>tcmalloc::PageHeap::New(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x5C25D0C</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
    </frame>
    <frame>
      <ip>0x5C46378</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>operator new[](unsigned long)</fn>
    </frame>

Signed-off-by: Sage Weil <sage@redhat.com>

Use suite_path, if set, for one-off jobs as well

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add filter and filter-off suite options.

Filter only runs suite jobs that contain the text passed as part of
their description or in the names of the composite set of yaml files.
Filter-off only runs suite jobs that do not contain the text passed
as either their description or in the names of the composite set of
yaml files.
Fixes: 8954
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #313 from ceph/wip-rhel

install: use rhel builder for rhel

Merge pull request #315 from ceph/wip-8877

Skip nuke and just destroy VPS if nuke is --unlock

Skip nuke and just destroy VPS if nuke is --unlock

If the machine is being unlocked after than we definitely want to
just destroy it. This way nukes will always suceed with --unlock
even if the guest is down or in a borked state.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

install: use rhel builder for rhel

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #312 from ceph/wip-tls-leak

valgrind: ignore TLS leak in glibc

valgrind: suppress eglibc leak of TLS

<error>
  <unique>0xf9</unique>
  <tid>1</tid>
  <kind>Leak_PossiblyLost</kind>
  <xwhat>
    <text>320 bytes in 1 blocks are possibly lost in loss record 248 of 289</text>
    <leakedbytes>320</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C29DB4</ip>
      <obj>/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>calloc</fn>
    </frame>
    <frame>
      <ip>0x4012074</ip>
      <obj>/lib/x86_64-linux-gnu/ld-2.15.so</obj>
      <fn>_dl_allocate_tls</fn>
      <dir>/build/buildd/eglibc-2.15/elf</dir>
      <file>dl-tls.c</file>
      <line>297</line>
    </frame>
    <frame>
      <ip>0x5289ABC</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.15.so</obj>
      <fn>pthread_create@@GLIBC_2.2.5</fn>
      <dir>/build/buildd/eglibc-2.15/nptl</dir>
      <file>allocatestack.c</file>
      <line>571</line>
    </frame>
    <frame>
      <ip>0xA6A89A</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Thread::try_create(unsigned long)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.83-444-ge74ad5f/src/common</dir>
      <file>Thread.cc</file>
      <line>112</line>
    </frame>
    <frame>
      <ip>0xA6A94E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Thread::create(unsigned long)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.83-444-ge74ad5f/src/common</dir>
      <file>Thread.cc</file>
      <line>122</line>
    </frame>

...

ubuntu@teuthology:/a/teuthology-2014-08-13_23:04:01-fs-next-testing-basic-multi/423897

Signed-off-by: Sage Weil <sage@redhat.com>

Perform some sanity checks on --name

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make --name required

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't fetch the suite if we can already find it

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Attach FileHandlers to the RootLogger object

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't remove lockfiles

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add retries to fetch_teuthology()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add BootstrapError

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

If bootstrap fails, remove the virtualenv

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix another variable name

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add message to exception log

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add retries to fetch_qa_suite()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use GitError instead of RuntimeError

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add GitError

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix some variable naming

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #310 from ceph/wip-valgrind

misc: pass premable through when no valgrind

misc: pass premable through when no valgrind

Signed-off-by: Sage Weil <sage@redhat.com>

Be more informative if owner info is missing

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix format string

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't redefine 'remote'

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't import from inside functions

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Attempt fetches on init, before running jobs

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Also rmtree on RuntimeError

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Disable pyc removal for now

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Bump all log levels to info

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Default to using the master branch of teuthology

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

For non-scheduled jobs, check out the suite repo.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

After resetting a repo, remove all pyc files

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make teuthology-{suite,worker} share repo code

Repo-fetching code, that is.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Reimplement the file locker as a context manager

Also give it a 'noop' flag to enable its callers to optionally not use
locking.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move exception into a new exceptions module

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move functions into repo_utils

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix NameError

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Correctly find both internal and external tasks

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove most ceph-specific tasks. They are in ceph-qa-suite now.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #306 from ceph/wip-9024

ignore errors if 'ps aux' fails

ignore errors if 'ps aux' fails

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Merge pull request #305 from ceph/wip-8640

implement `--version` and improve `setup.py`

create a --version flag for teuthology

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

pull the version and description in setup.py

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

add an initial version to teuthology

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

Tweak usage statement

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #304 from ceph/wip-fix-build-matrix

suite: fix build_matrix when non-yaml present in % dir

suite: fix build_matrix when non-yaml present in % dir

First, if we get a non-yaml file, return an empty list.

Second, if we are in a % mode, exclude an empty raw result from the
combination.  Otherwise, we get no resulting jobs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #303 from ceph/wip-8930

Tasks are failing since using "data" pool no longer part of default install

Reviewed-by: Sage Weil <sage@redhat.com>

Tasks are failing since using "data" pool no longer part of default install

Create a pool specifically for each task

Fixes: 8930
Signed-off-by: David Zafman <david.zafman@inktank.com>

Expand '~' in ssh key paths

... since paramiko doesn't bother to do this

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #301 from ceph/wip-nuke-vpm

nuke: allow nuking vpm hosts

ceph_manager: increase osd revival timeout from 75 -> 150 seconds

Saw this fail on a firefly run just bc of valgrind slowness.  Make the
timeout higher!

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind: ignore ec plugin factory leaks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #300 from ceph/wip-mpi-vers

task/mpi: Explicit check for version mismatch

Merge pull request #302 from ceph/wip-8850

tell us if you are really not running any more

add another call to use "ps aux"

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

tell us if you are really not running any more

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>

nuke: allow nuking vpm hosts

Skip the console check, though.

Signed-off-by: Sage Weil <sage@redhat.com>

task/mpi: Explicit check for version mismatch

Instead of proceeding to have mpiexec fail or hang,
do an up-front check that the MPI version is the same
on all of the nodes in the test.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix package_version_for_hash() call

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/mds_journal_migration: fix cleanup

Previously this was remounting in the background
which meant that if a workunit was executed immediately
afterwards it could put files in the mount dir before
the FS was mounted, causing the mount to fail.

Signed-off-by: John Spray <john.spray@redhat.com>

task/workunit: add sudo to dir delete

This failure to delete used to be hidden in
a catch-all exception handler that was
removed in 396dc073.

This was causing failures in kcephfs suite
like:
Command failed on plana96 with status 1: 'rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0'

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #296 from ceph/wip-standby-mds

Wip standby mds

task/osd_recovery: use rbd instead of metadata pool

Signed-off-by: Sage Weil <sage@redhat.com>

task/osd_backfill: use 'rbd' instead of 'data' pool

Signed-off-by: Sage Weil <sage@redhat.com>

task/workunit: general tidy

* PEP8-ize whitespace
* specialize a catch-all except:
* don't alias 'misc' to 'teuthology'

Signed-off-by: John Spray <john.spray@redhat.com>

task/workunit: clean up dir deletion

This was always trying to remove the mountpoint,
and then swallowing the exception and printing
it to the log.

Instead:
 * Ensure it's only trying to delete mnt if it created it
 * Don't swallow those exceptions: if deleting the dirs
   fails then something has gone wrong and it should bubble
   up.

Signed-off-by: John Spray <john.spray@redhat.com>

task/cephfs: generalise Filesystem for multi-MDS

This enables tasks like mds_journal_migration to be
run in an environment with standby-replay MDSs present.

Signed-off-by: John Spray <john.spray@redhat.com>

Remove oddly-placed, broken import

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Rename CephState to DaemonGroup and move to orchestra

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move DaemonState to teuthology.orchestra

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fail sooner if an invalid config_yaml is passed

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove kcon_most

It was unused.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #293 from ceph/wip-ship-utilities

move ship_utilities to install task (from ceph test)

ceph_deploy: use new ship_utilities location

Signed-off-by: Sage Weil <sage@redhat.com>

task/mds_journal_migration: use existing clients

Since refactoring ceph_fuse and kclient tasks
to store Mount objects on ctx, the mds_journal_migration
task can use those instead of explicitly instantiating its own.

Signed-off-by: John Spray <john.spray@redhat.com>

move ship_utilities to install task (from ceph test)

No need to install ceph just to get daemon-helper, adjust-ulimits, etc..
These utilities are used by lots of tasks, not just ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

Fix get_hash() arg ordering

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #287 from ceph/wip-mds-sessions

Wip mds sessions

Merge pull request #288 from ceph/wip-upgrade-fix

task/ceph: Make cephfs_setup cuttlefish-compatible

Merge pull request #291 from ceph/wip-rgw-pools

rgw: set data extra pool correctly

Backport: next, firefly
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: set data extra pool correctly

When configuring multiregion zone, set the data extra pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>

Add and use new fetch() method

The old fetch_branch() wasn't behaving properly with the ceph.com git
mirror. This method works with github.com and ceph.com. Add a couple
unit tests, and leave the old fetch_branch() in place for now, even
though nothing uses it.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #290 from ceph/wip-deep-scrub

task/ceph.py: deep-scrub instead of scrub

Fix a bug where results_email was being ignored

Referring to the setting in ~/.teuthology.yaml

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/ceph.py: deep-scrub instead of scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>

README: teuthology-suite

Signed-off-by: Sage Weil <sage@redhat.com>