Merge pull request #325 from ceph/wip-valgrind

valgrind: use --time-stamp=yes

valgrind: adjust suppressions for wheezy

Signed-off-by: Sage Weil <sage@redhat.com>

valgrind: use --time-stamp=yes

This doesn't help on the precise version, but maybe it will on
future versions... because if so, it would be really helpful
it matching valgrind errors to daemon logs!

In any case, though, I verified it is at worse harmless.

Signed-off-by: Sage Weil <sage@redhat.com>

Update downburst docs.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Merge pull request #324 from ceph/wip-hadoop-nuke

nuke: kill hadoop processes

nuke: kill hadoop processes

Signed-off-by: John Spray <john.spray@redhat.com>

Drop --allow-hosts None

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use setuptools, not distribute

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #323 from ceph/pypi

Prepare for PyPI

Reviewed-by: Sage Weil <sage@redhat.com>

Add install_requires and tests_require

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add MANIFEST.in

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add LICENSE

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #308 from ceph/wip-6146-wusui

Add ability to update with ceph-deploy

Merge pull request #314 from ceph/wip-8923

Speed up teuthology-kill.

Add more info in comments.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Merge pull request #319 from ceph/wip-8231

remote: Add ability to push a file to a remote machine

Merge pull request #299 from ceph/wip-7810

Wip 7810

orchestra: more docstring

Signed-off-by: John Spray <john.spray@redhat.com>

orchestra: fine tune docstrings.

Signed-off-by: John Spray <john.spray@redhat.com>

Don't explode on ConnectionError in list_locks()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

remote: Add ability to push a file to a remote machine

Add put_file() and _sftp_put_file() without sudo support

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #317 from dachary/wip-nuke-on-error

convert nuke-on-error to boolean

Merge pull request #321 from ceph/wip-install

install libcephfs-java

Only pass lists to ssh_keyscan()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

More hostname normalization

This time, for ssh_keyscan()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

For clones, look for error regardless of exit code

This fixes a bug on older git versions, where it would happily cloen a
repo and give you the wrong branch if the one you requested did not
exist.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Normalize input hostnames in various functions

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

task/install: install libcephfs-java (deb)

Accidentally removed in 0c75c6b1f7654b5a9b45d73c6b6aa47a523e1841

Signed-off-by: Sage Weil <sage@redhat.com>

task/install: move package list to top of file

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #320 from ceph/wip-sort-brief

lock.py: sort --brief output by status['name']

lock.py: sort --brief output by status['name']

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Fix a linter error

Wasn't much of a bug, but still.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix module reference

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Hostname handling for updatekeys

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Exit with code already returned by lock.main()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Less dangerous hostname mangling

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix --unlock

Note to self: clean up user/ctx.owner confusion

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Return old-style hostnames from lock_many()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't "import misc as teuthology"

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #318 from ceph/wip-defaults

change defaults for mira,plana,burnupi

More hostname handling fixes

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Revert "Canonicalize hostnames"

This reverts commit 29e8deefaeef21d1e7f6c40d6fca2605324dec69.

Remove the 'user@' prefix before checking status

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Allow passing user=None to remove the 'user@'

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix canonicalize_hostname()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove unused import

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Canonicalize hostnames in yet another place

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

suite: scheule in mira,burnupi,plana by deafult

(was plana)

Signed-off-by: Sage Weil <sage@redhat.com>

teuthology-queue: list multi queue by default

Signed-off-by: Sage Weil <sage@redhat.com>

Canonicalize hostnames

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use new vm_host attribute

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Don't dump all vm_host details when listing

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Reflect updated field names

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update provision module to use new lock server

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use Remote.shortname instead of regex

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add unlock_many() and use it when possible

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update internal task to use new lockserver

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make far less requests when simply locking

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Optimize some list_locks() calls

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Properly query for multiple machine types

...in list_locks()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use consistent URI for lock server

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove old lock server

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Change default lock server to paddles

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use /nodes/ as the lock server endpoint

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update to use new ssh-keyscan API

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Rewrite ssh public key updating

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use new ssh_keyscan() function

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

No need to be passing ctx to update_lock()

... or update_keys() or scan_for_locks()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add new ssh_keyscan() method

This is just a cleaner implementation that will replace the existing
one(s).

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Re-order functions in module

This is for readability.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

keyscan_check(): don't require ctx to be passed

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update lock_many(), update_lock(), and main()

Also remove lockstatus.send_request()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Fix PEP-8 issues

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Refactoring

Create a new teuthology.provision module and move virtual machine
functions there. Move hostname manipulation functions to
teuthology.misc.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update do_summary()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Rewrite list_locks()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Make (de)canonicalize_hostname() work without user

Add tests proving they still work *with* usernames as well.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add unit tests for (de)canonicalize_hostname()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update/rewrite several functions to use paddles

Functions updated: lock.lock_one(), lock.unlock_one(),
lock.destroy_if_vm(), lockstatus.get_status()

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

convert nuke-on-error to boolean

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph.conf: mon reweight min pgs per osd = 4

So teh cephtool reweight-by-pg test will pass with mini clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

daemon: add DaemonGroup.resolve_role_list

Used for tasks that want to take a flexible list
of roles from their user config, including a
null (everything) default and * wildcards.

Signed-off-by: John Spray <john.spray@redhat.com>

nuke: clear test firewall rules

Signed-off-by: John Spray <john.spray@redhat.com>

Add --suite-path

Slightly less ugly than setting PYTHONPATH

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #316 from ceph/wip-9183-wusui

Fix return from schedule_suite.

Fix return from schedule_suite.

Schedule_suite returns the number of jobs scheduled (accounting for
filters).  Information displayed also indicates how many jobs were
run and how many were filtered out.

Fixes: 9183
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #309 from ceph/wip-8954-wusui

Add filter and filter-out options to suites.

valgrind: make trusty tcmalloc supporession more general

To capture this:

  <kind>SyscallParam</kind>
  <what>Syscall param msync(start) points to unaddressable byte(s)</what>
  <stack>
    <frame>
      <ip>0x5081B3D</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.19.so</obj>
      <dir>/build/buildd/eglibc-2.19/nptl/../sysdeps/unix</dir>
      <file>syscall-template.S</file>
      <line>81</line>
    </frame>
    <frame>
      <ip>0x76BA123</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BCEF6</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BE151</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BE4E8</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
    </frame>
    <frame>
      <ip>0x76BAA30</ip>
      <obj>/usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1</obj>
      <fn>_ULx86_64_step</fn>
    </frame>
    <frame>
      <ip>0x5C42052</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>GetStackTrace(void**, int, int)</fn>
    </frame>
    <frame>
      <ip>0x5C349F4</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>tcmalloc::PageHeap::GrowHeap(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x5C34D02</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>tcmalloc::PageHeap::New(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x5C25D0C</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
    </frame>
    <frame>
      <ip>0x5C46378</ip>
      <obj>/usr/lib/libtcmalloc.so.4.1.2</obj>
      <fn>operator new[](unsigned long)</fn>
    </frame>

Signed-off-by: Sage Weil <sage@redhat.com>

Use suite_path, if set, for one-off jobs as well

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add filter and filter-off suite options.

Filter only runs suite jobs that contain the text passed as part of
their description or in the names of the composite set of yaml files.
Filter-off only runs suite jobs that do not contain the text passed
as either their description or in the names of the composite set of
yaml files.
Fixes: 8954
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Merge pull request #313 from ceph/wip-rhel

install: use rhel builder for rhel

Merge pull request #315 from ceph/wip-8877

Skip nuke and just destroy VPS if nuke is --unlock

Skip nuke and just destroy VPS if nuke is --unlock

If the machine is being unlocked after than we definitely want to
just destroy it. This way nukes will always suceed with --unlock
even if the guest is down or in a borked state.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Speed up teuthology-kill.

Read less information when its not needed for killing runs.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Print progress of reaidng job info when killing.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

install: use rhel builder for rhel

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #312 from ceph/wip-tls-leak

valgrind: ignore TLS leak in glibc

Add ability to upgrade with ceph-deploy

Fixes: 6146
Signed-off-by: Warren Usui <warren.usui@inktank.com>

valgrind: suppress eglibc leak of TLS

<error>
  <unique>0xf9</unique>
  <tid>1</tid>
  <kind>Leak_PossiblyLost</kind>
  <xwhat>
    <text>320 bytes in 1 blocks are possibly lost in loss record 248 of 289</text>
    <leakedbytes>320</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C29DB4</ip>
      <obj>/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>calloc</fn>
    </frame>
    <frame>
      <ip>0x4012074</ip>
      <obj>/lib/x86_64-linux-gnu/ld-2.15.so</obj>
      <fn>_dl_allocate_tls</fn>
      <dir>/build/buildd/eglibc-2.15/elf</dir>
      <file>dl-tls.c</file>
      <line>297</line>
    </frame>
    <frame>
      <ip>0x5289ABC</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.15.so</obj>
      <fn>pthread_create@@GLIBC_2.2.5</fn>
      <dir>/build/buildd/eglibc-2.15/nptl</dir>
      <file>allocatestack.c</file>
      <line>571</line>
    </frame>
    <frame>
      <ip>0xA6A89A</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Thread::try_create(unsigned long)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.83-444-ge74ad5f/src/common</dir>
      <file>Thread.cc</file>
      <line>112</line>
    </frame>
    <frame>
      <ip>0xA6A94E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Thread::create(unsigned long)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.83-444-ge74ad5f/src/common</dir>
      <file>Thread.cc</file>
      <line>122</line>
    </frame>

...

ubuntu@teuthology:/a/teuthology-2014-08-13_23:04:01-fs-next-testing-basic-multi/423897

Signed-off-by: Sage Weil <sage@redhat.com>

Perform some sanity checks on --name

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>