Merge pull request #93 from ceph/wip-9126-firefly

rgw: s/idle_timeout/default_idle_timeout/

rgw: s/idle_timeout/default_idle_timeout/

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from da00662191e9ec173be50b551253f4460f369f9da)

rgw: add default_idle_timeout to allow override

Globally overriding the rgw idle_timeout is not possible because it it
needs to be done on a per client.0, client.1, etc. basis. Add the
default_idle_timeout key to the rgw config : it defaults to the
previously hardcoded default (30) and can be changed via the override.

The existing tasks that were previously overriding the idle_timeout on a
per client basis are changed to use the default_idle_timeout instead for
consistency and to allow a global override.

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 54a7298cdd8d40aaf79615a1f580b66f83e48419)

Conflicts:
machine_types/vps.yaml
suites/smoke/verify/tasks/rgw_s3tests.yaml

ceph_manager: longer osd restart timeout

We bumped this on master and dumpling already

Signed-off-by: Sage Weil <sage@redhat.com>

Use Remote.user

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use teuthology's DaemonGroup

It has timeout support.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use newer orchestra API

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Update module references

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge branch 'teuth_firefly' into tasks_firefly

Import teuthology tasks (firefly branch)

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Move write_secret_file() into task_util/kclient.py

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add wait_until_fuse_mounted()

This was removed in a refactor in master, but it would take lots of work
to backport the entire refactor.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

ignore errors if 'ps aux' fails

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>
(cherry picked from commit f9aa9c1e647dc3376a33d0f7147501bffcddf838)

add another call to use "ps aux"

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>
(cherry picked from commit 4686115c4830939ea6c9d7a27fd89d7cec9d4d5d)

tell us if you are really not running any more

Signed-off-by: Alfredo Deza <alfredo.deza@inktank.com>
(cherry picked from commit ec342d814676d08eb833f478ac7ffdb21cc99c06)

Merge pull request #74 from ceph/wip_fix_fuse_upgrade-firefly

Yet anothe indentation fix

Yet anothe indentation fix

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #73 from ceph/wip_fix_fuse_upgrade-firefly

Fixed typos

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Fixed typos

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge pull request #72 from ceph/wip_fix_fuse_upgrade-firefly

Fixed to run blogbench.sh on client:0

Fixed to run blogbench.sh on client:0

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

valgrind: ignore ec plugin factory leaks

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7c3fc404d6a84c6601400e364bd98ea89df13922)

use notcmalloc builder for valgrind leak checking

This reverts 453d72b294af6d3db1ae7a212ba9b43209d69f26.  It turns out that
tcmalloc defeats the leak checking in valgrind.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 279f39d4c18004ed7b005b430d3862b8b2ec2f73)

Merge pull request #71 from ceph/wip_fix_fuse_upgrade-firefly

Fixed miss-config for ceph-fuse

Fixed ceph-fuse: [client.0] syntax

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Fixed miss-config for ceph-fuse

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

ceph_manager: increase osd revival timeout from 75 -> 150 seconds

Saw this fail on a firefly run just bc of valgrind slowness.  Make the
timeout higher!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f84458bd93b74873687f0099850ef5cb1e3de1b5)

Merge pull request #70 from ceph/wip_add_v0.80.5

Added new ceph version

Merge pull request #69 from ceph/wip_add_new_client1

Added client.1 in order to separate rgw tests to avoid failures on creat...

Reviewed-by: Sage Weil <sage@redhat.com>

Added new ceph version

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added client.1 in order to separate rgw tests to avoid failures on creating a directory on top of exiting directory

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

ceph_deploy: use new ship_utilities location

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 828bb2a6ea934b8788570ca3f9871a637dbd3c70)

move ship_utilities to install task (from ceph test)

No need to install ceph just to get daemon-helper, adjust-ulimits, etc..
These utilities are used by lots of tasks, not just ceph.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2dad906f4ac4658381f3984a8fe99f696f710fac)

rados/singleton-nomsgr/all/alloc-hint: fix config syntax

Put in [osd] section.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2b4f7402007e6a81dce10f3b3b25f8c410070195)

Merge branch 'firefly' of https://github.com/ceph/ceph-qa-suite into firefly

Added distros

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

rados/verify: increase api tests timeout

This has been timing out for weeks to months (hard to say since before that
we were consistently hitting an OSD crash on shutdown).  The runs I've looked
at are making progress and seem fine ... just slow because of valgrind and
because of thrashing.

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 175057f3bceb50623f6bfeded3fe0b9257edea7d)

Merge pull request #65 from ceph/wip_fix_missed_ceph_task

Fixed missing ceph task

Fixed missing ceph task

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

rados/singleton-nomsgr/all/alloc-hint: enable xfs extsize for test

This is off by default; enable it for this test.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 02a6c917eca6237118163ecd49ed7ca56bda275b)

Merge pull request #63 from ceph/wip-rgw-firefly

only test rgw on new firefly releases to avoid old bug

upgrade/firefly: only test rgw on newer firefly releases

This avoids #8311 by not testing rgw when the blank pool bug was present.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: set data extra pool correctly

When configuring multiregion zone, set the data extra pool

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 63fb271b7ecd9c1ae16d6fff3a1630df2dd09e08)

Merge pull request #58 from ceph/wip_add_v0.80.3

Added v0.80.3 to the mix

Removed whitelist lines since it's not needed for version v0.80.3

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added v0.80.3 to the mix

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Merge branch 'wip_stable_upgrade2' into firefly

upgrade/firefly: don't use overrides

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly: ignore mon scrub errors from early firefly versions

See #8815

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly: do intermediate s3tests on original test set

The later s3-tests fail on early firefly point releases.

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly: upgrade to firefly, not dumpling

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly: ignore crush tunables on upgrade

..then adjust them

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly: fix final steps

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly: upgrade mds after osd

Signed-off-by: Sage Weil <sage@redhat.com>

suites/upgrade/firefly: test rgw at the end

...and do all final tests for all runs

Signed-off-by: Sage Weil <sage@redhat.com>

upgrade/firefly: consolidate into a single collection

Signed-off-by: Sage Weil <sage@inktank.com>

Added upgrade tests - stable

Signed-off-by: Yuri Weinstein <yuri.weinstein@inktank.com>

Added requested comment.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Remove unused variables and functions.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

Support installing ceph kernels from gitbuilder on rpm machines.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>

tasks: Use '_' instead of '-' in names

Python modules are not allowed to contain hyphens
in the name.  Using hyphens only works here because
we're using low level __import__.

Change run_tasks to replace '-' with '_' when reading
configs, and rename the task modules to have valid
python module names.

Aside from general hygiene, the motivation to do this
is to allow one task to build on code in another
task by importing it.

Signed-off-by: John Spray <john.spray@inktank.com>
(cherry picked from commit 79dfe2cbefbbf14b69ae10c44a89a4fe45fedbe5)

Merge pull request #258 from ceph/wip-fix-format-samba

Fixed formatting and added a better variable for time out
Conflicts:

teuthology/task/samba.py

Merge pull request #251 from ceph/wip-8295

rgw: fix indentation for cache_pools

rgw: fix indentation for cache_pools

Fixes: #8295
Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #250 from ceph/wip-fix-thrasher

ceph_manager: reset osd weights to 1 when waiting for clean

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

ceph_manager: reset osd weights to 1 when waiting for clean

If we leave the weights adjusted, we can get PGs stuck in a remapped state
because we are probabilistically rejecting placement.  Avoid this by
setting on in osds to 1.0 before waiting for clean.

Signed-off-by: Sage Weil <sage@inktank.com>

Restrict paramiko to old versions for now

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Revert "Handle raw data I/O."

This reverts commit 257e1459fa064701d789f0ad54384bb80b45e6d9.

Revert "Clean up remote.py and misc.py changes."

This reverts commit 74eff43ee1a2b73159277370cfa9d194e42bf49c.

ceph_manager: fix float stringification

Signed-off-by: Sage Weil <sage@inktank.com>

ceph_manager: fix typo

From ce7fa1839f4b3e3db675b2d68a2bb57849f58c1e. Tested this time.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #248 from ceph/wip-thrash-osd-weights

ceph_manager: randomly reweight in osds

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

ceph_manager: randomly reweight in osds

Signed-off-by: Sage Weil <sage@inktank.com>

Bump paramiko to 1.12.0

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

rgw: option to create a cache pool

64mb for now!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>

Merge pull request #247 from ceph/requests-sessions

Use a requests.Session object for retries instead of safe_while

Add branch name to job config

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add suite name to job config

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add /build and /*.yaml to gitignore

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Calculate a timeout to use based on queue size

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add --runs, to print only run names

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Refactor teuthology.beanstalk

This architecture will make it easier to add new functionality.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Drop usage of safe_while

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Use the new ResultsReporter.session object

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Add a requests.Session object to ResultsReporter

By default it is set up to retry requests 10 times

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Remove unused timeout arg to ResultsReporter init

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Merge pull request #243 from ceph/wip-8116-wusui

Wip 8116 wusui

Clean up remote.py and misc.py changes.

Fixed method names to be non-redundant (remote_mktemp in remote is
now just mktemp, for example), and made some parameters be more
self descriptive.  Added some docstrings.  Fixed sudo setting in
get_file method.

Made chmod independent of the actual sftp file getting.
Do not do extra file copy if non-sudo read is needed.
Fixed some parameter names. Made sure temp files are removed.

Handle raw data I/O.

Paramiko 1.13.0 checks data in the Channel and fails if
invalid UTF-8 characters are sent.  The teuthology/misc.py
functions that piped cat output (get_file) and piped tar
output (pull_directory and pull_directory_tarball) formerly
did not work for Paramiko 1.13.0.

Code was changed to use SFTPClient to copy raw data.  The
remote_mktemp and function was changed to be a method of the
remote object.  Remote object methods to copy files and write
tar files were also added.

In misc.py, get_file() was changed to be a wrapper around the
remote object call.  Pull_directory() and pull_directory_tarball
were changed to use the remote object methods as well.

The remote methods used to send the data makes use of the
SFTPClient interface in Paramiko.

The remote_mktemp function was also modified slightly to insure
that new-lines do not appear in temp file names.

Fixes: 8116
Signed-off-by: Warren Usui <warren.usui@inktank.com>

When deleting all of a run's jobs, delete the run

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>

Changes so these are not installed and still removed

Changes invocation of serialize_remote_roles to internal task to avoid being run during nuke

Moves node: remote mapping to the internal task.

These will likely go somewhere better before merge

Fixes #8050 Adds a cluster.yaml that is written by interactive task

valgrind: fix tcmalloc suppression for trusty

Fixes: #8225
Signed-off-by: Sage Weil <sage@inktank.com>

Revert "valgrind.supp: be less picky about library versions"

This reverts commit f895d16c9e2fd59aab446254e53480cdb91092a1.

Merge pull request #244 from ceph/wip-7199-wusui

Wip 7199 wusui

Allow scrubbing while thrashing

Added ability to implement scrubbing while thrashing
(scrub_interval in config can be set to an interval
similar to how clean_interval is set).  Defaults to 0,
which indicates that no scrubbing will take place.
Add scrub_interval description to thrashosds docstring.

Fixes: 7199
Signed-off-by: Warren Usui <warren.usui@inktank.com>

Only attempt to use sudo if necessary