roles/overrides, placeholder.py: add log-ignorelist

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #1531 from kshtsk/wip-autonomous

teuthology-suite: allow dump job configs to a file instead of beanstalkd

Merge pull request #1529 from tchaikov/wip-requirements-cliff

requirements.txt: bump up cliff and other packages.

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

requirements: bump up python-openstackclient

please note, some packages not related to python-openstackclient are
also updated. the reason is that, because before this change, all
packages are pin'ed to a certain version, but after this change
only the direct dependencies are pin'ed using requirements.txt,
so, if any of the direct dependencies does not pin its own dependencies
to a certain version, pip-compile will pick the latest available version
from pypi to fulfill the requirement, and generated requirements.txt
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

requirements.in: add requirements.in back for tracking pinned direct requirements

and update `update-requirements.sh` to use requirements.in for building
requirements.txt

for some install_requires versus requirements.txt, see
https://packaging.python.org/discussions/install-requires-vs-requirements/.
in an ideal world, we should have not put those non-essential dependencies in
``setup.py``.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tox.ini: force pytest to add a formatter to logger

as `test_sh_progress` in `teuthology/test/test_misc.py` tries to access
LogRecord.asctime for verifying the behavior of `misc.sh()`

Signed-off-by: Kefu Chai <kchai@redhat.com>

update-requirements.sh: pass params to pip-compile

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1539 from batrick/i46300

ignore selinux errors from ksmtuned

Reviewed-by: Kefu Chai <kchai@redhat.com>

ignore selinux errors from ksmtuned

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1528 from kshtsk/wip-up-status

lock/cli: show up/down status when --brief

lock/cli: show up/down status when --brief

This change makes --brief option to display 'up' and 'down' statuses.

Fixes: https://tracker.ceph.com/issues/45569
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1536 from kshtsk/wip-py3-downburst

provision/downburst: py3 fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #1535 into master

* refs/pull/1535/head:
orchestra/remote: increase console timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

provision/downburst: py3 fixes

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/remote: increase console timeout

20s appears too short for some fog operations.

Fixes: https://tracker.ceph.com/issues/46503
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #1534 from ceph/wip-defaults

suite/util.py: Update Ubuntu default to Xenial

suite/util.py: Update Ubuntu default to Xenial

This will need to change to Bionic when we stop supporting luminous or must test Octopus.

Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #1532 from kshtsk/wip-ignore-git

teuthology: don't use git to define version for released teuthology

Reviewed-by: Kefu Chai <kchai@redhat.com>

teuthology: don't use git to define version for released teuthology

After installing teuthology using pip there can be found error messages
in the log when running any command, for example 'teuthology --version':

  fatal: not a git repository (or any parent up to mount point /)
  Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
  Can't get version from git rev-parse Command '['git', 'rev-parse', '--short', 'HEAD']' returned non-zero exit status 128.
  1.0.0

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/run: include 'user' and 'timestamp' in job conf

This is useful to avoid unnecessary parsing of job name in order
to get run user and timestamp.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

schedule: add more than one queue backend

This allows to have more than one beanstalk queue for jobs.
For example, use paddles or local sqlite for managing queue.

At the moment it gives a possibility to omit 'beanstalk'
queue in order to get job configurations letting a user
to run teuthology job autonomously.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

schedule: print yaml_dump instead of pprint

This will allow to redirect output of dry-run to a file
which will produce a multi-document yaml, and can be
parsed easily afterwards.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite: allow override architecture

By default for generation job configuration and filtering
them out there is 'arch' field is used determined automatically
via request to paddles database for give machine_type.

This makes it possible to override the 'arch' value with
teuthology-suite '--arch' parameter. This is only useful
at the moment just when a user don't one to make any request
to paddles.
Originally the arch is used to filter out the suites
which are not supposed to be queued on the given nodes.

In future we probably need to have tests with heterogeneous
configuration which will use multiple architectures.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1530 from shyukri/wip-py3-exception-msg

run_tasks.py: Python3 don't have exception.message anymore

run_tasks.py: Python3 don't have exception.message anymore.

This was probably working fine due to six library which was removed
recently: https://github.com/ceph/teuthology/commit/9f99b9298265be583c350c71de47109bac4bf6f1

Error seen:

```
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/worker/src/github.com_ceph_teuthology_master/teuthology/contextutil.py", line 33, in nested
    yield vars
  File "/home/worker/src/github.com_ceph_teuthology_master/teuthology/task/install/__init__.py", line 612, in task
    yield
  File "/home/worker/src/github.com_ceph_teuthology_master/teuthology/run_tasks.py", line 151, in run_tasks
    if e.message == 'too many values to unpack':
AttributeError: 'ValueError' object has no attribute 'message'
```

Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>

Merge pull request #1527 from smithfarm/wip-opensuse-kernel-distro

kernel: add kernel version matching code for opensuse

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

kernel: add kernel version matching code for opensuse

The current code for checking if the running kernel version matches the
most recent kernel version in the repos does not work on opensuse when
"-k distro" is given.

This commit adds an opensuse-specific codepath with a version match
check that works in the opensuse testing environment.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #1526 from kshtsk/wip-reimage

scripts: add reimage cli tool

scripts: add cli tools for reimaging nodes without locking

Add teuthology-reimage cli tool to be able to provision nodes using
Fog or Pelagos without locking and unlocking.

This is useful, for example, when someone just locks the node for
development or debugging purposes and do not want to release while
resetting the image, because it can happen that there are no free
nodes available.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

provision/pelagos: allow decanonicalized name in constructor

This is actually a bug fix for pelagos class, because fog's
name constructor supposed to use decanonicalized hostname.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1472 from kshtsk/wip-dont-report-status

schedule: do not report status for first and last in suite jobs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1525 from kshtsk/wip-journalctl-suffix

orchestra/daemon/cephadmunit: add journalctl suffix

Merge pull request #1432 from grapheo12/scrapeAddition

Added Results Scraping after a suite is run

Added Scraper Script from github.com/jcsp/scrape with tests

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Merge pull request #1523 from kshtsk/wip-check-repos

task/install: add check for repos

task/install: add mention of 'repos' in task doc

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install/rpm: verify zypper repo is valid

Add zypper ref right after addrepo so we can fail
earlier when remote repo is gone or has any issues.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/daemon/cephadmunit: add journalctl suffix

The logger is running in background and the output is mixed
with other loggers. Adding a recognizable suffix will
help to distinguish logging threads.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1511 from ceph/dependabot/pip/httplib2-0.18.0

build(deps-dev): bump httplib2 from 0.10.3 to 0.18.0

Merge pull request #1512 from ceph/dependabot/pip/psutil-5.6.6

build(deps): bump psutil from 5.2.2 to 5.6.6

Merge pull request #1513 from ceph/dependabot/pip/ansible-2.8.8

build(deps): bump ansible from 2.8.2 to 2.8.8

schedule: do not report status for first and last in suite jobs

Addresses the issue when teuthology run gets stuck with
first_in_suite or laste_in_suite jobs in queued state.

Attention: This change requires the next steps,
which are not mutually exclusive:

 1) server teuthology worker restart, otherwise old
    worker's code will try to remove reported job
    from paddles and exit with unexpected exception.
 2) user's teuthology runner environment should be
    updated to recent code, because new workers will
    not cleanup FIS and LIS jobs and they will remain
    in paddles, correspondingly the run will get stuck.

Requires: a34fb6a3694538db8496b3bc599c3e62b9acaca6

Fixes: http://tracker.ceph.com/issues/43291
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1524 from kshtsk/fix-worker-crash-on-try-job-delete

Fix worker crash on report.try_delete_jobs

Merge pull request #1518 from sunilkumarn417/fix-regex

Fix regex for 'ps -ef' in SystemDState::pid()

build(deps-dev): bump httplib2 from 0.10.3 to 0.18.0

Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.10.3 to 0.18.0.
- [Release notes](https://github.com/httplib2/httplib2/releases)
- [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG)
- [Commits](https://github.com/httplib2/httplib2/compare/v0.10.3...v0.18.0)

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump ansible from 2.8.2 to 2.8.8

Bumps [ansible](https://github.com/ansible/ansible) from 2.8.2 to 2.8.8.
- [Release notes](https://github.com/ansible/ansible/releases)
- [Commits](https://github.com/ansible/ansible/compare/v2.8.2...v2.8.8)

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump psutil from 5.2.2 to 5.6.6

Bumps [psutil](https://github.com/giampaolo/psutil) from 5.2.2 to 5.6.6.
- [Release notes](https://github.com/giampaolo/psutil/releases)
- [Changelog](https://github.com/giampaolo/psutil/blob/master/HISTORY.rst)
- [Commits](https://github.com/giampaolo/psutil/compare/release-5.2.2...release-5.6.6)

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1515 from tchaikov/wip-requirements.in

drop requirements.in

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

worker: fix crash on try_delete_jobs

In some circumstances workers try to delete a job which
is not created yet. It quits immediately with unhandled
exception error.

We do not want to allow crashes whatever error happens,
because worker can continue working and process other
jobs correctly.

Also some schedulers might not add corresponding
first in suite and last in suite jobs to the reporter,
since it is not required.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/daemon/systemd: fix proc_regex in pid() method

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

docs/INSTALL.rst: drop python2 related bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

drop requirements.in

dependabot adds an entry in generated requirements.txt like
```
-e file:///home/dependabot/dependabot-updater/tmp/dependabot_20200617-72-1n8af4b  # via -r requirements.in
```

this is expected. as dependabot does not read docs/INSTALL.rst, hence
failed to apply
```
sed '/^-e / d' to the generated file.

instead of teaching dependabot to read the manual, it's simpler
just ditch requirements.in .

see also #1367

* update-requirements.sh: add a helper script for updating
  requirements.txt
* requirements.in: removed
* docs/INSTALL.rst: updated to point user to the new
  helper script

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1521 from kshtsk/fix-install-repos

run: fix typo after refactor

run: fix typo after refactor

This addresses issue introduced by PR 1520 commit:
b9fe404403db8f983a402ed64b85a2a292de66af

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1520 from tchaikov/wip-check-per-job-verify_ceph_hash

suite/run.py: check config['verify_ceph_hash'] before verifying ceph …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/run: do not check ceph packages if verify_ceph_hash is disabled

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/run: use default value for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

suite/run.py: check config['verify_ceph_hash'] before verifying ceph packages

there is chance that we don't have ceph packages built for the
combination of specified os_type and os_version, for instance, in the
case of rados/thash_old_clients, older ceph clients are installed on
el7, but the ceph cluster is deployed using cephadm, which in turn pull
ceph container images built using the ceph being tested and el8.

since we've dropped the build of master on el7, there is no need to
verify if ceph package is available if cephadm is used for deploying the
cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1519 from tchaikov/wip-query-packages-of-el8

teuthology/suite: use rhel8 for querying package version

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology: use rhel8 for querying package version

as we don't build packages for el7 on master anymore. but we do build
packages for nautilus and octopus.

* suite: use rhel8 for querying package version
* orchestra/opsys: update default OS versions
  centos/rhel 7.6 -> 8.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1516 from kshtsk/wip-increase-clock-timeout

teuthology/task/clock: increase timeout

Merge pull request #1514 from sunilkumarn417/master

Fixed empty string split issue

teuthology/task/clock: increase timeout

Increase timeout because 1 minute is not always enough
to configure ntp.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fixed empty string split issue

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1510 from tchaikov/wip-drop-py2

drop python2 support

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

rename requirements3.txt to requirements.txt

since there is no need to differentiate requirements3.txt from
requirements2.txt.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Added comment explaining the usage of the unpatched kill

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Used unpatched os.kill with test_basic in test_exit.py

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Changed the import of mock to unittest.mock

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

teuthology/orchestra/run: drop code handling py2

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/suite: drop six.moves.input()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/provision: drop six/string_types

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/run_tasks.py: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra: drop six.StringIO

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/provision: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

misc, repo_utils: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/suite: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra: drop six

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology: drop six.ensure_str()

use get_file(...).decode() instead

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/kill: drop six.ensure_str

Signed-off-by: Kefu Chai <kchai@redhat.com>

tasks: drop six.StringIO

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/task: drop six.string_types

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/task: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology: drop six.reraise()

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: update pacakge metadata to support python3

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/test: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

drop python2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1509 from tchaikov/wip-secure-yaml-load

nuke: s/yaml.load/yaml.safe_load/

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox.ini: use python3 for testing openstack tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

nuke: s/yaml.load/yaml.safe_load/

to silence following warning:

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1508 from lxbsz/ulimit

task/install/adjust-ulimits: use a larger numbers for open files

Reviewed-by: Kefu Chai <kchai@redhat.com>

adjust-ulimits: use a larger numbers for open files

For some cephfs test case it will run handreds of threads and with
some mounts in each, we need a larger limit on the number of open
file descriptors.

Fixes: https://tracker.ceph.com/issues/45829
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1507 from tchaikov/wip-docs-virtualenv-py2

docs: create a python3 virtualenv instead a py2 one

Reviewed-by: Jos Collin <jcollin@redhat.com>

docs: create a python3 virtualenv instead a py2 one

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1506 from sunilkumarn417/master

eliminating kernel task duplication

Merge pull request #1505 from ceph/bootstrap-fedora-rhel

bootstrap: python3 packages for fedora and redhat based systems

python3 packages for fedora and redhat based systems

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

eliminating kernel task duplication

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1503 from ceph/wip-valgrind-exits-on-error

teuthology/misc: Valgrind exits on first detected error.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/misc: Valgrind exits on first detected error.

Valgrind v3.14 brought the `--exit-on-first-error` option
which causes termination of the process on first encountered
error. Although it does nothing for memory leaks, it should
help with pin-pointing e.g. `UninitValue` without imposing
the costly `--track-origins=yes`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>