Added Scraper Script from github.com/jcsp/scrape with tests

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Merge pull request #1523 from kshtsk/wip-check-repos

task/install: add check for repos

task/install: add mention of 'repos' in task doc

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

task/install/rpm: verify zypper repo is valid

Add zypper ref right after addrepo so we can fail
earlier when remote repo is gone or has any issues.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1511 from ceph/dependabot/pip/httplib2-0.18.0

build(deps-dev): bump httplib2 from 0.10.3 to 0.18.0

Merge pull request #1512 from ceph/dependabot/pip/psutil-5.6.6

build(deps): bump psutil from 5.2.2 to 5.6.6

Merge pull request #1513 from ceph/dependabot/pip/ansible-2.8.8

build(deps): bump ansible from 2.8.2 to 2.8.8

Merge pull request #1524 from kshtsk/fix-worker-crash-on-try-job-delete

Fix worker crash on report.try_delete_jobs

Merge pull request #1518 from sunilkumarn417/fix-regex

Fix regex for 'ps -ef' in SystemDState::pid()

build(deps-dev): bump httplib2 from 0.10.3 to 0.18.0

Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.10.3 to 0.18.0.
- [Release notes](https://github.com/httplib2/httplib2/releases)
- [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG)
- [Commits](https://github.com/httplib2/httplib2/compare/v0.10.3...v0.18.0)

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump ansible from 2.8.2 to 2.8.8

Bumps [ansible](https://github.com/ansible/ansible) from 2.8.2 to 2.8.8.
- [Release notes](https://github.com/ansible/ansible/releases)
- [Commits](https://github.com/ansible/ansible/compare/v2.8.2...v2.8.8)

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump psutil from 5.2.2 to 5.6.6

Bumps [psutil](https://github.com/giampaolo/psutil) from 5.2.2 to 5.6.6.
- [Release notes](https://github.com/giampaolo/psutil/releases)
- [Changelog](https://github.com/giampaolo/psutil/blob/master/HISTORY.rst)
- [Commits](https://github.com/giampaolo/psutil/compare/release-5.2.2...release-5.6.6)

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #1515 from tchaikov/wip-requirements.in

drop requirements.in

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

worker: fix crash on try_delete_jobs

In some circumstances workers try to delete a job which
is not created yet. It quits immediately with unhandled
exception error.

We do not want to allow crashes whatever error happens,
because worker can continue working and process other
jobs correctly.

Also some schedulers might not add corresponding
first in suite and last in suite jobs to the reporter,
since it is not required.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

orchestra/daemon/systemd: fix proc_regex in pid() method

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

docs/INSTALL.rst: drop python2 related bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

drop requirements.in

dependabot adds an entry in generated requirements.txt like
```
-e file:///home/dependabot/dependabot-updater/tmp/dependabot_20200617-72-1n8af4b  # via -r requirements.in
```

this is expected. as dependabot does not read docs/INSTALL.rst, hence
failed to apply
```
sed '/^-e / d' to the generated file.

instead of teaching dependabot to read the manual, it's simpler
just ditch requirements.in .

see also #1367

* update-requirements.sh: add a helper script for updating
  requirements.txt
* requirements.in: removed
* docs/INSTALL.rst: updated to point user to the new
  helper script

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1521 from kshtsk/fix-install-repos

run: fix typo after refactor

run: fix typo after refactor

This addresses issue introduced by PR 1520 commit:
b9fe404403db8f983a402ed64b85a2a292de66af

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1520 from tchaikov/wip-check-per-job-verify_ceph_hash

suite/run.py: check config['verify_ceph_hash'] before verifying ceph …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/run: do not check ceph packages if verify_ceph_hash is disabled

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/run: use default value for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

suite/run.py: check config['verify_ceph_hash'] before verifying ceph packages

there is chance that we don't have ceph packages built for the
combination of specified os_type and os_version, for instance, in the
case of rados/thash_old_clients, older ceph clients are installed on
el7, but the ceph cluster is deployed using cephadm, which in turn pull
ceph container images built using the ceph being tested and el8.

since we've dropped the build of master on el7, there is no need to
verify if ceph package is available if cephadm is used for deploying the
cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1519 from tchaikov/wip-query-packages-of-el8

teuthology/suite: use rhel8 for querying package version

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology: use rhel8 for querying package version

as we don't build packages for el7 on master anymore. but we do build
packages for nautilus and octopus.

* suite: use rhel8 for querying package version
* orchestra/opsys: update default OS versions
  centos/rhel 7.6 -> 8.1

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1516 from kshtsk/wip-increase-clock-timeout

teuthology/task/clock: increase timeout

Merge pull request #1514 from sunilkumarn417/master

Fixed empty string split issue

teuthology/task/clock: increase timeout

Increase timeout because 1 minute is not always enough
to configure ntp.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Fixed empty string split issue

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1510 from tchaikov/wip-drop-py2

drop python2 support

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

rename requirements3.txt to requirements.txt

since there is no need to differentiate requirements3.txt from
requirements2.txt.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Added comment explaining the usage of the unpatched kill

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Used unpatched os.kill with test_basic in test_exit.py

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

Changed the import of mock to unittest.mock

Signed-off-by: Shubham Mishra <smishra99.iitkgp@gmail.com>

teuthology/orchestra/run: drop code handling py2

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/suite: drop six.moves.input()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/provision: drop six/string_types

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/run_tasks.py: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra: drop six.StringIO

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/provision: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

misc, repo_utils: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/suite: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/orchestra: drop six

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology: drop six.ensure_str()

use get_file(...).decode() instead

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/kill: drop six.ensure_str

Signed-off-by: Kefu Chai <kchai@redhat.com>

tasks: drop six.StringIO

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/task: drop six.string_types

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology/task: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

teuthology: drop six.reraise()

Signed-off-by: Kefu Chai <kchai@redhat.com>

setup.py: update pacakge metadata to support python3

Signed-off-by: Kefu Chai <kchai@redhat.com>

scripts/test: drop six.ensure_str()

Signed-off-by: Kefu Chai <kchai@redhat.com>

drop python2 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1509 from tchaikov/wip-secure-yaml-load

nuke: s/yaml.load/yaml.safe_load/

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

tox.ini: use python3 for testing openstack tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

nuke: s/yaml.load/yaml.safe_load/

to silence following warning:

YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1508 from lxbsz/ulimit

task/install/adjust-ulimits: use a larger numbers for open files

Reviewed-by: Kefu Chai <kchai@redhat.com>

adjust-ulimits: use a larger numbers for open files

For some cephfs test case it will run handreds of threads and with
some mounts in each, we need a larger limit on the number of open
file descriptors.

Fixes: https://tracker.ceph.com/issues/45829
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #1507 from tchaikov/wip-docs-virtualenv-py2

docs: create a python3 virtualenv instead a py2 one

Reviewed-by: Jos Collin <jcollin@redhat.com>

docs: create a python3 virtualenv instead a py2 one

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1506 from sunilkumarn417/master

eliminating kernel task duplication

Merge pull request #1505 from ceph/bootstrap-fedora-rhel

bootstrap: python3 packages for fedora and redhat based systems

python3 packages for fedora and redhat based systems

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

eliminating kernel task duplication

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #1503 from ceph/wip-valgrind-exits-on-error

teuthology/misc: Valgrind exits on first detected error.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

teuthology/misc: Valgrind exits on first detected error.

Valgrind v3.14 brought the `--exit-on-first-error` option
which causes termination of the process on first encountered
error. Although it does nothing for memory leaks, it should
help with pin-pointing e.g. `UninitValue` without imposing
the costly `--track-origins=yes`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #1502 from ceph/fix-darwin

tox: Avoid system site-packages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1490 from kshtsk/wip-auto-t

teuthology-suite: automate -t argument default value

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tox: Avoid system site-packages

Signed-off-by: Zack Cerza <zack@redhat.com>

teuthology-suite: automate -t argument default value

This change is required to determine which teuthology branch should
be used for scheduling a run for the given ceph branch.

Drop the code using --ceph-branch and when --teuthology-branch is not
supplied, since this mechanism is not used and not supported anymore.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1499 from sebastian-philipp/write-file-unicode

orchestra/run: py2 fix for copy_and_close

Merge pull request #1497 from ceph/drop-mysql-dep

Drop MySQL dependency

Drop results.generate_coverage()

This was just an oversight from #1419

Signed-off-by: Zack Cerza <zack@redhat.com>

Drop MySQL dependency

I think it's been years since anything in the project used the
mysqlclient package, so let's stop making everyone install it and everything it requires.

Signed-off-by: Zack Cerza <zack@redhat.com>

orchestra/run: py2 fix for copy_and_close

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1495 from kshtsk/plgs-logs

provision/pelagos: improve logging

provision/pelagos: improve logging

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1496 from tchaikov/wip-bootstrap-py3

bootstrap: s/python-pip/python3-pip/ for python3 on debian derivatives

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1498 from ceph/wip-waitlonger

console.py: Wait longer for IPMI timeouts

console.py: Wait longer for IPMI timeouts

The smithi have really crappy/flaky BMCs.  I occasionally see `'wait for power on' reached maximum tries (5) after waiting for 20.0 seconds` but when I go to check the BMC, it just needed some more time.  So let's give it more time.

Signed-off-by: David Galloway <dgallowa@redhat.com>

bootstrap: s/python-pip/python3-pip/ for python3 on debian derivatives

python-pip was removed in debian/bullseye and ubuntu/focal, so let's
replace it with python3-pip. and more importantly, for python3 support
we should install python3-pip instead.

Fixes: https://tracker.ceph.com/issues/45384
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1491 from ShraddhaAg/update-docs

docs: add docs.ceph.com link in README.rst

Merge pull request #1486 from kshtsk/wip-sbtd-email

run_tasks: add email notification for sleep-before-teardown

Merge pull request #1492 from toabctl/fix-opensuse-py3-bootstrap

Fix opensuse py3 bootstrap

Merge pull request #1493 from sebastian-philipp/toml

setup.py: add toml for qa/tasks/cephadm.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

setup.py: add toml for qa/tasks/cephadm.py

podman's /etc/containers/registries.conf is written
in toml. In order to modify it, we need to parse the
config.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #1482 from kshtsk/wip-desc-without-yaml

suite/build_matrix: get rid of '.yaml' in descriptions

run_tasks: add email notification for sleep-before-teardown

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

bootstap: Drop --setuptools from virtualenv installation

--setuptools is a no-op since ~2013 so drop the switch.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

bootstrap: Support python3 on openSUSE/SLES

Install the correct system packages when using a python3 env. Similar
to what is done in the ubuntu/debian case.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

docs: add docs.ceph.com link in README.rst

This commit adds a hyperlink to Running Integration Tests Using
Teuthology in docs/README.rst under the section Test Suites.

Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Merge pull request #1489 from ceph/ignore-package-check

run: exclude internal check_packages and buildpackages_prep for redhat

skip internal check_packages and buildpackages_prep for redhat config

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Merge pull request #1474 from tchaikov/wip-no-clear-exec-py3

teuthology/run_tasks: do not call sys.exc_clear() for clearing exception

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #1473 from kshtsk/wip-schedule-once

suite: schedule first and last in suite jobs once

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #1483 from kshtsk/wip-rename-describe-tests

rename teuthology-describe-tests to teuthology-describe

describe_tests: improve summary output

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

teuthology-describe-tests: rename to teuthology-describe

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

describe-tests: rename short options for better usability

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

suite/build_matrix: get rid of '.yaml' in descriptions

Test descriptions have structured representation
fragments the test config is composed from.
It's worth noticing that no other files except yaml
are used for description composition.
Although having the .yaml extentions in the description
looks redundant and has no use in the code.

This change removes many '.yaml' sufixes from
descriptions which make it more compact and easy
to read.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #1479 from ceph/rh-internal-tasks

fixes for redhat internal tasks