dependabot[bot] [Mon, 8 Feb 2021 20:52:33 +0000 (20:52 +0000)]
build(deps): bump httplib2 from 0.18.0 to 0.19.0
Bumps [httplib2](https://github.com/httplib2/httplib2) from 0.18.0 to 0.19.0.
- [Release notes](https://github.com/httplib2/httplib2/releases)
- [Changelog](https://github.com/httplib2/httplib2/blob/master/CHANGELOG)
- [Commits](https://github.com/httplib2/httplib2/compare/v0.18.0...v0.19.0)
Signed-off-by: dependabot[bot] <support@github.com>
David Galloway [Fri, 5 Feb 2021 17:37:38 +0000 (12:37 -0500)]
Merge pull request #1608 from kshtsk/fix-docs
readme: fix teuthology docs link at docs.ceph.com
kyr [Fri, 5 Feb 2021 17:19:24 +0000 (18:19 +0100)]
Merge pull request #1601 from sebastian-philipp/prio-add-job-count
teuthology-suite: Add job count to priority error msg.
Kyr Shatskyy [Fri, 5 Feb 2021 17:15:49 +0000 (18:15 +0100)]
readme: fix teuthology docs link at docs.ceph.com
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
kyr [Fri, 5 Feb 2021 16:19:02 +0000 (17:19 +0100)]
Merge pull request #1607 from kshtsk/ver-1.1.0
version: increase version to 1.1.0 since we have dispatcher
kyr [Fri, 5 Feb 2021 16:11:44 +0000 (17:11 +0100)]
Merge pull request #1606 from kshtsk/supervisor-log
dispatcher: add .log extension for supervisor log
Kyr Shatskyy [Fri, 5 Feb 2021 16:09:40 +0000 (17:09 +0100)]
version: increase version to 1.1.0 since we have dispatcher
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Fri, 5 Feb 2021 16:04:52 +0000 (17:04 +0100)]
dispatcher: add .log extension for supervisor log
It would be great to have an extension for easy log identification.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Dan Mick [Thu, 4 Feb 2021 23:09:08 +0000 (15:09 -0800)]
Merge pull request #1605 from jdurgin/wip-supervisor-connect-error
dispatcher/supervisor: always unlock machines and save status
Josh Durgin [Thu, 4 Feb 2021 22:56:53 +0000 (17:56 -0500)]
dispatcher/supervisor: always unlock machines and save status
If we can't connect to the machines anymore, we still need to clean
up.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Tue, 2 Feb 2021 20:04:42 +0000 (12:04 -0800)]
Merge pull request #1604 from jdurgin/wip-dispatcher-commit-bug
dispatcher/repo_utils: handle missing commits better
Reviewed-by: David Galloway <dgallowa@redhat.com>
Josh Durgin [Tue, 2 Feb 2021 19:57:34 +0000 (14:57 -0500)]
dispatcher: keep operating if preparing a job fails
prep_job() handles updating the job status already.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Tue, 2 Feb 2021 19:48:47 +0000 (14:48 -0500)]
repo_utils: clone entire branch if commit is specified
If the commit is not the head of the branch, we need more history to be
able to check it out.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Tue, 2 Feb 2021 19:19:07 +0000 (14:19 -0500)]
worker: handle CommitNotFoundErrors
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Tue, 2 Feb 2021 16:52:22 +0000 (08:52 -0800)]
Merge pull request #1603 from jdurgin/wip-dispatcher-bug
dispatcher: allow empty os_type for fake config
Reviewed-by: David Galloway <dgallowa@redhat.com>
Josh Durgin [Tue, 2 Feb 2021 15:06:11 +0000 (10:06 -0500)]
dispatcher: allow empty os_type for fake config
This is the same default as reimaging uses,
though it's not too important in the supervisor.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Sebastian Wagner [Tue, 26 Jan 2021 12:02:35 +0000 (13:02 +0100)]
teuthology-suite: Add job count to priority error msg.
Don't let users guess the job count.
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Josh Durgin [Thu, 28 Jan 2021 01:28:30 +0000 (17:28 -0800)]
Merge pull request #1546 from ShraddhaAg/add-minimal-dispatcher
Add teuthology-dispatcher
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Wed, 27 Jan 2021 16:01:59 +0000 (08:01 -0800)]
Merge pull request #1599 from ceph/wip-exact-commits
Use the same version of teuthology and ceph-qa-suite for a whole run
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Vasu Kulkarni [Thu, 21 Jan 2021 16:01:35 +0000 (08:01 -0800)]
Merge pull request #1598 from SrinivasaBharath/wip-deb-rm
task/install/redhat: Removing packages based on OS in cleanup task
Josh Durgin [Wed, 20 Jan 2021 18:16:12 +0000 (10:16 -0800)]
Merge branch 'master' into add-minimal-dispatcher
Josh Durgin [Tue, 19 Jan 2021 19:06:45 +0000 (11:06 -0800)]
Merge pull request #1600 from rzarzynski/wip-valgrind-controllable-exit
teuthology/misc: make the Valgrind's early exit configurable.
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Radoslaw Zarzynski [Tue, 19 Jan 2021 13:56:32 +0000 (14:56 +0100)]
teuthology/misc: make the Valgrind's early exit configurable.
This commit is a follow-up to
a98eb3e1405c8ca8f6933eb0356c03955e4e2e83
where Valgrind has been configured to exit on first-seen error as it
was (wrongly!) assumed that all components are green when it comes
to the Valgrind verification.
This assumption turned out to be broken for RGW which got a few issues
over the course as a result of having the Valgrind checks knocked out
as a side effect of the python3 transition [1]. In the consequence,
multiple problems accumulated and introducing a mechanism to disable
the early exit to e.g. develop a list of these issues looks desirable.
[1]: https://github.com/ceph/teuthology/pull/1503#issuecomment-
762837504
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Josh Durgin [Sun, 17 Jan 2021 02:31:33 +0000 (21:31 -0500)]
worker, run: use exact commits for teuthology and qa suite
This ensures we use the same version across all jobs in a run.
We already have suite_sha1 set by older versions of teuthology, but
for folks who haven't updated their suite command, and thus don't set
teuthology_sha1 in the job config, look up the sha1 of in the worker.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Sun, 17 Jan 2021 02:21:02 +0000 (21:21 -0500)]
repo_utils: allow fetching a specific sha1 to per-commit directories
Using a checkout of a single branch used by potentially many
workers/teuthology processes can result in errors when one job
updates the local branch while another job is reading it.
This causes issues particularly easily when using non-master
teuthology branches, and with the teuthology-dispatcher.
This also allows us to guarantee we're using the same
version across an entire run, even if e.g. the master
qa suite is updated between jobs.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Sun, 17 Jan 2021 02:16:59 +0000 (21:16 -0500)]
suite: pass the commit of teuthology with each job's config
We already pass the suite sha1, but do not use it yet. This is the
missing piece to be able to use the same version of everything across
a whole run.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Sun, 17 Jan 2021 02:04:08 +0000 (21:04 -0500)]
repo_utils: allow checking out a specific commit
Since we're cloning a particular branch with git clone --shallow,
assume we're still passed a branch that contains the commit. Otherwise
we'd waste time and space cloning all the branches in the repo.
Assume that this is only used for checking out a particular sha1 once,
to avoid repetitive work.
There are two use cases for these utilities:
1) on the user's machine when they're scheduling a suite - there it
will make sense to maintain a single checkout of a branch e.g. teuthology master
2) on the queue consumer side - here it's best if we use the same commit for an
entire run, so checking out by sha1 makes more sense
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
srinivasabharath [Wed, 13 Jan 2021 06:30:01 +0000 (01:30 -0500)]
task/install/redhat: Removing packages based on OS in cleanup task
Signed-off-by: Bharath <skanta@redhat.com>
Josh Durgin [Thu, 14 Jan 2021 21:00:42 +0000 (13:00 -0800)]
Merge pull request #1597 from jdurgin/wip-workunit-fingerprint
exceptions: only use one of label or command for fingerprint
Reviewed-by: Neha Ojha <nojha@redhat.com>
Josh Durgin [Wed, 13 Jan 2021 03:33:38 +0000 (22:33 -0500)]
exceptions: only use one of label or command for fingerprint
Commands like those running workunits include the ceph sha1 being
tested, so they're not useful for grouping. This also lets us group
together other tests if we like, for example to map tests with small
differences in configuration to the same fingerprint for sentry.
Also use the plain command, it's already a string at this point
so there's no reason to add spaces between its characters.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Patrick Donnelly [Tue, 12 Jan 2021 15:31:46 +0000 (07:31 -0800)]
Merge PR #1595 into master
* refs/pull/1595/head:
orchestra: squelch Traceback for expected auth failures
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Patrick Donnelly [Fri, 8 Jan 2021 18:08:04 +0000 (10:08 -0800)]
orchestra: squelch Traceback for expected auth failures
The Traceback clutters the log and messes up greps for Tracebacks.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Josh Durgin [Wed, 6 Jan 2021 15:46:50 +0000 (07:46 -0800)]
Merge pull request #1593 from jdurgin/wip-sentry-sdk
sentry: use new library and group CommandFailedErrors more finely
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Josh Durgin [Mon, 4 Jan 2021 15:43:54 +0000 (10:43 -0500)]
exceptions: group CommandFailedErrors in sentry more finely
By default sentry uses the stack trace / error type / rough error
message, which ends up with many failures from different workunits
grouped together. Include the actual command run, the exit status, and
the optional label to group these more accurately. This will group
failures of the same workunit together, for example.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Josh Durgin [Mon, 4 Jan 2021 15:39:38 +0000 (10:39 -0500)]
run_tasks: use new sentry_sdk
raven has been deprecated
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
kyr [Thu, 17 Dec 2020 10:06:36 +0000 (11:06 +0100)]
Merge pull request #1591 from rakeshgm/typo-correct
run_tasks: correct typo tuethology -> teuthology
rakeshgm [Thu, 17 Dec 2020 08:31:26 +0000 (14:01 +0530)]
run_tasks: correct typo tuethology -> teuthology
Signed-off-by: rakeshgm <rakeshgm@redhat.com>
Jason Dillaman [Wed, 16 Dec 2020 21:10:24 +0000 (16:10 -0500)]
Merge pull request #1590 from lxbsz/task_ship_utilities
teuthology: run the ship_utilities task only once
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Xiubo Li [Tue, 15 Dec 2020 02:49:28 +0000 (10:49 +0800)]
teuthology: run the ship_utilities task only once
This will make sure that the utilities won't removed until the last
user is unwound.
Signed-off-by: Xiubo Li <xiubli@redhat.com>
kyr [Mon, 14 Dec 2020 20:14:48 +0000 (21:14 +0100)]
Merge pull request #1589 from kshtsk/wip-wait
Add teuthology-wait command
Kyr Shatskyy [Fri, 11 Dec 2020 17:48:16 +0000 (18:48 +0100)]
scripts: add wait script for watching run
While using teuthology-suite with --wait option it is
usefull sometimes to split the suite scheduling and
the run waiting. For example, when using tools like
Jenkins we might want to schedule a suite, report
about successful schedule and start waiting only in
the next steps.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Fri, 11 Dec 2020 18:56:01 +0000 (19:56 +0100)]
suite: improve info message about waiting the run
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Josh Durgin [Fri, 11 Dec 2020 03:31:50 +0000 (19:31 -0800)]
Merge pull request #1584 from kshtsk/wip-quiet-run
orchestra: introduce quiet mode for remote.run
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
kyr [Thu, 10 Dec 2020 00:28:35 +0000 (01:28 +0100)]
Merge pull request #1588 from lxbsz/install1
install: pass the 'shaman' to Shaman class
Xiubo Li [Wed, 9 Dec 2020 15:06:59 +0000 (23:06 +0800)]
install: pass the 'shaman' to Shaman class
Signed-off-by: Xiubo Li <xiubli@redhat.com>
kyr [Tue, 8 Dec 2020 16:59:20 +0000 (17:59 +0100)]
Merge pull request #1585 from kshtsk/wip-rocket
teuthology-suite: add Rocket.Chat notification
Kyr Shatskyy [Mon, 30 Nov 2020 20:37:05 +0000 (21:37 +0100)]
teuthology-suite: add Rocket.Chat notification
Add Rocket.Chat notification for sleep before teardown.
For details see https://rocket.chat/
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Jason Dillaman [Mon, 7 Dec 2020 16:34:49 +0000 (11:34 -0500)]
Merge pull request #1578 from lxbsz/override
packaging: try noarch repo if arch repo doesn't exist
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
kyr [Wed, 2 Dec 2020 08:55:57 +0000 (09:55 +0100)]
Merge pull request #1581 from ideepika/add-interactive-on-error
teuthology command: introduce --interactive-on-error flag
Kefu Chai [Wed, 2 Dec 2020 07:19:57 +0000 (15:19 +0800)]
Merge pull request #1586 from tchaikov/wip-gzip-archive
task.internal.archive: gzip archived file size > 128MB
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Kefu Chai [Wed, 2 Dec 2020 05:11:09 +0000 (13:11 +0800)]
task.internal.archive: gzip archived file size > 128MB
* misc: add an optional write_to argument to misc.pull_directory()
so the caller can optionally specify the function to write
to local file.
* task/internal: add a global option "log-compress-min-size" which
defaults to "128MB". if the size of a file pulled from remote
host is greater or equal to the specified size, it will be
compressed with gzip with the extension of ".gz" before
stored in the archive directory.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Kefu Chai [Wed, 2 Dec 2020 05:10:33 +0000 (13:10 +0800)]
docs/detailed_test_config: document "archive-on-error" option
Signed-off-by: Kefu Chai <kchai@redhat.com>
Xiubo Li [Thu, 5 Nov 2020 03:25:23 +0000 (22:25 -0500)]
packageing: add force_noarch option support
Add one "force_noarch : True" option to force to use the "noarch"
to build the uri, False as default.
Signed-off-by: Xiubo Li <xiubli@redhat.com>
Kyr Shatskyy [Mon, 30 Nov 2020 15:27:10 +0000 (16:27 +0100)]
orchestra: introduce quiet mode for remote.run
Applied changes:
- Add quiet option to remote.run and subsidiary function calls
- Logging commands now directed to DEBUG instead of INFO logger
This is usefull when we want suppress logs for some kind of commmands like
reading binary files or logging useless data to stdout/stderr as well as
dumping some vulnarable information.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kefu Chai [Sat, 21 Nov 2020 05:44:42 +0000 (13:44 +0800)]
Merge pull request #1575 from mdw-at-linuxbox/ssh-ecdsa
orchestra/connection: accept ecdsa (and future) host key types.
Reviewed-by: Kefu Chai <kchai@redhat.com>
Marcus Watts [Sat, 31 Oct 2020 19:31:35 +0000 (15:31 -0400)]
orchestra/connection: accept ecdsa (and future) host key types.
Out of the box, centos 8 ssh daemon makes this file,
/etc/ssh/ssh_host_ecdsa_key.pub
containing a key of type "ecdsa-sha2-nistp256", which was
not recognized by the existing teuthology logic.
Use logic in paramiko.hostkeys to recognize the new key types.
Signed-off-by: Marcus Watts <mwatts@redhat.com>
Deepika Upadhyay [Mon, 16 Nov 2020 09:07:44 +0000 (14:37 +0530)]
docs/detailed_test_config: document ``--block`` option
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
Deepika Upadhyay [Thu, 12 Nov 2020 17:08:41 +0000 (22:38 +0530)]
teuthology command: introduce --interactive-on-error flag
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
Vasu Kulkarni [Thu, 12 Nov 2020 03:18:34 +0000 (19:18 -0800)]
Merge pull request #1580 from ceph/rh_ds_yml
redhat downstream yaml location changed and redhat install tasks to accept lists
Dan Mick [Tue, 10 Nov 2020 20:34:37 +0000 (12:34 -0800)]
Merge pull request #1577 from dmick/wip-container-build-complete
Check shaman not only for repo but for build complete
rakeshgm [Tue, 27 Oct 2020 13:25:07 +0000 (18:55 +0530)]
redhat ds yaml location changed and redhat install tasks to accept lists
fetch the dowstream yaml location from
downstream teuth_config and redhat install tasks
to accept lists too
Signed-off-by: rakeshgm <rakeshgm@redhat.com>
Dan Mick [Mon, 2 Nov 2020 22:11:26 +0000 (22:11 +0000)]
Check shaman not only for repo but for build complete
The package repo might be done, but the build must be totally
complete for the container to be present. For a specific build
(in global variables, for now), demand that the build be complete
before claiming packages exist. This allows suite jobs to
fail or --newest to continue searching when the container isn't
done.
Signed-off-by: Dan Mick <dmick@redhat.com>
Kefu Chai [Mon, 9 Nov 2020 13:32:25 +0000 (21:32 +0800)]
Merge pull request #1572 from lxbsz/deb
task/install/deb: retry installing the package if resource temporarily unavailable
Reviewed-by: Kefu Chai <kchai@redhat.com>
kyr [Sun, 1 Nov 2020 11:32:41 +0000 (12:32 +0100)]
Merge pull request #1574 from ceph/dependabot/pip/cryptography-3.2
build(deps): bump cryptography from 2.8 to 3.2
kyr [Sun, 1 Nov 2020 11:32:17 +0000 (12:32 +0100)]
Merge pull request #1576 from mdw-at-linuxbox/centos-8-stream
misc: CentOSStream => rpm
Marcus Watts [Sat, 31 Oct 2020 19:30:25 +0000 (15:30 -0400)]
misc: CentOSStream => rpm
lsb_release -is identifies CentOS 8 Stream as CentOSStream.
Teuthology should recognize this as 'rpm'.
Signed-off-by: Marcus Watts <mwatts@redhat.com>
kyr [Wed, 28 Oct 2020 10:07:18 +0000 (11:07 +0100)]
Merge pull request #1559 from ryg-/wip-fix-domain
Makes (de)canonicalize_hostname ready to empty domains
dependabot[bot] [Tue, 27 Oct 2020 22:43:06 +0000 (22:43 +0000)]
build(deps): bump cryptography from 2.8 to 3.2
Bumps [cryptography](https://github.com/pyca/cryptography) from 2.8 to 3.2.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/2.8...3.2)
Signed-off-by: dependabot[bot] <support@github.com>
Roman Grigorev [Wed, 16 Sep 2020 05:34:50 +0000 (07:34 +0200)]
Makes (de)canonicalize_hostname ready to empty domains
teuthology/misc.py : addied support of empty domains to
misc.canonicalize_hostname and misc.decanonicalize_hostname.
teuthology/test/test_misc.py: added test cases for new code
Signed-off-by: Roman Grigorev <rgrigorev@suse.de>
Xiubo Li [Fri, 23 Oct 2020 04:50:25 +0000 (00:50 -0400)]
deb: retry installing the package if resource temporarily unavailable
Fixes: https://tracker.ceph.com/issues/46878
Signed-off-by: Xiubo Li <xiubli@redhat.com>
kyr [Thu, 22 Oct 2020 23:55:29 +0000 (01:55 +0200)]
Merge pull request #1560 from ryg-/wip-pelagos-speedup-parking
provision/pelagos: improve parking speed
Roman Grigorev [Fri, 29 May 2020 12:04:26 +0000 (14:04 +0200)]
provision/pelagos: Speed up pelagos node parking
Speed up the process of booting nodes to a special image that
is excluded node from access from other nodes.
Signed-off-by: Roman Grigorev <rgrigorev@suse.de>
Kefu Chai [Wed, 21 Oct 2020 10:54:41 +0000 (18:54 +0800)]
Merge pull request #1568 from ceph/wip-rpm-upgrade
task/install/rpm: fix issued with upgrading packages under DNF
Reviewed-by: Kefu Chai <kchai@redhat.com>
Patrick Donnelly [Mon, 12 Oct 2020 15:17:07 +0000 (08:17 -0700)]
Merge PR #1554 into master
* refs/pull/1554/head:
rpm: retry installing the package if the mirror server is busy
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Jason Dillaman [Tue, 6 Oct 2020 20:33:43 +0000 (16:33 -0400)]
task/install/rpm: fix issued with upgrading packages under DNF
yum is an alias for dnf on modern RHEL-like systems but it's not
100% compatible. The "install" command does not upgrade packages
unless you give it a specific version to install. Additionally,
the code was incorrectly inserting two blank arguments that resulted
in "yum install" failing.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Kefu Chai [Tue, 6 Oct 2020 08:49:18 +0000 (16:49 +0800)]
Merge pull request #1567 from tchaikov/wip-pip-use-feature
bootstrap: correct the check for --use-feature=2020-resolver
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>
Kefu Chai [Mon, 5 Oct 2020 22:46:37 +0000 (06:46 +0800)]
bootstrap: correct the check for --use-feature=2020-resolver
Signed-off-by: Kefu Chai <kchai@redhat.com>
Kefu Chai [Mon, 5 Oct 2020 15:00:35 +0000 (23:00 +0800)]
Merge pull request #1566 from toabctl/fix-pip-new-resolver
bootstrap: Do not use --use-feature=2020-resolver for pip upgrade
Reviewed-by: Kefu Chai <kchai@redhat.com>
Thomas Bechtold [Mon, 5 Oct 2020 13:28:29 +0000 (15:28 +0200)]
bootstrap: Conditionally use --use-feature=2020-resolver for pip
pip might be too old so the --use-feature flag is not available. That
leads to:
$ ./bootstrap
Removing old virtualenv because it uses system site-packages
created virtual environment CPython3.8.5.final.0-64 in 173ms
creator CPython3Posix(dest=/home/tom/devel/ceph/teuthology/virtualenv, clear=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/tom/.local/share/virtualenv)
added seed packages: pip==20.1.1, setuptools==50.3.0, wheel==0.35.1
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Usage:
pip install [options] <requirement specifier> [package-index-options] ...
pip install [options] -r <requirements file> [package-index-options] ...
pip install [options] [-e] <vcs project url> ...
pip install [options] [-e] <local project path> ...
pip install [options] <archive url/path> ...
no such option: --use-feature
So for the pip upgrade itself, check if the flag is available. From
there on, use the new resolver.
Signed-off-by: Thomas Bechtold <tbechtold@suse.com>
Kefu Chai [Thu, 24 Sep 2020 02:55:55 +0000 (10:55 +0800)]
Merge pull request #1563 from tchaikov/pip-resolver
bootstrap: use pip --use-feature=2020-resolver
Reviewed-by: David Galloway <dgallowa@redhat.com>
Kefu Chai [Wed, 23 Sep 2020 14:26:50 +0000 (22:26 +0800)]
bootstrap: use pip --use-feature=2020-resolver
follow the suggestion from pip
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Kefu Chai [Fri, 18 Sep 2020 03:00:19 +0000 (11:00 +0800)]
Merge pull request #1561 from tchaikov/wip-doc-rtd
doc: add requirements.txt
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Kefu Chai [Fri, 18 Sep 2020 01:54:33 +0000 (09:54 +0800)]
doc: add requirements.txt
so we can built this project using readthedocs.org
Signed-off-by: Kefu Chai <kchai@redhat.com>
Shraddha Agrawal [Tue, 8 Sep 2020 15:41:44 +0000 (21:11 +0530)]
dispatcher: make archive flag optional
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
kyr [Sun, 6 Sep 2020 18:08:40 +0000 (20:08 +0200)]
Merge pull request #1557 from kshtsk/wip-id_rsa-0500
Use remote.write_file instead of misc.create_file
Kyr Shatskyy [Sat, 5 Sep 2020 09:23:56 +0000 (11:23 +0200)]
task/ssh_keys: use remote.write_file instead misc.create_file
Because misc.create_file with data arguments creates file,
changes permissions, and later appends the data using
misc.append_lines_to_file, this logic makes the algorythm
above to fail if the file created without write permissions.
2020-09-05T10:00:32.003 INFO:teuthology.task.ssh_keys:pushing keys to smithi086.front.sepia.ceph.com for ubuntu
2020-09-05T10:00:32.003 INFO:teuthology.orchestra.run.smithi086:> rm -f -- /home/ubuntu/.ssh/id_rsa
2020-09-05T10:00:32.046 INFO:teuthology.orchestra.run.smithi086:> touch /home/ubuntu/.ssh/id_rsa && chmod 500 -- /home/ubuntu/.ssh/id_rsa
2020-09-05T10:00:32.095 INFO:teuthology.orchestra.run.smithi086:> set -ex
2020-09-05T10:00:32.096 INFO:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/.ssh/id_rsa conv=notrunc oflag=append
2020-09-05T10:00:32.140 INFO:teuthology.orchestra.run.smithi086.stderr:+ dd of=/home/ubuntu/.ssh/id_rsa conv=notrunc oflag=append
2020-09-05T10:00:32.142 INFO:teuthology.orchestra.run.smithi086.stderr:dd: failed to open '/home/ubuntu/.ssh/id_rsa': Permission denied
2020-09-05T10:00:32.142 DEBUG:teuthology.orchestra.run:got remote process result: 1
However we have more advanced remote.write_file function now,
which does not have such issues and moreover creates file
with the data provided in a single hop without trying to
download the file locally.
Related-to: 243ff3bbf218102b24ed992e3931bf8c76cdaadd
Related-to: 45aba9cf8d72fd30ca607200694c638cfd240fd8
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Fri, 4 Sep 2020 20:53:05 +0000 (22:53 +0200)]
task/ansible: reset fail_log file index
The yaml.safe_load reads the fail_log opened file
and shifts the offset to the end of stream.
However in case of error we need to shift offset
to the begin of the file stream, so we can read
data again.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Fri, 4 Sep 2020 16:47:03 +0000 (18:47 +0200)]
orchestra/remote: raise FileNotFoundError in read_file
Raise FileNotFoundError exception if the file passed to
remote.read_file was not found.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
kyr [Fri, 4 Sep 2020 14:29:50 +0000 (16:29 +0200)]
Merge pull request #1538 from kshtsk/wip-remote-write-file
Add remote file operations
Kyr Shatskyy [Mon, 20 Jul 2020 13:16:32 +0000 (15:16 +0200)]
orchestra.remote: add move_file, copy_file and read_file
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Thu, 16 Jul 2020 17:35:11 +0000 (19:35 +0200)]
misc: optimize append_lines_to_file() to reuse remote.write_file
The misc.append_lines_to_file method has been using get
remote file to temporary file and then write back to
remote host.
The orchestra.remote.write_file method has 'append' parameter
when using which there is no data downloaded from remote host.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Thu, 16 Jul 2020 16:17:18 +0000 (18:17 +0200)]
misc: deprecate (sudo_)write_file methods
Deprecate misc.write_file() and misc.sudo_write_file()
in favor of the orchestra.remote package methods.
The code of the misc's methods is now calling the
remote's ones.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Thu, 16 Jul 2020 16:14:57 +0000 (18:14 +0200)]
orchestra/remote: add (sudo_)write_file method
Add advanced write_file and sudo_write_file to remote context
for more intuitive and handy coding
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
kyr [Tue, 1 Sep 2020 06:50:55 +0000 (08:50 +0200)]
Merge pull request #1548 from kshtsk/wip-laptop
Setup development environment for teuthology on your laptop
Shraddha Agrawal [Tue, 1 Sep 2020 02:51:41 +0000 (08:21 +0530)]
teuthology/dispatcher: handle exception while spawning subprocess
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>
Xiubo Li [Thu, 27 Aug 2020 05:40:52 +0000 (01:40 -0400)]
rpm: retry installing the package if the mirror server is busy
When installing some packages, if the mirror server failed with
503 code, which means the mirror server temporarily not available,
we should retry it later. But the yum tool just skips it and
retries other mirrors, which may not contain them.
For the cephfs suites, there maybe will fire the many test cases
at the same time, and for each test case it may fire several nodes
to install tens of packages at the same time. This may cause the
mirror server overloaded.
We need one safe method to retry it.
Fixes: https://tracker.ceph.com/issues/47166
Signed-off-by: Xiubo Li <xiubli@redhat.com>
Kyr Shatskyy [Fri, 21 Aug 2020 11:42:31 +0000 (13:42 +0200)]
provision/downburst: add machine customization
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Fri, 14 Aug 2020 11:04:23 +0000 (13:04 +0200)]
provision/downburst: replace stock systemd-logger for opensuse-15.2
Teuthology uses rsyslog, but opensuse-15.2 has conflicting
systemd-logger preinstalled.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Fri, 14 Aug 2020 08:51:23 +0000 (10:51 +0200)]
provision/downburst: update distro conversion table
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
Kyr Shatskyy [Wed, 12 Aug 2020 17:04:46 +0000 (19:04 +0200)]
docs: teuthology ready laptop guide
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>