From: Laura Flores Date: Fri, 3 May 2024 17:07:40 +0000 (-0500) Subject: teuthology: modify logic to check for multiple completed builds X-Git-Tag: 1.2.0~47^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=b0359bd1f9662c5b8274b2d2a9491e2bcd4f1912;p=teuthology.git teuthology: modify logic to check for multiple completed builds The current logic assumes that there is only one build for each distro/flavor per SHA1. However, there is a bug in the jenkins infrastrucutre that sometimes causes multiple builds to trigger for one SHA1. In many of these cases, the first build succeeds, but the second fails. Teuthology only looks at the latest build, notices that it failed, and gives up. However, with this logic, teuthology can go back farther and notice that there is indeed a successful build earlier in the lineup. Here is an example in which the first centos 8 x86_64 build succeeded, but a second build on top of it failed. Teuthology could only detect the latest failed build: https://shaman.ceph.com/builds/ceph/wip-pdonnell-testing-20240503.010653-debug/ec1d3bd17a3db9d74296aa618f8d63c801bb647e/ Addresses this failure in teuthology: lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-suite -v -m smithi -c wip-pdonnell-testing-20240503.010653-debug -s fs --subset 111/12000 -p 75 --dry-run 2024-05-03 16:39:35,231.231 INFO:teuthology.suite:Using random seed=9685 2024-05-03 16:39:35,232.232 INFO:teuthology.suite.run:kernel sha1: distro 2024-05-03 16:39:35,673.673 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci.git wip-pdonnell-testing-20240503.010653-debug -> ec1d3bd17a3db9d74296aa618f8d63c801bb647e 2024-05-03 16:39:35,673.673 INFO:teuthology.suite.run:ceph sha1: ec1d3bd17a3db9d74296aa618f8d63c801bb647e 2024-05-03 16:39:35,674.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ec1d3bd17a3db9d74296aa618f8d63c801bb647e 2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default 2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/8 arm64 default 2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/9 x86_64 crimson 2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/9 x86_64 default 2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/8 arm64 default 2024-05-03 16:39:36,176.176 DEBUG:teuthology.packaging:build: centos/8 x86_64 crimson 2024-05-03 16:39:36,177.177 DEBUG:teuthology.packaging:build: centos/8 x86_64 default 2024-05-03 16:39:36,178.178 INFO:teuthology.suite.util:Container build incomplete Traceback (most recent call last): File "./teuthology/virtualenv/bin/teuthology-suite", line 8, in sys.exit(main()) File "/cephfs/home/lflores/teuthology/scripts/suite.py", line 226, in main return teuthology.suite.main(args) File "/cephfs/home/lflores/teuthology/teuthology/suite/__init__.py", line 143, in main run = Run(conf) File "/cephfs/home/lflores/teuthology/teuthology/suite/run.py", line 56, in __init__ self.base_config = self.create_initial_config() File "/cephfs/home/lflores/teuthology/teuthology/suite/run.py", line 94, in create_initial_config self.choose_ceph_version(ceph_hash) File "/cephfs/home/lflores/teuthology/teuthology/suite/run.py", line 216, in choose_ceph_version util.schedule_fail(msg, self.name, dry_run=self.args.dry_run) File "/cephfs/home/lflores/teuthology/teuthology/suite/util.py", line 77, in schedule_fail raise ScheduleFailError(message, name) teuthology.exceptions.ScheduleFailError: Scheduling lflores-2024-05-03_16:39:35-fs-wip-pdonnell-testing-20240503.010653-debug-distro-default-smithi failed: Packages for os_type 'centos', flavor default and ceph hash 'ec1d3bd17a3db9d74296aa618f8d63c801bb647e' not found More work should be done to fix the "double build" issue in jenkins, so this can be thought of as a workaround. Signed-off-by: Laura Flores --- diff --git a/teuthology/packaging.py b/teuthology/packaging.py index 9aece1271..2f6e6ba13 100644 --- a/teuthology/packaging.py +++ b/teuthology/packaging.py @@ -1003,9 +1003,10 @@ class ShamanProject(GitbuilderProject): build['distro'] == search_result['distro'] and build['distro_version'] == search_result['distro_version'] and build['flavor'] == search_result['flavor'] and - build['distro_arch'] == self.arch + build['distro_arch'] == self.arch and + build['status'] == 'completed' ): - return build['status'] == 'completed' + return True return False def _get_repo(self):