as stdout redirect is performed by the shell, and shell is launched by
the "jenkins" user, so permission denied is expected when we are trying
to create a file under /etc/yum.repos.d/. in that case, we'll have
following failure:
/tmp/jenkins8899238112048417478.sh: line 14:
/etc/yum.repos.d/google-chrome.repo: Permission denied
Build step 'Execute shell' marked build as failure
scripts/build_utils.sh: add lang=none option to sources.list
some mirrors do not contains the package description translations. when
apt tries to download the packages descriptions, it fails like:
E: Failed to fetch
http://mirror.yandex.ru/mirrors/launchpad/ubuntu-toolchain-r/dists/xenial/main/i18n/Translation-en
404 Not Found [IP: 213.180.204.183 80]
E: Some index files failed to download. They have been ignored, or old
ones used instead.
for more info, see sources.list(5) and apt.conf(5). the default value
of "lang" is set by Acquire::Languages, which is in turn "en" and
"environment" by default. we need to override them with "none".
scripts/build_utils.sh: increase mem per job to 2.2GB
it's observed that on our arm64 build host, using the existing
$total_mem/1600, n_build_jobs=37 . but we still have following error
when compiling Ceph:
Sage Weil [Wed, 19 Sep 2018 22:24:09 +0000 (17:24 -0500)]
ceph-dev-[new-]setup/build/build: -DWITH_CEPH_DEBUG_MUTEX=ON for notcmalloc
Enable the debug ceph::mutex in the notcmalloc build. The normal build
will use the defaults, which (if it's a Release or RelWithDebug build)
will not include extra debug checks.
Relevant changes:
- All non-compliant commits are reported (previously, only the first
wrong one was reported, stopping further processing).
- Signed-off-by regex put in place to check for
<something@somewhere.tld> (including lt and gt symbols). This rule
could be relaxed, as some commits in the past were sent without
the enclosing <...>
- `extract_sha` function has been removed
Relevant changes:
- All non-compliant commits are reported (previously, only the first
wrong one was reported, stopping further processing).
- Signed-off-by regex put in place to check for
<something@somewhere.tld> (including lt and gt symbols). This rule
could be relaxed, as some commits in the past were sent without
the enclosing <...>
- `extract_sha` function has been removed
Relevant changes:
- All non-compliant commits are reported (previously, only the first
wrong one was reported, stopping further processing).
- Signed-off-by regex put in place to check for
<something@somewhere.tld> (including lt and gt symbols). This rule
could be relaxed, as some commits in the past were sent without
the enclosing <...>
- `extract_sha` function has been removed
Change build discard options for ceph and ceph-volume jobs
So that it isn't limited by the number, it is limited by days only. This
will allow builds to avoid 404'ing when the limit is reached, which is
unexpected if there are too many builds and not long enough time between
them (like 100 builds in one day, but limit is 20 before discarding)
ceph-*-build: evaluate num jobs using total mem not free mem
there is chance that the system is under extremely high load when `vmstat`
is called. so the "free memory" can not reflect the available memroy
when building Ceph. so use "total memory" instead.
David Galloway [Tue, 4 Sep 2018 17:43:41 +0000 (13:43 -0400)]
ansible: Install python-pip from epel on centos slaves
I'm not sure why we're just now seeing this. Perhaps it was manually
fixed on prado.ceph.com and a redeploy overwrote it? Regardless
non-libvirt slaves are failing to join Jenkins currently.
Signed-off-by: David Galloway <dgallowa@redhat.com>
Sébastien Han [Tue, 28 Aug 2018 17:23:55 +0000 (10:23 -0700)]
ceph-ansible-pr-syntax-check: group_vars operator
When defaults/main.yml is touched we will generate 2 group_vars files,
one for all.yml.sample and one for rhcs.yml.sample, this is making the
conditional never pass since there is one more value in the count.
Basically if defaults/main.yml and osd/defaults/main.yml are touched
this will result in 3 group_vars file but the count for
defaults/main.yml is 2.
In the end we want to fail if the number of defaults/main.yml is
greather than the number of group_vars files.
Kefu Chai [Wed, 22 Aug 2018 07:27:32 +0000 (15:27 +0800)]
ceph-*-build: set jobs number according to free memory
limits the job number of building ceph/ceph pull requests and deb packages
with the (size of free memory in MB)/1800.
guess we are using more compile-time optimizations now, so to compile
ceph source requires more memory. sometimes, a single cc1plus takes
more than 3GB memeory. that's why we are seeing more and more OOM
in our arm64 builders. following is a sample from omani09 -- a
arm64 builder compiling a ceph/ceph PR targeting master:
also, the performance of over-all compiling is also impacted by the I/O
subsystem. so lower the number of job could actually reduce the time
of the compiling processes to completing for the I/O queue of local
device. so we can use an conservative number for calc an upper bound of
job number for "make" instead using $(nproc). in this change,
$(free_memory_in_mega / 1800) is used as the upper limit of n_jobs.
on a typical arm64 builder with 48 cores and 64 GB mem, the n_jobs is
now 34 .
when building rpm packages, the number of build jobs is specified by
_smp_mflags macro, which is defined by
/usr/lib/rpm/platform/*/macros and /usr/lib/rpm/redhat/macros.
see
https://github.com/rpm-software-management/rpm/blob/master/platform.in#L53
and rhel/centos use following patch
https://git.centos.org/blob/rpms!redhat-rpm-config.git/eaaa6282147d0797a3733f3b91671b7a0752d448/SOURCES!redhat-rpm-config-9.1.0-ncpus-max.patch;jsessionid=xv8lqw4ipwwetge0i19ejo9t
so one cannot build rpm packages on centos/rhel with more than 16 jobs
when using redhat-rpm-config. and 16 is a safe number for us.
Andrew Schoen [Fri, 24 Aug 2018 12:54:46 +0000 (08:54 -0400)]
ceph-volume-ansible-prs: disable centos7 tests
We're having infrastructure issues with OVH and the centos7 vagrant
vm kernel panics after booting. Disabling these for now until the issue
can be resolved.
Andrew Schoen [Wed, 22 Aug 2018 19:38:47 +0000 (15:38 -0400)]
ceph-volume-ansible-prs: deploy the target branch, not the source branch
We're going to start deploying the target branch, the branch we're
merging PRs into, instead of the source branch. This way we don't have to
wait on packages as we can assume there is somethign already available
for the target branch. The ceph-volume code we're testing will be
rsynced to the testing nodes from the source branch.
Sébastien Han [Wed, 22 Aug 2018 09:30:53 +0000 (11:30 +0200)]
ceph-ansible-pr-syntax-chec: add test_ceph_release_in_ceph_default
function
This new function searches for statements like ' -
ceph_release_num[ceph_release] ' in the ceph-defaults role. These must
never be used since the ceph_release is set by role running after
ceph-defaults.
However, the ceph-ansible ci passes this variable when running so it's
always defined, even before ceph-defaults runs. This is particularity
from our CI. So this is done, our Ci won't complain but general users
will end with an error like "'dict object' has no attribute" since the
variable does not exist at the time of the ceph-defaults play.
Sébastien Han [Wed, 22 Aug 2018 09:25:52 +0000 (11:25 +0200)]
ceph-ansible-pr-syntax-check: add two functions to search files
Adding 2 new functions:
* git_diff_to_head which prints the diff for the current new code
* match_file which searches and print a particular file that was
modified by the current code.
Sébastien Han [Mon, 20 Aug 2018 08:39:21 +0000 (10:39 +0200)]
ceph-ansible-pr-syntax-check: use latest ansible version
There is no reason to pin to an old ansible version. Since this is doing
syntax checking we should have the latest Ansible version to compare
against our latest ceph-ansible code.
Sébastien Han [Mon, 13 Aug 2018 12:34:52 +0000 (14:34 +0200)]
ceph-ansible-pr-syntax-check: add cbheck for coding convention
One of our convention in ceph-ansible is to not use capital letters in
task's name. So let's hardcode it in the initial phase of the pipeline.
ALso, this will let user discover what's wrong without us doing the
review and finding the problem.
projects / ceph-build.git / log