ceph-*-build: set jobs number according to free memory
limits the job number of building ceph/ceph pull requests and deb packages
with the (size of free memory in MB)/1800.
guess we are using more compile-time optimizations now, so to compile
ceph source requires more memory. sometimes, a single cc1plus takes
more than 3GB memeory. that's why we are seeing more and more OOM
in our arm64 builders. following is a sample from omani09 -- a
arm64 builder compiling a ceph/ceph PR targeting master:
30474 jenkins+ 20 0
2573092 2.388g 16220 R 100.0 3.8 9:11.04 cc1plus
34339 jenkins+ 20 0
3279768 3.005g 16296 R 100.0 4.8 5:07.63 cc1plus
36382 jenkins+ 20 0 322276 278932 12740 R 100.0 0.4 0:09.78 cc1plus
...
also, the performance of over-all compiling is also impacted by the I/O
subsystem. so lower the number of job could actually reduce the time
of the compiling processes to completing for the I/O queue of local
device. so we can use an conservative number for calc an upper bound of
job number for "make" instead using $(nproc). in this change,
$(free_memory_in_mega / 1800) is used as the upper limit of n_jobs.
on a typical arm64 builder with 48 cores and 64 GB mem, the n_jobs is
now 34 .
when building rpm packages, the number of build jobs is specified by
_smp_mflags macro, which is defined by
/usr/lib/rpm/platform/*/macros and /usr/lib/rpm/redhat/macros.
see
https://github.com/rpm-software-management/rpm/blob/master/platform.in#L53
and rhel/centos use following patch
https://git.centos.org/blob/rpms!redhat-rpm-config.git/
eaaa6282147d0797a3733f3b91671b7a0752d448/SOURCES!redhat-rpm-config-9.1.0-ncpus-max.patch;jsessionid=xv8lqw4ipwwetge0i19ejo9t
so one cannot build rpm packages on centos/rhel with more than 16 jobs
when using redhat-rpm-config. and 16 is a safe number for us.
Signed-off-by: Kefu Chai <kchai@redhat.com>