From 38be14bc0fa32be6877dea08ebd35495d39e464f Mon Sep 17 00:00:00 2001 From: Kefu Chai Date: Thu, 3 Jun 2021 20:41:36 +0800 Subject: [PATCH] ceph.spec.in: increase the mem_per_job to 3GiB to lower the number of jobs, we are experiencing build failures on a builder with 48c96t, 193 free mem. the failures were caused by OOM killer which kills the c++ compiler [498376.128969] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/jenkins.service,task=cc1plus,pid=1387895,uid=1110 [498376.145288] Out of memory: Killed process 1387895 (cc1plus) total-vm:3323312kB, anon-rss:3164568kB, file-rss:0kB, shmem-rss:0kB, UID:1110 [498376.315185] oom_reaper: reaped process 1387895 (cc1plus), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [498377.882072] cc1plus invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 before this change, we use the total memory to calculate the number of jobs, and assume that each job takes at most 2.5GiB mem. in the case above, the # of job is 96. after this change, we use the free memory, and increse the mem per job to 3.0GiB. in the case above, the # of job would be 85. Signed-off-by: Kefu Chai --- ceph.spec.in | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ceph.spec.in b/ceph.spec.in index b4a1e68e5ca..10a9f3f636a 100644 --- a/ceph.spec.in +++ b/ceph.spec.in @@ -127,7 +127,7 @@ %define smp_limit_mem_per_job() %( \ kb_per_job=%1 \ - kb_total=$(head -1 /proc/meminfo | sed 's/MemTotal:\\s*\\(.*\\) kB.*/\\1/') \ + kb_total=$(head -3 /proc/meminfo | sed -n 's/MemAvailable:\\s*\\(.*\\) kB.*/\\1/p') \ jobs=$(( $kb_total / $kb_per_job )) \ [ $jobs -lt 1 ] && jobs=1 \ echo $jobs ) @@ -137,8 +137,8 @@ # 32-bit builds can use 3G memory max, which is not enough even for -j2 %global _smp_ncpus_max 1 %else -# 2.5 GiB mem per job -%global _smp_ncpus_max %{smp_limit_mem_per_job 2500000} +# 3.0 GiB mem per job +%global _smp_ncpus_max %{smp_limit_mem_per_job 3000000} %endif %endif -- 2.39.5