build/ops: restart ceph-osd@.service after 20s instead of 100ms

Instead of the default 100ms pause before trying to restart an OSD, wait
20 seconds instead and retry 30 times instead of 3. There is no scenario
in which restarting an OSD almost immediately after it failed would get
a better result.

It is possible that a failure to start is due to a race with another
systemd unit at boot time. For instance if ceph-disk@.service is
delayed, it may start after the OSD that needs it. A long pause may give
the racing service enough time to complete and the next attempt to start
the OSD may succeed.

This is not a sound alternative to resolve a race, it only makes the OSD
boot process less sensitive. In the example above, the proper fix is to
enable --runtime ceph-osd@.service so that it cannot race at boot time.

The wait delay should not be minutes to preserve the current runtime
behavior. For instance, if an OSD is killed or fails and restarts after
10 minutes, it will be marked down by the ceph cluster.  This is not a
change that could break things but it is significant and should be
avoided.

Refs: http://tracker.ceph.com/issues/17889

Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit b3887379d6dde3b5a44f2e84cf917f4f0a0cb120)

diff --git a/systemd/ceph-osd@.service b/systemd/ceph-osd@.service
index 592b324a8f18b44db051fe75f93e50b3745d8f95..bd5a78be84c256b9ef04edc8dff285003dc8c228 100644 (file)
--- a/systemd/ceph-osd@.service
+++ b/systemd/ceph-osd@.service
@@ -18,7 +18,8 @@ PrivateTmp=true
 TasksMax=infinity
 Restart=on-failure
 StartLimitInterval=30min
-StartLimitBurst=3
+StartLimitBurst=30
+RestartSec=20s
 
 [Install]
 WantedBy=ceph-osd.target