init-ceph: make crush update on osd start time out

If the monitor is not currently available, this crush update would block
forever, preventing the OSD and (potentially) the rest of the system
from starting up.  Instead, make it time out after 10 seconds and then
abort startup.  This prevents startup of an OSD if we failed to update
the CRUSH position for some reason.

In fact, do not start up the OSD if the CRUSH update fails for any
reason--not just a timeout!

Works-around: #5612
Signed-off-by: Sage Weil <sage@inktank.com>

diff --git a/src/init-ceph.in b/src/init-ceph.in
index 46877d755587dbb85d5f31d51820fb3f13243001..1a80a42b03ec9a14df3669940998e1199002bace 100644 (file)
--- a/src/init-ceph.in
+++ b/src/init-ceph.in
@@ -324,7 +324,7 @@ for name in $what; do
                    get_conf osd_weight "" "osd crush initial weight"
                    defaultweight="$(do_cmd "df $osd_data/. | tail -1 | awk '{ d= \$2/1073741824 ; r = sprintf(\"%.2f\", d); print r }'")"
                    get_conf osd_keyring "$osd_data/keyring" "keyring"
-                   do_cmd "$BINDIR/ceph \
+                   do_cmd "timeout 10 $BINDIR/ceph \
                        --name=osd.$id \
                        --keyring=$osd_keyring \
                        osd crush create-or-move \
@@ -333,8 +333,7 @@ for name in $what; do
                        ${osd_weight:-${defaultweight:-1}} \
                        root=default \
                        host=$host \
-                       $osd_location \
-                       || :"
+                       $osd_location"
                fi
            fi