msg/async/rdma: Add fork safe on RDMA

Using forks with RDMA is dangerous, as memory is registered directly to
hardware. In the special case where the child processes do not need to
do any RDMA, calling ibv_fork_init() will instruct the OS to keep the
physical pages mapped to the parent process, hence allowing it to keep
doing RDMA safely. Should be called before any other RDMA related calls.

issue: 972567

Change-Id: I0816f62aadb02d8aa060db0c648198026bd71514
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>

diff --git a/src/msg/async/rdma/RDMAStack.cc b/src/msg/async/rdma/RDMAStack.cc
index 46ec73b7c9f1af765d0cde36ffd2a1e707870e87..f7061ce6b36b2b238ad8804c38c799b6b50bd148 100644 (file)
--- a/src/msg/async/rdma/RDMAStack.cc
+++ b/src/msg/async/rdma/RDMAStack.cc
@@ -524,6 +524,15 @@ void RDMAWorker::handle_pending_message()
 
 RDMAStack::RDMAStack(CephContext *cct, const string &t): NetworkStack(cct, t)
 {
+  //
+  //On RDMA MUST be called before fork
+  //
+  int rc = ibv_fork_init();
+  if (rc) {
+     lderr(cct) << __func__ << " failed to call ibv_for_init(). On RDMA must be called before fork. Application aborts." << dendl;
+     ceph_abort();
+  }
+
   //Check ulimit
   struct rlimit limit;
   getrlimit(RLIMIT_MEMLOCK, &limit);