From: Sarit Zubakov <saritz@mellanox.com>
Date: Mon, 20 Feb 2017 13:45:17 +0000 (+0000)
Subject: msg/async/rdma: Add fork safe on RDMA
X-Git-Tag: v12.0.2~309^2
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=8064dbf89f11839119285d6ba168ca28d3197853;p=ceph.git

msg/async/rdma: Add fork safe on RDMA

Using forks with RDMA is dangerous, as memory is registered directly to
hardware. In the special case where the child processes do not need to
do any RDMA, calling ibv_fork_init() will instruct the OS to keep the
physical pages mapped to the parent process, hence allowing it to keep
doing RDMA safely. Should be called before any other RDMA related calls.

issue: 972567

Change-Id: I0816f62aadb02d8aa060db0c648198026bd71514
Signed-off-by: Sarit Zubakov <saritz@mellanox.com>
---

diff --git a/src/msg/async/rdma/RDMAStack.cc b/src/msg/async/rdma/RDMAStack.cc
index 46ec73b7c9f1..f7061ce6b36b 100644
--- a/src/msg/async/rdma/RDMAStack.cc
+++ b/src/msg/async/rdma/RDMAStack.cc
@@ -524,6 +524,15 @@ void RDMAWorker::handle_pending_message()
 
 RDMAStack::RDMAStack(CephContext *cct, const string &t): NetworkStack(cct, t)
 {
+  //
+  //On RDMA MUST be called before fork
+  //
+  int rc = ibv_fork_init();
+  if (rc) {
+     lderr(cct) << __func__ << " failed to call ibv_for_init(). On RDMA must be called before fork. Application aborts." << dendl;
+     ceph_abort();
+  }
+
   //Check ulimit
   struct rlimit limit;
   getrlimit(RLIMIT_MEMLOCK, &limit);