msg/simple: significantly reduce minimal memory usage of Pipes

Pipe class instance weights over 17 KB in size, and most of it is used by
msgvec structure, which weights 16 KB. Reduce it to 4KB by reducing
number of iovecs in this array down to 256 (= 1024/4). This won't cause
noticeable perf decrease, as even under extreme loads, we exceed 256 used
iovecs in only less than 1% of all write calls. For high-density nodes and
large clusters, savings will exceed hundreds of megabytes per node.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

diff --git a/src/msg/simple/Pipe.cc b/src/msg/simple/Pipe.cc
index 9bfa230ff2296e250d7270bf7da80d59281f3272..f6eee3394154bbc63daf5cefd9cfb49cfdc94218 100644 (file)
--- a/src/msg/simple/Pipe.cc
+++ b/src/msg/simple/Pipe.cc
@@ -2372,7 +2372,7 @@ int Pipe::write_message(const ceph_msg_header& header, const ceph_msg_footer& fo
             << " writing " << donow 
             << dendl;
     
-    if (msg.msg_iovlen >= IOV_MAX-2) {
+    if (msg.msg_iovlen >= SM_IOV_MAX-2) {
       if (do_sendmsg(&msg, msglen, true))
        goto fail;
       

diff --git a/src/msg/simple/Pipe.h b/src/msg/simple/Pipe.h
index 664f733f4688f2db934ff9e8442da1789e7f0bf4..d9e346c7e4012e282eb0f37993a4d4cfd2206041 100644 (file)
--- a/src/msg/simple/Pipe.h
+++ b/src/msg/simple/Pipe.h
@@ -27,6 +27,8 @@ class SimpleMessenger;
 class IncomingQueue;
 class DispatchQueue;
 
+static const int SM_IOV_MAX = (IOV_MAX >= 1024 ? IOV_MAX / 4 : IOV_MAX);
+
   /**
    * The Pipe is the most complex SimpleMessenger component. It gets
    * two threads, one each for reading and writing on a socket it's handed
@@ -175,7 +177,7 @@ class DispatchQueue;
 
   private:
     int sd;
-    struct iovec msgvec[IOV_MAX];
+    struct iovec msgvec[SM_IOV_MAX];
 #if !defined(MSG_NOSIGNAL) && !defined(SO_NOSIGPIPE)
     sigset_t sigpipe_mask;
     bool sigpipe_pending;