From: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
Date: Mon, 8 Feb 2016 13:05:03 +0000 (+0100)
Subject: msg/simple: significantly reduce minimal memory usage of Pipes
X-Git-Tag: v10.1.0~404^2
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=refs%2Fpull%2F7567%2Fhead;p=ceph.git

msg/simple: significantly reduce minimal memory usage of Pipes

Pipe class instance weights over 17 KB in size, and most of it is used by
msgvec structure, which weights 16 KB. Reduce it to 4KB by reducing
number of iovecs in this array down to 256 (= 1024/4). This won't cause
noticeable perf decrease, as even under extreme loads, we exceed 256 used
iovecs in only less than 1% of all write calls. For high-density nodes and
large clusters, savings will exceed hundreds of megabytes per node.

Signed-off-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
---

diff --git a/src/msg/simple/Pipe.cc b/src/msg/simple/Pipe.cc
index 9bfa230ff229..f6eee3394154 100644
--- a/src/msg/simple/Pipe.cc
+++ b/src/msg/simple/Pipe.cc
@@ -2372,7 +2372,7 @@ int Pipe::write_message(const ceph_msg_header& header, const ceph_msg_footer& fo
 	     << " writing " << donow 
 	     << dendl;
     
-    if (msg.msg_iovlen >= IOV_MAX-2) {
+    if (msg.msg_iovlen >= SM_IOV_MAX-2) {
       if (do_sendmsg(&msg, msglen, true))
 	goto fail;
       
diff --git a/src/msg/simple/Pipe.h b/src/msg/simple/Pipe.h
index 664f733f4688..d9e346c7e401 100644
--- a/src/msg/simple/Pipe.h
+++ b/src/msg/simple/Pipe.h
@@ -27,6 +27,8 @@ class SimpleMessenger;
 class IncomingQueue;
 class DispatchQueue;
 
+static const int SM_IOV_MAX = (IOV_MAX >= 1024 ? IOV_MAX / 4 : IOV_MAX);
+
   /**
    * The Pipe is the most complex SimpleMessenger component. It gets
    * two threads, one each for reading and writing on a socket it's handed
@@ -175,7 +177,7 @@ class DispatchQueue;
 
   private:
     int sd;
-    struct iovec msgvec[IOV_MAX];
+    struct iovec msgvec[SM_IOV_MAX];
 #if !defined(MSG_NOSIGNAL) && !defined(SO_NOSIGPIPE)
     sigset_t sigpipe_mask;
     bool sigpipe_pending;