doc: update memory-profiling with examples

* Update the ceph tell from ceph daemon tell id to the new
  ceph tell deamon.id form

* Add usages examples for easier copy / paste

* Add MON to the list of daemons that can be profiled

* Document CEPH_HEAP_PROFILER_INIT=true

* Remove trailing empty lines

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

diff --git a/doc/rados/troubleshooting/memory-profiling.rst b/doc/rados/troubleshooting/memory-profiling.rst
index 1d878a88b7fdc1c7d1282bc9c79add87a6118699..39ceba694ef8792d40a88827a105f6f50c2b82d3 100644 (file)
--- a/doc/rados/troubleshooting/memory-profiling.rst
+++ b/doc/rados/troubleshooting/memory-profiling.rst
@@ -2,9 +2,9 @@
  Memory Profiling
 ==================
 
-Ceph OSD and metadata server daemons can generate heap profiles using
-``tcmalloc``. To generate heap profiles, ensure you have ``google-perftools``
-installed::
+Ceph MON, OSD and MDS can generate heap profiles using
+``tcmalloc``. To generate heap profiles, ensure you have
+``google-perftools`` installed::
 
        sudo apt-get google-perftools
 
@@ -13,15 +13,61 @@ The profiler dumps output to your ``log file`` directory (i.e.,
 To view the profiler logs with Google's performance tools, execute the
 following:: 
 
-       google-pprof -gv {log-path/filename}
+    google-pprof --text {path-to-daemon}  {log-path/filename}
+
+For example::
+
+    $ ceph tell osd.0 heap start_profiler
+    $ ceph tell osd.0 heap dump
+    osd.0 tcmalloc heap stats:------------------------------------------------
+    MALLOC:        2632288 (    2.5 MiB) Bytes in use by application
+    MALLOC: +       499712 (    0.5 MiB) Bytes in page heap freelist
+    MALLOC: +       543800 (    0.5 MiB) Bytes in central cache freelist
+    MALLOC: +       327680 (    0.3 MiB) Bytes in transfer cache freelist
+    MALLOC: +      1239400 (    1.2 MiB) Bytes in thread cache freelists
+    MALLOC: +      1142936 (    1.1 MiB) Bytes in malloc metadata
+    MALLOC:   ------------
+    MALLOC: =      6385816 (    6.1 MiB) Actual memory used (physical + swap)
+    MALLOC: +            0 (    0.0 MiB) Bytes released to OS (aka unmapped)
+    MALLOC:   ------------
+    MALLOC: =      6385816 (    6.1 MiB) Virtual address space used
+    MALLOC:
+    MALLOC:            231              Spans in use
+    MALLOC:             56              Thread heaps in use
+    MALLOC:           8192              Tcmalloc page size
+    ------------------------------------------------
+    Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
+    Bytes released to the OS take up virtual address space but no physical memory.
+    $ google-pprof --text \
+                   /usr/bin/ceph-osd  \
+                   /var/log/ceph/ceph-osd.0.profile.0001.heap
+     Total: 3.7 MB
+     1.9  51.1%  51.1%      1.9  51.1% ceph::log::Log::create_entry
+     1.8  47.3%  98.4%      1.8  47.3% std::string::_Rep::_S_create
+     0.0   0.4%  98.9%      0.0   0.6% SimpleMessenger::add_accept_pipe
+     0.0   0.4%  99.2%      0.0   0.6% decode_message
+     ...
+
+Another heap dump on the same daemon will add another file. It is
+convenient to compare to a previous heap dump to show what has grown
+in the interval. For instance::
+
+    $ google-pprof --text --base out/osd.0.profile.0001.heap \
+          ceph-osd out/osd.0.profile.0003.heap
+     Total: 0.2 MB
+     0.1  50.3%  50.3%      0.1  50.3% ceph::log::Log::create_entry
+     0.1  46.6%  96.8%      0.1  46.6% std::string::_Rep::_S_create
+     0.0   0.9%  97.7%      0.0  26.1% ReplicatedPG::do_op
+     0.0   0.8%  98.5%      0.0   0.8% __gnu_cxx::new_allocator::allocate
 
 Refer to `Google Heap Profiler`_ for additional details.
 
-Once you have the heap profiler installed, start your cluster and begin using
-the heap profiler. You may enable or disable the heap profiler at runtime, or
-ensure that it runs continuously. For the following commandline usage, replace
-``{daemon-type}`` with ``osd`` or ``mds``, and replace ``daemon-id`` with the
-OSD number or metadata server letter.
+Once you have the heap profiler installed, start your cluster and
+begin using the heap profiler. You may enable or disable the heap
+profiler at runtime, or ensure that it runs continuously. For the
+following commandline usage, replace ``{daemon-type}`` with ``mon``,
+``osd`` or ``mds``, and replace ``{daemon-id}`` with the OSD number or
+the MON or MDS id.
 
 
 Starting the Profiler
@@ -29,23 +75,26 @@ Starting the Profiler
 
 To start the heap profiler, execute the following:: 
 
-       ceph {daemon-type} tell {daemon-id} heap start_profiler
+       ceph tell {daemon-type}.{daemon-id} heap start_profiler
 
 For example:: 
 
-       ceph osd tell 1 heap start_profiler
+       ceph tell osd.1 heap start_profiler
 
+Alternatively the profile can be started when the daemon starts
+running if the ``CEPH_HEAP_PROFILER_INIT=true`` variable is found in
+the environment.
 
 Printing Stats
 --------------
 
 To print out statistics, execute the following:: 
 
-       ceph {daemon-type} tell {daemon-id} heap stats
+       ceph  tell {daemon-type}.{daemon-id} heap stats
 
 For example:: 
 
-       ceph osd tell 0 heap stats
+       ceph tell osd.0 heap stats
 
 .. note:: Printing stats does not require the profiler to be running and does
    not dump the heap allocation information to a file.
@@ -56,11 +105,11 @@ Dumping Heap Information
 
 To dump heap information, execute the following:: 
 
-       ceph {daemon-type} tell {daemon-id} heap dump
+       ceph tell {daemon-type}.{daemon-id} heap dump
 
 For example:: 
 
-       ceph mds tell a heap dump
+       ceph tell mds.a heap dump
 
 .. note:: Dumping heap information only works when the profiler is running.
 
@@ -71,11 +120,11 @@ Releasing Memory
 To release memory that ``tcmalloc`` has allocated but which is not being used by
 the Ceph daemon itself, execute the following:: 
 
-       ceph {daemon-type} tell {daemon-id} heap release
+       ceph tell {daemon-type}{daemon-id} heap release
 
 For example:: 
 
-       ceph osd tell 2 heap release
+       ceph tell osd.2 heap release
 
 
 Stopping the Profiler
@@ -83,14 +132,11 @@ Stopping the Profiler
 
 To stop the heap profiler, execute the following:: 
 
-       ceph {daemon-type} tell {daemon-id} heap stop_profiler
+       ceph tell {daemon-type}.{daemon-id} heap stop_profiler
 
 For example:: 
 
-       ceph {daemon-type} tell {daemon-id} heap stop_profiler
+       ceph tell osd.0 heap stop_profiler
 
 .. _Logging and Debugging: ../log-and-debug
 .. _Google Heap Profiler: http://google-perftools.googlecode.com/svn/trunk/doc/heapprofile.html
-
-
-