From: Loic Dachary Date: Thu, 2 Oct 2014 13:16:28 +0000 (+0200) Subject: doc: update memory-profiling with examples X-Git-Tag: v0.88~92^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=64ca7443132e860b63c8584ac3e28546c1583773;p=ceph.git doc: update memory-profiling with examples * Update the ceph tell from ceph daemon tell id to the new ceph tell deamon.id form * Add usages examples for easier copy / paste * Add MON to the list of daemons that can be profiled * Document CEPH_HEAP_PROFILER_INIT=true * Remove trailing empty lines Signed-off-by: Loic Dachary --- diff --git a/doc/rados/troubleshooting/memory-profiling.rst b/doc/rados/troubleshooting/memory-profiling.rst index 1d878a88b7fd..39ceba694ef8 100644 --- a/doc/rados/troubleshooting/memory-profiling.rst +++ b/doc/rados/troubleshooting/memory-profiling.rst @@ -2,9 +2,9 @@ Memory Profiling ================== -Ceph OSD and metadata server daemons can generate heap profiles using -``tcmalloc``. To generate heap profiles, ensure you have ``google-perftools`` -installed:: +Ceph MON, OSD and MDS can generate heap profiles using +``tcmalloc``. To generate heap profiles, ensure you have +``google-perftools`` installed:: sudo apt-get google-perftools @@ -13,15 +13,61 @@ The profiler dumps output to your ``log file`` directory (i.e., To view the profiler logs with Google's performance tools, execute the following:: - google-pprof -gv {log-path/filename} + google-pprof --text {path-to-daemon} {log-path/filename} + +For example:: + + $ ceph tell osd.0 heap start_profiler + $ ceph tell osd.0 heap dump + osd.0 tcmalloc heap stats:------------------------------------------------ + MALLOC: 2632288 ( 2.5 MiB) Bytes in use by application + MALLOC: + 499712 ( 0.5 MiB) Bytes in page heap freelist + MALLOC: + 543800 ( 0.5 MiB) Bytes in central cache freelist + MALLOC: + 327680 ( 0.3 MiB) Bytes in transfer cache freelist + MALLOC: + 1239400 ( 1.2 MiB) Bytes in thread cache freelists + MALLOC: + 1142936 ( 1.1 MiB) Bytes in malloc metadata + MALLOC: ------------ + MALLOC: = 6385816 ( 6.1 MiB) Actual memory used (physical + swap) + MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped) + MALLOC: ------------ + MALLOC: = 6385816 ( 6.1 MiB) Virtual address space used + MALLOC: + MALLOC: 231 Spans in use + MALLOC: 56 Thread heaps in use + MALLOC: 8192 Tcmalloc page size + ------------------------------------------------ + Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). + Bytes released to the OS take up virtual address space but no physical memory. + $ google-pprof --text \ + /usr/bin/ceph-osd \ + /var/log/ceph/ceph-osd.0.profile.0001.heap + Total: 3.7 MB + 1.9 51.1% 51.1% 1.9 51.1% ceph::log::Log::create_entry + 1.8 47.3% 98.4% 1.8 47.3% std::string::_Rep::_S_create + 0.0 0.4% 98.9% 0.0 0.6% SimpleMessenger::add_accept_pipe + 0.0 0.4% 99.2% 0.0 0.6% decode_message + ... + +Another heap dump on the same daemon will add another file. It is +convenient to compare to a previous heap dump to show what has grown +in the interval. For instance:: + + $ google-pprof --text --base out/osd.0.profile.0001.heap \ + ceph-osd out/osd.0.profile.0003.heap + Total: 0.2 MB + 0.1 50.3% 50.3% 0.1 50.3% ceph::log::Log::create_entry + 0.1 46.6% 96.8% 0.1 46.6% std::string::_Rep::_S_create + 0.0 0.9% 97.7% 0.0 26.1% ReplicatedPG::do_op + 0.0 0.8% 98.5% 0.0 0.8% __gnu_cxx::new_allocator::allocate Refer to `Google Heap Profiler`_ for additional details. -Once you have the heap profiler installed, start your cluster and begin using -the heap profiler. You may enable or disable the heap profiler at runtime, or -ensure that it runs continuously. For the following commandline usage, replace -``{daemon-type}`` with ``osd`` or ``mds``, and replace ``daemon-id`` with the -OSD number or metadata server letter. +Once you have the heap profiler installed, start your cluster and +begin using the heap profiler. You may enable or disable the heap +profiler at runtime, or ensure that it runs continuously. For the +following commandline usage, replace ``{daemon-type}`` with ``mon``, +``osd`` or ``mds``, and replace ``{daemon-id}`` with the OSD number or +the MON or MDS id. Starting the Profiler @@ -29,23 +75,26 @@ Starting the Profiler To start the heap profiler, execute the following:: - ceph {daemon-type} tell {daemon-id} heap start_profiler + ceph tell {daemon-type}.{daemon-id} heap start_profiler For example:: - ceph osd tell 1 heap start_profiler + ceph tell osd.1 heap start_profiler +Alternatively the profile can be started when the daemon starts +running if the ``CEPH_HEAP_PROFILER_INIT=true`` variable is found in +the environment. Printing Stats -------------- To print out statistics, execute the following:: - ceph {daemon-type} tell {daemon-id} heap stats + ceph tell {daemon-type}.{daemon-id} heap stats For example:: - ceph osd tell 0 heap stats + ceph tell osd.0 heap stats .. note:: Printing stats does not require the profiler to be running and does not dump the heap allocation information to a file. @@ -56,11 +105,11 @@ Dumping Heap Information To dump heap information, execute the following:: - ceph {daemon-type} tell {daemon-id} heap dump + ceph tell {daemon-type}.{daemon-id} heap dump For example:: - ceph mds tell a heap dump + ceph tell mds.a heap dump .. note:: Dumping heap information only works when the profiler is running. @@ -71,11 +120,11 @@ Releasing Memory To release memory that ``tcmalloc`` has allocated but which is not being used by the Ceph daemon itself, execute the following:: - ceph {daemon-type} tell {daemon-id} heap release + ceph tell {daemon-type}{daemon-id} heap release For example:: - ceph osd tell 2 heap release + ceph tell osd.2 heap release Stopping the Profiler @@ -83,14 +132,11 @@ Stopping the Profiler To stop the heap profiler, execute the following:: - ceph {daemon-type} tell {daemon-id} heap stop_profiler + ceph tell {daemon-type}.{daemon-id} heap stop_profiler For example:: - ceph {daemon-type} tell {daemon-id} heap stop_profiler + ceph tell osd.0 heap stop_profiler .. _Logging and Debugging: ../log-and-debug .. _Google Heap Profiler: http://google-perftools.googlecode.com/svn/trunk/doc/heapprofile.html - - -