5

I have been trying to find memory leak in my application for a week now without any success. I tried to do a heap dump and use jhat to look at the dump and trace down the memory leak.

Is this a best approach? Whats the best way to track down the memory leak with the heap dump.

Appreciate your help.

VM used : java version "1.6.0_25" Java(TM) SE Runtime Environment (build 1.6.0_25-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)

JVM Options : -Xmx1600m -XX:+UseParallelGC -XX:MaxPermSize=256m -Xms1600m -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -verbose:gc -Xloggc:/tmp/gc.log

OOME Stack trace: Couldn't get this. The kernel killed the process with out of memory error.

GC Log : The last few lines

48587.245: [GC [PSYoungGen: 407168K->37504K(476160K)] 506729K->137065K(1568448K), 3.0673560 secs] [Times: user=3.53 sys=0.00, real=3.07 secs] 
50318.617: [GC [PSYoungGen: 444224K->37536K(476416K)] 543785K->175177K(1568704K), 3.6635990 secs] [Times: user=3.70 sys=0.00, real=3.67 secs] 
50453.841: [GC [PSYoungGen: 70092K->2912K(476672K)] 207734K->178513K(1568960K), 1.0164250 secs] [Times: user=1.29 sys=0.00, real=1.02 secs] 
50454.858: [Full GC (System) [PSYoungGen: 2912K->0K(476672K)] [PSOldGen: 175601K->137776K(1092288K)] 178513K->137776K(1568960K) [PSPermGen: 60627K->60627K(74368K)], 2.0082140 secs] [Times: user=2.09 sys=0.00, real=2.01 secs] 
52186.496: [GC [PSYoungGen: 407104K->37312K(444416K)] 544880K->175088K(1536704K), 3.3705440 secs] [Times: user=3.93 sys=0.00, real=3.37 secs] 
53919.975: [GC [PSYoungGen: 444416K->37536K(476608K)] 582192K->213032K(1568896K), 3.4242980 secs] [Times: user=4.09 sys=0.00, real=3.42 secs] 
54056.872: [GC [PSYoungGen: 70113K->2880K(476480K)] 245609K->216320K(1568768K), 0.9691980 secs] [Times: user=1.19 sys=0.00, real=0.97 secs] 
54057.842: [Full GC (System) [PSYoungGen: 2880K->0K(476480K)] [PSOldGen: 213440K->99561K(1092288K)] 216320K->99561K(1568768K) [PSPermGen: 60628K->60628K(72320K)], 2.2203320 secs] [Times: user=2.23 sys=0.01, real=2.22 secs] 
55796.688: [GC [PSYoungGen: 406976K->37504K(476160K)] 506537K->137065K(1568448K), 3.2680080 secs]

Update: Upon checking the kernel log messages, its a oom-killer. But still why is the system killing the process, isn't it because the process eating up lot of system resources ( memory ).

Kathir
  • 5,748
  • 7
  • 32
  • 67
  • http://stackoverflow.com/questions/6754923/how-to-find-native-memory-leaks-caused-by-a-java-code – Jeffrey Jul 20 '11 at 01:05
  • 1
    The link is talking abt native code but i will give a try using jconsole. – Kathir Jul 20 '11 at 01:12
  • Have you tried to use "print Memory info" in those suspected part of code? It may help – Kit Ho Jul 21 '11 at 18:04
  • Java won't use more than the specified max heap size. You should set the max heap to whatever your application requires, and then make sure that the server in question can provide that much memory to the java process. I added more details below. – jtoberon Jul 24 '11 at 17:12

3 Answers3

11

The question about java memory leaks is a duplicate of this, that, etc. Still, here are a few thoughts:

Start by taking a few heap snapshots as described in the answer linked above.

Then, if you know the whole application well, you can eyeball the instance counts and find which type has too many instances sticking around. For example, if you know that a class is a singleton, yet you see 100 instances of that class in memory, then that's a sure sign that something funny is going on there. Alternatively you can compare the snapshots to find which types of objects are growing in number over time; the key here is that you're looking for relative growth over some usage period.

Once you know what's leaking, you trace back through the references to find the root reference that cannot be collected.

Finally, remember that it's possible that you see an OutOfMemoryError not because you're leaking memory, but rather because some part of your heap is too small for the application. To check whether this is the case:

  • Include in your question the type of VM that you're using.
  • Include in your question the arguments that you pass when starting java. What are your min, max, and permgen heap sizes? What type of garbage collector are you using?
  • Include in your question the OOME stack trace, in case there's some useful information there.
  • Turn on verbose GC logging, so that you can see which part(s) of the heap are growing.
  • Turn on the HeapDumpOnOutOfMemoryError parameter, so that you get a heap dump at the very end when the process dies.

Update: I'm not sure what "kernel killed the process with out of memory error" in your latest update means, but I think you might be saying that the linux out of memory killer was invoked. Was this the case? This problem is completely separate from a java OutOfMemoryError. For more details about what's happening, take a look at the links from the page I just linked to, including this and that. But the solution to your problem is simple: use less memory on the server in question. I suppose that you could drop the min and max heap size of the java process in question, but you need to be sure that you won't trigger real java OutOfMemoryErrors. Can you move some processes elsewhere? Can you correlate the memory killer with the start up of a specific process?

Community
  • 1
  • 1
jtoberon
  • 7,505
  • 1
  • 31
  • 44
  • I have followed the steps you have mentioned. I used jprofiler and jconsole. When i ran the GC from jprofiler it recovered all the memory but when i did 'htop' on the server it was still showing a lot of memory being used by the tomcat and it crasehd with outofmemoryexception. I have no clue why the system process still shows the memory as being used. The tomcat is running on a centos. – Kathir Jul 20 '11 at 05:29
  • Understanding a java heap dump is different than debugging an OutOfMemoryError. I'll update my answer with some more information about the later. – jtoberon Jul 20 '11 at 13:15
  • Also, you haven't said what you found when you compared instance counts. Did any type of object grow in number? For what it's worth, I don't understand what relevance "the system process still shows the memory as being used" has, but the java process may not stop when an OOME happens. – jtoberon Jul 20 '11 at 13:25
  • The instance counts were showing a lot of HashMapEntry instances. I believe these entries are from Hibernate and we are fetching a lot of database records. But eventually they all get gc'ed. – Kathir Jul 20 '11 at 17:21
  • Is Hibernate caching data? When you compare several heap dumps, does this cache size grow over time? How do you know that data will be evicted from the cache -- e.g. is there a max size, does the cache use weak references, etc? Also, if you're loading a lot of data, then you might have set your max heap size too low. – jtoberon Jul 20 '11 at 17:58
  • OK. Let me try to use less memory setting and see. – Kathir Jul 25 '11 at 05:16
  • I have used Eclipse MAT to look at the java heap and that showed a class in Google web toolkit using most of the application memory but not releasing it for GC. This thread really helped me understanding a lot about the heap and all the memory configuration i can try with java. Thank you. – Kathir Jul 27 '11 at 20:56
2

Memory leaks can be solved by following below mentioned 3 simple steps:

Step 1: Capture heap dump during early stage

Start your application. Let it take real traffic for 10 minutes. At this point capture heap dump. Heap Dump is basically snapshot of your memory. It contains all objects that are residing in the memory, values stored in those objects, inbound & outbound references of those object. You can capture Heap dump can be captured using following command:

   jmap -dump:format=b,file=<file-path> <pid> 

   where

   pid: is the Java Process Id, whose heap dump should be captured
   file-path: is the file path where heap dump will be written in to.

If you don't want to use jmap for capturing heap dumps, here are several other options to capture heap dumps.

Step 2: Capture heap dump before application crashes

After doing step #1, let the application run. Before application crashes take another heap dump once again. Often times it might be challenging to capture heap dumps before it crashes, because we don't know when application will crash. Is it after 30 minutes, 3 hours, 3 days? Thus, it's ideal to start your application with following JVM property:

    -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<file-path>

    file-path: is the file path where heap dump will be written in to.

This property will trigger heap dump right when application experiences OutOfMemoryError.

Step 3: Analyze heap dumps

Objects which are causing memory leaks grow over the period. If you can the identify objects whose size has grown between the heap dumps captured in step #1 and step #2, then those are the objects which is causing memory leak.

You can consider using heap dump analyzer tool such as HeapHero.io , Eclipse MAT for this purpose. When you load the heap dumps in to any of the tool, there will be a section reporting largest objects in the memory. Compare this section between heap dump captured in step #1 and step #2. If you notice any abnormal growth of objects, then they are the ones which is causing memory leak in your application.

Ram Lakshmanan
  • 81
  • 4
0

I recommend that you review my original article on this subject. Java heap dump analysis can be complex at first but using tools such as Eclipse Memory Analyzer simplifies the process.

JVM Heap Dump's are useful for the following scenarios:

  • Identification of a Java-level memory leak.
  • Identification of a Java-level class loader leak.
  • To learn more about your Java application memory footprint under a certain load/traffic scenario.

Reference: Java heap dump: are you up to the task?

P-H
  • 381
  • 3
  • 5