GC overhead limit exceed when reading large file

Question

I want to read my .csv file line by line to save myself from loading everything into RAM at once. I thought this is the way to do it. I also wrote the code in a way that no variables are declared within the loop to save the JVM from always creating new objects and running the Garbage Collector.

However, I keep running into this "GC overhead limited exceeded" error. My CPU also runs with nearly 100%.

Here the problem was caused by the HashMap storing millions String objects - but mine should "only" store about 20.000 of my Node objects.

Please help me find the problematic part of my code. The error reports the line marked in source code below.

This is my code:

HashMap<String,TweetNode> allNodes = new HashMap<String,TweetNode>();
    // read file
    try {
        BufferedReader br = new BufferedReader(new FileReader(graphFile));
        noOfNodes = 0;
        String line = br.readLine();
        String firstNode;
        String[] lineContent;
        while (line != null) {
            lineContent = line.split("\t"); // error occurs here!
            // always look at the first node
            firstNode = lineContent[0];
            if (! allNodes.containsKey(firstNode)) {
                allNodes.put(firstNode, new TweetNode(noOfNodes, firstNode));
                noOfNodes++;
            }
            allNodes.get(firstNode).addNeighbour(lineContent[1], Double.valueOf(lineContent[2]));
            line = br.readLine();
        }
        br.close();
    } 
    // ... catch stuff ...
return allNodes;
}

@StefanFalk Can you elaborate on what you mean by that? Do you mean using a debugging tool? I am not so experienced in that. Do they show when GC collection kicks in? How would you suggest to use it? — anjuta, Jun 09 '15 at 09:41
Thanks for all those downvotes without comments. I really learn a lot of from this. [/irony off] — anjuta, Jun 09 '15 at 09:42
@anjuta you are probably simply filling up existing heap memory when you populate your `HashMap`. Investigate used heap memory with the `jstat` tool for example. I am not sure but my guess is that this question is downvoted due to the lack of research into the memory behavior of your application. — K Erlandsson, Jun 09 '15 at 11:01
@K Erlandsson and @Bhargav Thank your for your help. It indeed was a memory problem. Even when running with -Xmx6g it failed - the file was that large. Now on a server with much more memory it seems to run fine. What I don't understand is why it does not crash with Java HeapSize Error? — anjuta, Jun 10 '15 at 14:49

score 0 · Answer 1 · answered Jun 09 '15 at 12:14

0

The only problem I can see here is your map. Map is filling up the heap memory. You should be running your application with low heap memory. Check the current values by visiting below arguments and set those to a reasonable high value.

The flag Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM), while Xms specifies the initial memory allocation pool.

answered Jun 09 '15 at 12:14

Bhargav Kumar R

2,012
3
20
38

1

I also thought that the map might cause a Java HeapSize Error - but why does it complain about garbage collection? I would not expect my program to cause much gc activity at all - I don't create new objects all the time, their content is rather updated. What am I getting wrong here? – anjuta Jun 10 '15 at 14:50
What is your heap memory size – Bhargav Kumar R Jun 10 '15 at 14:53
I didn't specify it, but from the answers here http://stackoverflow.com/questions/4667483/how-is-the-default-java-heap-size-determined I guess it should be 2gb maximum. – anjuta Jun 10 '15 at 15:36
The quite unreadable information given from the command in the above linked question's answer was: uintx ErgoHeapSizeLimit = 0 {product} uintx HeapSizePerGCThread = 87241520 {product} uintx InitialHeapSize := 130023424 {product} uintx LargePageHeapSizeThreshold = 134217728 {product} uintx MaxHeapSize := 2071986176 {product} – anjuta Jun 10 '15 at 15:37
I can see ~2GB given to heap..but try to connect to jconsole for your application to see how much memory actually ur app getting..This gives idea on next steps to be done – Bhargav Kumar R Jun 11 '15 at 08:15
Thank you for trying to help. I actually don't want to torture my computer like this anymore, so I am not going to run the program again. It was quite hard to get my computer's attention to stop executing the program and it ran very hot in this process. So I'd rather not experiment further. I understand now that I was mislead by the error message and it was not (or at least, not really) a problem of GC, but a problem of heap space. – anjuta Jun 11 '15 at 09:32
I would be happy if u call it as learning instead of torturing :D – Bhargav Kumar R Jun 15 '15 at 04:46

GC overhead limit exceed when reading large file

1 Answers1