2

I implemented a heuristic in Java that solves an optimization problem for a given input. The heuristic can run for thousands of iterations and create lots of objects of varying complexity.

In order to test it, I have thousands of test inputs. My main method takes all inputs and sequentially starts the heuristic for each input in a loop. The results are stored in a separate file for each input.

When I run the program, it always stops after producing 218 or 219 and throws an "OutOfMemoryError". Once it says Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded and once Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.

My guess is, the program creates too many objects over time until it runs out of memory when computing the 218th or 219th input. Every instance is computed in an independent run. Hence, it should solve the problem to clear the memory and getting rid of all created objects after the result for an input is stored and before the next input is parsed. Is that correct? I heard using System.gc() is bad practice, but what else would you recommend in my case?

Edit: To specify what I want: Instead of pressing "start" for each input, I implemented the loop to do that for me. However, it seems like it doesn't behave the same way and it keeps old objects from previous runs. Can I change my java code in such a way that it behaves similar to starting the program anew for each input? Or do I have to use a shell skript that starts my heuristic for each input separatly to make it work? I have never used any JVM parameters and it seems to me like they don't really tackle the problem.

Resolved: There was in fact a memory leak that I discovered and fixed. No System.gc() needed. Thanks for helping anyways!

CGFoX
  • 2,543
  • 3
  • 31
  • 56
  • 1
    You can catch the `OutOfMemoryError`, but you shouldn't (you can't do anything about it anyways). You have two options: Run Java with extra heap space `-Xmx2048m`, or debug your code to not leave so many objects on the heap. Set the unused objects to `null` after you're done with them. And yes, don't call `System.gc()`, it won't help you. – Kon Jan 30 '15 at 17:25
  • Extra heap space won't help if the objects from previous inputs remain on the heap. It will only delay the OutOfMemoryError. And setting all objects that I don't need anymore to `null` seems weird and like a lot of effort. It would be nicer to just clean everything after each run. – CGFoX Jan 30 '15 at 17:33
  • Once the JVM has started and done any non-trivial amount of work there is no way to set it back to "pristine" condition. But if you claim that each iteration of your code is "independent" but you're still getting OOM, your claim of independence is false and you're somehow hanging on to results between iterations. – Hot Licks Jan 30 '15 at 18:26
  • 1
    (You don't need to set all your objects to null (even if that were somehow possible). You just need to make sure that you're not, accidentally or on purpose, maintaining a "history" of prior runs, through a linked list, a Map that is never cleared, or something similar.) – Hot Licks Jan 30 '15 at 18:28
  • 1
    I know this is old, but I'm facing a similar issue, I think for a similar reason. Care to share where your memory leak was? – horsefeathers Apr 27 '17 at 18:13

3 Answers3

0

Yes leave GC handling with JVM. You need to follow some of the steps mentioned below in order:

  • Increase your heap size using Xmx... parameter
  • Set proper GC algorithm and parameters. If you have already have GC parameters try to tune the parameters
  • Try using -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=<path for heap dump> option when you start your JVM, so you get heap dump when your jvm runs OOM. By using the heap dump, you could use profilers like jprofiler/yourkit/jvisualvm etc to investigate memory leaks and then rectify the same.
SMA
  • 33,915
  • 6
  • 43
  • 65
0

First, when you start a JVM to run your tests, disable the GC overhead limit:

-XX:-UseGCOverheadLimit

I recommend this because you already know you're purposefully stressing the garbage collector, and you don't want it to warn you about GC overhead.

Second, take a look at how you can break up your tests better, in such a way that you're allowing objects from the previous test to be garbage collected. Don't keep active pointers to large structures of objects after each test completes.

Third, if you still need more memory due to exceeding Java heap space, use:

-Xms<size>        set initial Java heap size
-Xmx<size>        set maximum Java heap size

If you know you'll be using the memory anyhow, it works best to set both of these to the same value, which prevents thrashing during execution.

Don't bother explicitly calling System.gc(), it's ultimately pointless because garbage collection is always going to happen when it's necessary.

Fourth, another JVM setting which could be useful in your circumstances:

-XX:NewRatio=<n>   Ratio of old/new generation sizes. The default value is 2.

It's normally not recommended to set this lower than 2 (2/3 old, 1/3 new), but in your situation I might suggest you try setting this to 1 (1/2 old, 1/2 new).

See also GC overhead limit exceeded and check out Java HotSpot VM Options.

Community
  • 1
  • 1
gknicker
  • 5,321
  • 2
  • 19
  • 37
0

Give this a try: http://javaandroidandrest.blogspot.de/2012/06/wait-for-jvm-garbage-collector.html

From the site:

Using functions like System.gc(); or Runtime.getRuntime().gc(); only suggest to the JVM that you want to run the garbage collector. I found a way on the internet not to force the grabage collector but to wait until the garbage collector runs.

MarkusAtCvlabDotDe
  • 982
  • 4
  • 10