0

I am implementing an external sort for a big file (~30GB), so after I have written the chunks in disk, I create chunks times BufferedReader(new OutputStreamWriter(new FileOutputStream(outputPath), "UTF-8"), maxBufferSize) being maxBufferSize = Runtime.getRuntime().freeMemory() / chunks. However I get a OutOfMemory Exception.

I guess that the garbage collector didn't have time enough to clean the memory (when I stop with the debugger it does not throw the exception), but in that case, why Runtime.getRuntime().freeMemory() is given that result?

Is it possible to explicitly call the garbage collection or the only option is sleep the process for some time?

Markinhos
  • 706
  • 1
  • 6
  • 12
  • Possible duplicate of [Can we call the Garbage Collector explicitly?](http://stackoverflow.com/questions/15632734/can-we-call-the-garbage-collector-explicitly) – Eric J. Jan 05 '16 at 03:22
  • Note also that freeMemory() tells you the total amount of free space on the heap, but there is no guarantee that there is a single contiguous block that large (in fact, that will usually not be the case). – Eric J. Jan 05 '16 at 03:23
  • It would be important to konw what kind of `OutOfMemoryError` you get - without knowing that we might send you off in the completely wrong direction. Also some (relevant) code would be nice.... – piet.t Jan 05 '16 at 07:19
  • why are you using such large buffers in the first place? – the8472 Jan 05 '16 at 12:10

2 Answers2

2

Is it possible to explicitly call the garbage collection

Yes it is possible. But it won't do any good.

The JVM will only throw an OOME after performing a full GC. Calling System.gc() explicitly will (most likely) just waste CPU time.

Actually, I think your real problem is here:

I create chunks times BufferedReader(new OutputStreamWriter(new FileOutputStream(outputPath), "UTF-8"), maxBufferSize) being maxBufferSize = Runtime.getRuntime().freeMemory() / chunks.

When you consider the various object overheads, (maxBufferSize + overheads) * chunks is probably a bit greater than the amount of free memory.

In general, it is a bad idea to run with a Java heap close to full. Even if you don't run out of space entirely, you can find that running close to full will trigger many (too many) garbage collections.

In this case, you don't really get a lot of benefit from having really large I/O buffers. Buffers in the range of 8KB to 64KB should be fine ... is my gut feeling. See also Peter Lawrey's comment!

Stephen C
  • 632,615
  • 86
  • 730
  • 1,096
  • Buffers greater than 32 KB can be marginally slower. I suspect this is due to the size of the L1 cpu cache, but I have found it varies as to what is the optimal size. – Peter Lawrey Jan 10 '16 at 18:53
0

You can try using System.gc(), but its not the most useful. You could try giving the program more memory.

Bob Smith
  • 48
  • 7