Writing large files using BufferedOutputStream

Question

I am trying read and write large files (larger than 100 MBs) using BufferedInputStream & BufferedOutputStream. I am getting Heap Memory issue & OOM exception.
The code looks like :

BufferedInputStream buffIn = new BufferedInputStream(iStream);
/** iStream is the InputStream object  **/

BufferedOutputStream buffOut=new BufferedOutputStream(new FileOutputStream(file));
byte []arr = new byte [1024 * 1024];
int available  = -1;
while((available = buffIn.read(arr)) > 0) {   
    buffOut.write(arr, 0, available); 
}      
buffOut.flush();
buffOut.close();

My question is when we use the BufferedOutputStreeam is it holding the memory till the full file is written out ?
What is the best way to write large files using BufferedOutputStream?

it's buffering until you flush it. flush it in the while loop. — jcomeau_ictx, Feb 24 '12 at 04:24
@jcomeau_ictx - no, BufferedOutputStream does not buffer until you flush it, it has a fixed internal buffer size. — jtahlborn, Feb 24 '12 at 04:46
@HunterMcMillen Flushing inside the loop completely negates the point of using a `BufferedOutputStream` at all, and it won't do anything to the memory usage whatsoever. — user207421, Feb 24 '12 at 09:15
possible duplicate of [What is an OutOfMemoryError and how do I debug and fix it](http://stackoverflow.com/questions/24510188/what-is-an-outofmemoryerror-and-how-do-i-debug-and-fix-it) — Raedwald, Sep 23 '14 at 12:37
It’s unlikely that this single megabyte causes an OOM. Besides that, both, the `BufferedInputStream` and `BufferedOutputStream` are unlikely to have any effect when you use an array that large. The best thing, they do, is going out of the way when the array is that large, to avoid performance degradation. The 8kiB of their unused own byte array is unlikely to hurt either. — Holger, Feb 03 '17 at 17:18

score 5 · Answer 1 · answered Feb 24 '12 at 04:40

5

there is nothing wrong with the code you have provided. your memory issues must lie elsewhere. the buffered streams have a fixed memory usage limit.

the easiest way to determine what has caused an OOME, of course, is to have the OOME generate a heap dump and then examine that heap dump in a memory profiler.

answered Feb 24 '12 at 04:40

jtahlborn

50,774
5
71
112

This is correct. But, there is a problem with the code, buffOut.write() might write fewer bytes than what you ask it to write, in which case, you would need to continue writing until you've written all of the available bytes. – Bill Feb 24 '12 at 04:47
1

@Bill - no, that is not true. `write()` always writes all the bytes (it has no return value). – jtahlborn Feb 24 '12 at 05:07
Are you running this code in many threads at once? It's pretty hard to see that this is the source of your out of memory error. – Paul Jowett Feb 24 '12 at 06:34

Writing large files using BufferedOutputStream

1 Answers1