How to write large result into a file in Java

Question

I want to generate some data and write them into a file. The size of these data is larger than the capacity of memory. Which way should I use to write these data?

Now I am using this way:

BufferedWriter output = new BufferedWriter(new OutputStreamWriter(
                new FileOutputStream(fileName), "utf-8"
        ));

I don't know if it works when the data is very large, and if there is other way with better performance.

Also, the encoding must be "utf-8".

How is the data being generated? All at once, or in smaller chunks? (If in smaller chunks, then your current solution is fine. ) — Louis Wasserman, May 17 '14 at 20:03
Or you can use FileWriter directly, as mentioned in http://stackoverflow.com/questions/1062113/fastest-way-to-write-huge-data-in-text-file-java — conFusl, May 17 '14 at 20:09
Perfectly allright. Tip: instead of a String `"utf-8"` you could use a `Charset` constant `StandardCharsets.UTF_8`. — Joop Eggen, May 17 '14 at 21:08

score 1 · Answer 1 · answered May 17 '14 at 20:20

1

You can handle very large files using this method

Open input stream
Open output stream
create byte buffer
while (read stuff into byte buffer) {
    write byte buffer to output stream
}

That's how you do this. You use a buffer (say 8k)

 byte[] b = new byte[8192];

Read into that, write it out. Repeat until you run out of things to read in.

answered May 17 '14 at 20:20

MCHAppy

964
8
15

Calculate the buffer size after calculating VM heap size. You may want to store constant values on a file e.g. XML and then transfer via FIleChannels which is 250% faster – user3535778 May 17 '14 at 20:23
Ummmm ...i didn't understand you well , can you post this solution as an answer ? it appears to be useful for me – MCHAppy May 17 '14 at 20:26

score 0 · Accepted Answer · edited May 23 '17 at 10:24

When you start the JVM you can either specify heap (RAM) size via the -Xmx and -Xms flags or just run it by java -jar *.jar and have it use default heap size see How is the default java heap size determined? You can use launch4j to make a native excecutable for most os' or you can make the jar automatically restart itself with more RAM (All these need to be done with caution in order to avoid stupidly using RAM that do not exists).

If you have constant values , store them somewhere before you make the file programmatically (database/files/blah).

I believe most of the people would do this

Open input stream Open output stream create byte buffer while (read stuff into byte buffer) { write byte buffer to output stream }

PREFER FILECHANNELS THOUGH , they are better and faster. Perform some googling by yourself , I am sure you will solve your problem. Be sure to flush any buffers before you use them and close streams etc... at the end!

P.S. Strings do not take THAT much to be written to a file. IDK if one second matters THAT MUCH!

How to write large result into a file in Java

2 Answers2