Can't send large files over socket in Java

Question

I got working server and client applications, they work perfect while sending small files, but when I try to send for example movie file that is 700mb over socket it gives me

Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space

I searched the internet and found some tutorials on sending large files, but couldn't quite understand them, but I think my porblem is in writing file.

This is the code that server uses to write my file:

output = new FileOutputStream(directory + "/" + fileName);
            long size = clientData.readLong();
            byte[] buffer = new byte[1024];

            while (size > 0 && (bytesRead = clientData.read(buffer, 0, (int) Math.min(buffer.length, size))) != -1) {
                output.write(buffer, 0, bytesRead);
                size -= bytesRead;
            }
            output.close();

And here is the code that my client uses to send a file:

byte[] fileLength = new byte[(int) file.length()];  

        FileInputStream fis = new FileInputStream(file);  
        BufferedInputStream bis = new BufferedInputStream(fis);

        DataInputStream dis = new DataInputStream(bis);     
        dis.readFully(fileLength, 0, fileLength.length);  

        OutputStream os = socket.getOutputStream();  

        //Sending size of file.
        DataOutputStream dos = new DataOutputStream(os);   
        dos.writeLong(fileLength.length);
        dos.write(fileLength, 0, fileLength.length);     
        dos.flush();  

        socket.close();

The best thing you can do is to run your program like this http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss Then you can analyse it using jvisualvm for example. — biziclop, Jul 12 '12 at 10:01
I think your problem is that you're trying to find X MB of content into a Y MB memory, where X > Y. It doesn't matter how you write that file if that's true. That's what the exception is telling you. — duffymo, Jul 12 '12 at 10:02
you're trying to eat a big watermelon in one bite, so you're dead. try to read a small piece of file and send it out and repeat the operation. — LiuYan 刘研, Jul 12 '12 at 10:21
possible duplicate of [Sending large files over socket](http://stackoverflow.com/questions/11452373/sending-large-files-over-socket) — Charles, Jul 12 '12 at 17:13

score 7 · Accepted Answer · answered Jul 12 '12 at 10:02

7

It gives you OutOfMemoryError because you are trying to read the entire file into memory before sending it. This is 100% completely and utterly unnecessary. Just read and write chunks, much as you are doing in the receiving code.

answered Jul 12 '12 at 10:02

user207421

289,834
37
266
440

can you edit my code? Because readFully() is only way I know how to do it. Thanks – Rohit Malish Jul 12 '12 at 10:09
@RohitMalish It *isn't* 'the only way [you] know how to do it'. Have a look at your own receiving code. That's how to do it. You've already written it. Just get rid of the `readLong()` part and everything that flows from it: read until EOF. – user207421 Jul 12 '12 at 10:16
Well I once tried that method but it gave me differend kind of errors like file was empty when i recieved it etc – Rohit Malish Jul 12 '12 at 10:26
@RohitMalish I can only suggest you try it again, now that you have some comprehension of what you're doing. – user207421 Jul 12 '12 at 10:33

Tudor · Answer 2 · 2012-07-12T11:41:36.383

4

The problem is that you are trying to load the entire file at once into memory (via readFully), which exceeds your heap space (which by default is 256mb I think).

You have two options:

Good option: load the file in chunks (e.g. 1024 bytes at a time) and send it like that. Takes a bit more effort to implement.
Bad option: increase your heap space via -Xmx option. Not recommended, I mostly mentioned this just in case you will need a bit more heap space for a program, but in this case it's a really bad idea.

For option one you can try something like:

DataInputStream in = new DataInputStream(new FileInputStream("file.txt"));
byte[] arr = new byte[1024];
try {
    int len = 0;
    while((len = in.read(arr)) != -1)
    {
        // send the first len bytes of arr over socket.
    } 
} catch(IOException ex) {
    ex.printStackTrace();
}

edited Jul 12 '12 at 11:41

answered Jul 12 '12 at 10:03

Tudor

58,972
12
89
138

I'm not really into java, but isn't 1024 too small for a 700MB file? It will create a lot of calls.. – Karoly Horvath Jul 12 '12 at 10:06
@Karoly Horvath: It was just an example. I used 1024 because that's the value he was using on the receiver side. – Tudor Jul 12 '12 at 10:07
Now it doesnt complain when sending large files but for example now I recieve small files empty, and larger files for example video doesnt play even though it does show that is is 700+ MB, any idea why? – Rohit Malish Jul 12 '12 at 12:24
@Rohit Malish: Hmm... this seems strange. Maybe you should open another question with the code you have now, because you are not likely to attract more visitors on this question since it's about a different problem. – Tudor Jul 12 '12 at 12:40

score 2 · Answer 3 · answered Jul 12 '12 at 10:10

OutOfMemoryError is thrown when your program reaches the heap size limit. You loads entire file to your RAM. The default heap size is 1/4 of physical memory size or 1GB, so your program reaches the limit (you probably have got 2GB RAM, so the heap size is 512MB).

You should read your file in chunks (e.g. 10MB), so you won't reach the heap size limit, and you can resend chunks in the case of some error, instead of resending whole file. You can even read chunks in different thread, so when you're sending 1 chunk, you can load the second chunk, and when you'll sent the first one, you can start sending the second immediately.

So your heap size is 1GB. You could reach the limit because you don't only use memory for the file, but for other things, too (such as JVM, GUI, etc.) — m4tx, Jul 12 '12 at 10:12

score 1 · Answer 4 · answered Jul 12 '12 at 10:04

The problem is that in your client you create a byte array for the entire file.

byte[] fileLength = new byte[(int) file.length()];  //potentially huge buffer allocated here

You should do the same thing you do on the server side and read the file chunk by chunk into a fixed size buffer.

score 0 · Answer 5 · answered Jul 12 '12 at 10:05

Just like you're using a buffer on the server side the send the data, use a buffer in your applet/client to read it. If you're transferring a large file, you will clearly run out of memory when you use readFully(). This method is only useful for cases where you know your data will be really, really small.

Can't send large files over socket in Java

5 Answers5

Linked