2

I am using org.apache.http.HttpClient to get a response from an URL, and convert the HttpEntity response to a string using EntityUtils.toString().

However, when the data returned is huge, sometimes it throws an OutOfMemory error. This makes sense, as the heap space can't hold the entire response string in memory.

My question is how can I convert the HttpEntity response from HttpClient to a string and decrease the chances of these OutOfMemoryErrors (assuming I can't increase the heap size)?

I'd prefer the the alternative way to be as efficient as EntityUtils.toString() or close to as efficient.

The exception:

java.lang.OutOfMemoryError: Java heap space
at org.apache.http.util.CharArrayBuffer.<init>(CharArrayBuffer.java:59)
at org.apache.http.util.EntityUtils.toString(EntityUtils.java:230)
at org.apache.http.util.EntityUtils.toString(EntityUtils.java:273)

This is my code:

import org.apache.http.util.EntityUtils;
import org.apache.http.client.*;

url = "http://test.com/test"

HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet(url);

HttpResponse response = client.execute(get);
HttpEntity entity = response.getEntity();
String result = EntityUtils.toString(entity);  //alternative???
Henley Chiu
  • 17,838
  • 29
  • 110
  • 187
  • Don't convert to a `String` would be the logic conclusion imho. What do you need to do with the result? – M. Deinum Mar 19 '14 at 15:20
  • @M.Deinum Unfortunately this is an API that is returning the string back to clients. So I can't change the return format. – Henley Chiu Mar 19 '14 at 15:21
  • 3
    What are you planning on doing with the `String`? Presumably printing it to somewhere (the console? a file? a logger?). In all of these cases you can stream the contents rather than read the whole lot into memory. – Boris the Spider Mar 19 '14 at 15:21
  • exactly how large is the string response? – matt b Mar 19 '14 at 15:21
  • Stream it directly to the client instead of a String. Use `entity.writeTo` instead of converting it to a String. – M. Deinum Mar 19 '14 at 15:22
  • If the API needs to return a `String`, and the `String` does not fit in the heap, I guess you have to increase the heap size, or change your API (To return a `Stream` clients can consume). – Nader Ghanbari Mar 19 '14 at 15:39

1 Answers1

0

The HttpEntity has methods like getContentLength() and getContent() which is an Input Stream.

Working with Streams will decrease the likelihood of heap space running out.

However, you will have to make your logic handle the response if a different format. Here is a helpful post dealing with Input Streams: Reading a plain text file in Java

Community
  • 1
  • 1
Scott Izu
  • 1,901
  • 21
  • 11