HttpURLConnection does not read the whole response

Question

I use HttpURLConnection to do HTTP POST but I dont always get back the full response. I wanted to debug the problem, but when I step through each line it worked. I thought it must be a timing issue so I added Thread.sleep and it really made my code work, but this is only a temporary workaround. I wonder why is this happening and how to solve. Here is my code:

public static InputStream doPOST(String input, String inputMimeType, String url, Map<String, String> httpHeaders, String expectedMimeType) throws MalformedURLException, IOException {

    URL u = new URL(url);
    URLConnection c = u.openConnection();
    InputStream in = null;
    String mediaType = null;
    if (c instanceof HttpURLConnection) {

        //c.setConnectTimeout(1000000);
        //c.setReadTimeout(1000000);

        HttpURLConnection h = (HttpURLConnection)c;
        h.setRequestMethod("POST");
        //h.setChunkedStreamingMode(-1);
        setAccept(h, expectedMimeType);
        h.setRequestProperty("Content-Type", inputMimeType);

        for(String key: httpHeaders.keySet()) {
            h.setRequestProperty(key, httpHeaders.get(key));

            if (logger.isDebugEnabled()) {
                logger.debug("Request property key : " + key + " / value : " + httpHeaders.get(key));
            }

        }

        h.setDoOutput(true);
        h.connect();

        OutputStream out = h.getOutputStream();

        out.write(input.getBytes());

        out.close();

        mediaType = h.getContentType();

        logger.debug(" ------------------ sleep ------------------ START");
        try {
            Thread.sleep(2000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        logger.debug(" ------------------ sleep ------------------ END");

        if (h.getResponseCode() < 400) {
            in = h.getInputStream();
        } else {
            in = h.getErrorStream();
        }
    }
    return in;

}

later I do the following to read the input stream

        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        while (is.available() > 0) {
            bos.write(is.read());
        }
        is.close();

        //is.read(bytes);
        if (logger.isDebugEnabled()) {
            logger.debug(" Response lenght is : " + is.available());
            //logger.debug("RAW response is " + new String(bytes));
            logger.debug("RAW response is " + new String(bos.toByteArray()));
        }

It genearates the following HTTP headers

POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39",oauth_nonce="YnDb5eepuLm%2Fbs",oauth_signature="dbN%2FWeWs2G00mk%2BX6uIi3thJxlM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276524919", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: keep-alive
Content-Length: 1107

In other posts it was suggested to turn off keep-alive by using the

http.keepAlive=false

system property, I tried that and the headers changed to

POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39", oauth_nonce="Eaiezrj6X4Ttt0", oauth_signature="ND9fAdZMqbYPR2j%2FXUCZmI90rSI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276526608", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: close
Content-Length: 1107

the Connection header is "close" but I still cannot read the whole response. Any idea what do I do wrong?

There's ambiguity in your question. *Which* response are you talking about? With that whole code you are actually not reading a response, but creating a request. You're reading its response using `h.getInputStream()` at the bottom, but you're actually ignoring it and/or not showing how you process it. — BalusC, Jun 14 '10 at 15:03

score 14 · Accepted Answer · answered Jun 14 '10 at 15:41

14

I think your problem is in this line:

while (is.available() > 0) {

According to the javadoc, available does not block and wait until all data is available, so you might get the first packet and then it will return false. The proper way to read from an InputStream is like this:

int len;
byte[] buffer = new byte[4096];
while (-1 != (len = in.read(buffer))) {
  bos.write(buffer, 0, len);
}

Read will return -1 when there nothing left in the inputstream or the connection is closed, and it will block and wait for the network while doing so. Reading arrays is also much more performant than using single bytes.

answered Jun 14 '10 at 15:41

Jörn Horstmann

31,936
11
65
111

Hi Jörn This was the problem I was sure something is worng on the HTTP transport level so read the whole JavaDoc of HttpURLConnection, and wasnt paying attention to InputStream Thanks! Peter – Peter Szanto Jun 14 '10 at 16:14
oh and forgot to mention the bug is not only in my code, but Sun's JAXB too. Originally I passed the InputStream directly to Object o = unmarshaller.unmarshal(bis); where unmarshaller is instanceof javax.xml.bind.Unmarshaller and it resulted the same problem. – Peter Szanto Jun 14 '10 at 16:20

score 0 · Answer 2 · answered Jun 14 '10 at 14:57

0

Maybe I missed it, but what's the datatype of "input" in your code? Something that's strange about InputStreams in general is that the read( ... ) methods tend to block until data is available, then return only that data. You'll actually need to keep reading from your InputStream and appending to a ByteArrayInputStream or some other structure until you explicitly force a EOFException.

answered Jun 14 '10 at 14:57

Curtis

3,720
1
16
25

1

It's likely a `String` which represents the querystring. Also see [How to use URLConnection](http://stackoverflow.com/questions/2793150/how-to-use-java-net-urlconnection-to-fire-and-handle-http-requests). – BalusC Jun 14 '10 at 15:06
Hi Curtis I edited my post to include the method signature explaining what input is and the code snipplet that reads the InputStream – Peter Szanto Jun 14 '10 at 15:17
'until you explicitly force a EOFException'. But you won't get one if you're calling read() or readLine(). You only get EOFException when calling readXXX() for any other X. The read() methods return -1 at EOS. The code sample in Jörn Horstmann's reply is the correct technique. – user207421 Jun 15 '10 at 07:21

score 0 · Answer 3 · answered Apr 13 '11 at 12:48

If you are reading the whole Message at once you can compare the isr.available() to the expected content lenght. This is how I did it:

public byte[] readData(HttpURLConnection conn)
        throws IOException, InterruptedException {
    String _connlen = conn.getHeaderField("Content-Length");
    int connlen = Integer.parseInt(_connlen);
    InputStream isr = null;
    byte[] bytes = new byte[connlen];

    try {
        isr = conn.getInputStream();

        //security count that it doesn't begin to hang
        int maxcounter = 0;
        //wait till all data is avalibal, max 5sec
        while((isr.available() != connlen) && (maxcounter  < 5000)){
            Thread.sleep(1);
            maxcounter++;
        }
        //Throw if not all data could be read
        if(maxcounter >= 5000)
            throw new IllegalAccessError(); 

        //read the data         
        if(isr.read(bytes, 0, connlen) < 0)
            throw new IllegalAccessError();     


    } finally {
        if (isr != null)
            isr.close();
    }

    return bytes;
}

HttpURLConnection does not read the whole response

3 Answers3

Linked