Getting a JSON data and reading as a string is slow (compared to my browser)

Question

I have read many questions on SO about like this, this, this and this.

My problem is the following: I have json data that reside on a server. Tha data are cached and are requested as gzip, so that the whole size is about 110kb. If you try to retrieve this data with firefox (and count the time with firebug) it takes about 3 seconds to download and display.

The very same data take about 10 - 15 seconds (or even more sometimes) to get downloaded and converted to string on my android app. (Note that I am using a Nexus 10 tablet which is one of the fastest in temrs of processing of its kind).

See my code below:

List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);

// ... nameValuePairs omitted

HttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, 3500);
HttpConnectionParams.setSoTimeout(params, 14000);
HttpConnectionParams.setTcpNoDelay(params, false);

params.setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
params.setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, true);   

HttpClient httpclient = new DefaultHttpClient(params); 
HttpPost httppost = new HttpPost(mContext.getResources().getString(R.string.newsUrl));
httppost.addHeader("Accept-Encoding", "gzip");

try { 
    // Add your data 
    httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs, "UTF-8")); 
    Log.i(TAG, "NewsFetcher, Request started at: " + (int)(System.currentTimeMillis() / 1000));

    // Execute HTTP Post Request 
    HttpResponse response = httpclient.execute(httppost);

    Log.i(TAG, "NewsFetcher, Request ended at: " + (int)(System.currentTimeMillis() / 1000));

    if (response.getStatusLine().getStatusCode() == 200 || response.getStatusLine().getStatusCode() == 204 || response.getStatusLine().getStatusCode() == 304 ) {

        Header contentEncoding = response.getFirstHeader("Content-Encoding");

        String webAppResponse; 
        if (contentEncoding != null && contentEncoding.getValue().equalsIgnoreCase("gzip")) { 
            Log.i(TAG, "NewsFetcher, InputStream(gzip) received at : " + (int)(System.currentTimeMillis() / 1000)); 
            webAppResponse = IOUtils.toString(new GZIPInputStream(response.getEntity().getContent()), "UTF-8"); 
        } else { 
            Log.i(TAG, "NewsFetcher, InputStream(plain) received at : " + (int)(System.currentTimeMillis() / 1000)); 
            webAppResponse = IOUtils.toString(response.getEntity().getContent(), "UTF-8"); 
        }

        Log.i(TAG, "NewsFetcher, InputStream ended at : " + (int)(System.currentTimeMillis() / 1000));

        // try to detect any error 
        try { 
            JSONArray webAppResultJSON = new JSONArray(webAppResponse); 
            ArrayList<NewsItem> parsedItems = parseJSON(webAppResultJSON);

            return new FetchResult(UpdateStatus.FETCH_OK, parsedItems); 
        // ... exceptions omitted
        } catch (Exception e) { 
            Log.e(TAG, "General Exception 1"); 
            e.printStackTrace(); 
            return new FetchResult(UpdateStatus.FETCH_GENERAL_ERROR, null); 
        } 
    } else {
        return new FetchResult(UpdateStatus.FETCH_HTTP_ERROR, null); 
    }
// ... exceptions omitted
} catch (Exception e) { 
    Log.e(TAG, "General Exception 2");
    e.printStackTrace(); 
    return new FetchResult(UpdateStatus.FETCH_GENERAL_ERROR, null); 
}

And here are the results from logcat

06-28 10:43:11.486: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, Request started at: 1372405391
06-28 10:43:17.986: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, Request ended at: 1372405397
06-28 10:43:17.986: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, InputStream(gzip) received at : 1372405397
06-28 10:43:30.266: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, InputStream ended at : 1372405410

If you subtract 1372405391 from 1372405410 it is 19 seconds! Can you suggest any optimization on the code above ?

I have also tried to use HttpURLConnection instead of HttpPost but no luck...

Step #1 is to actually find out *which part is slow[est]*; a good start to any profiling session. Anyway, that speed difference doesn't seem terribly unsurprising given several factors involved. — user2246674, Jun 28 '13 at 07:53
@user2246674 Well, I cannot distinguish if the network part of the code is slow, or the conversion to String. I use Apache Commons `IOUtils` class so I suppose that conversion from InputStream to String is ok. I suppose that something is wrong with the http request. — antoniom, Jun 28 '13 at 07:55
For profiling tips, see http://stackoverflow.com/questions/3871311/how-can-i-profile-my-android-app , http://stackoverflow.com/questions/5824638/profiling-and-optimizing-a-game-android etc. also see http://stackoverflow.com/questions/12170552/parse-1-mb-json-on-android-very-slow?rq=1 (suggests using a different JSON deserializer) — user2246674, Jun 28 '13 at 07:56
Could it be that you have a slow internet connection on your device? (when you said you checked it on firefox, did you mean on your computer or your mobile device) — dors, Jun 28 '13 at 07:57
@dors I checked it out on firefox on my local computer. Both my tablet and computer use the same wifi connection — antoniom, Jun 28 '13 at 08:06

score 0 · Answer 1 · answered Jun 28 '13 at 08:27

0

It seems that

            webAppResponse = IOUtils.toString(new GZIPInputStream(response.getEntity().getContent()), "UTF-8");

takes most of the time (13 seconds).

I once wrote this method which works pretty well. maybe it will work better than IOUtils:

public String inputStreamToString (InputStream inputStream) throws IOException {

    StringBuilder builder = new StringBuilder();

    BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));

    String line;

    while ((line = reader.readLine()) != null) {
        builder.append(line);
    }

    return builder.toString();
}

Just pass your GZIPInputStream into it.

Hope it will help

answered Jun 28 '13 at 08:27

dors

5,442
8
40
60

I have tried it but the performance is pretty much the same. It seems that `IOUtils.toString` or your method `inputStreamToString` starts processing the InputStream while this stream keeps accepting data. So the delay is on the network, not the conversion method itself. – antoniom Jun 28 '13 at 08:33
That is not true. httpclient.execute(httppost); is blocking and will only continue after the httprequest is done. From what I see, it takes 6 seconds for the transaction to finish – dors Jun 28 '13 at 08:37
I monitored the network traffic of my app on DDMS. How you could explain the fact that the application was still receiving data, and the LogCat showed that it had reached on `IOUtils.toString` ? Maybe httpclient.execute(httppost) blocks until the response headers are received, not the response body. – antoniom Jun 28 '13 at 08:40
This makes no sense... if that was true, people would work with incomplete data and a lot of problems would occur – dors Jun 28 '13 at 08:43
Well my suspicions are right. `IOUtils` have nothing to do. It's all about network connection. You can use my code above, and try to download a 50MB file e.g. http://ipv4.download.thinkbroadband.com/50MB.zip As soon as the server replies back the code steps to `IOUtils.toString`. It does not block and wait for the 50MB file to be downloaded. – antoniom Jun 28 '13 at 09:02
Well... I guess this is my bad :) it seems weird it would take so long to download 110kb. Can you run the code you ran on your computer's firefox on a browser on your mobile device? – dors Jun 28 '13 at 09:24
By requesting the same URL at android's google chrome i get slower responses than my firefox, but not as slow as the one on my application. It takes about 5-10 seconds to load and display the results. – antoniom Jun 28 '13 at 09:38

Getting a JSON data and reading as a string is slow (compared to my browser)

1 Answers1