I have an application that I need to read some open data from a (Socrata) dataset using their API. Since there are many records on that dataset, I am looping it through using offsets inside a while
//initialize variables
//...
do{
System.out.println("1");
BufferedReader in = new BufferedReader(new InputStreamReader(new URL(url+offset).openStream()));
String reply=IOUtils.toString( in );
//read the page to a String and convert to JSON
JSONObject obj = new JSONObject(reply);
//close connection
in.close();
JSONArray results=obj.getJSONArray("results");
//do something here
offset+=100;
}while(results.length()!=0&&offset<=endOffset);
This seems to be working fine for a few iterations(roughly 20-50), but then for no reason and not always on the same offset, it just hangs after the
String reply=IOUtils.toString( in );
for ever.
I tried to use guava as this post suggested with no difference
and this
String reply = CharStreams.toString( new InputStreamReader( new URL(runHost+offset).openStream(), "UTF-8" ) );
from here which seems to be doing a better job but again hanged after around 70 iterations.
Is there at least a way to stop the loop if there is a long delay and continue with the rest of the program?( as a worst case scenario)
EDIT
regarding @Mad Matts commend, result is the JSONArray converted from the JSONObject I get from the API.(code updated)
I use && at the condition since I need the loop to repeat if the array is not empty(results.length()!=0) and the offset is smaller than the endOffset. That last part I used it just to avoid the hanging of the procedure. I can at least run it for 10 offsets at a time, get my results and run it again for the next 10 offsets and so on.
