How to "browse" a URL connection in Java?

Question

I searched and haven't find a solution for my problem, and I'm not an expert in Java, so maybe the solution is simpler that I think.

I need to get some data from an authenticaded https page, What I need is to get the items 1,2...X So far is working, but not as I would like.

I have something similar to this:,

public String getItems(String itsItem) throws  Exception
{
    itsURL = new URL("https://some.site.com/search/servlet/servletFindView?item=" + itsItem);
    URLConnection uc = itsURL.openConnection();
    uc.setRequestProperty ("Authorization", itsBasicAuth);

    InputStream in = uc.getInputStream();
    InputStreamReader isr = new InputStreamReader(in);
    // I do stuff here with the imputStreamReader and get the data I need
    .
    .
    .
 }

 String item1 = getItems("1");
 String item2 = getItems("2");
 String item3 = getItems("3");
 ...
 String itemX = getItems("X");

What happens here is that for every Item I need is required a new connection, and our Site only allows 10 "new connections", then I need to wait like 30 minutes for more new connections.

When Im doing in the browser, I just login and serach the items I need without the need of "re authenticate or reconnect"

My question is, if there is some other apporach I should take, like for example Create a new connection and then "browse" in that same connection for the several items I need? That way I wont have the limit of 10 connections.

Like for example connect and authenticate to:

some.site.com/search/servlet/servletFindView?item=

and then just browse to:

some.site.com/search/servlet/servletFindView?item=1

some.site.com/search/servlet/servletFindView?item=2

. . .

some.site.com/search/servlet/servletFindView?item=X

Note that they are not items in the same page, the items are in different pages, page1, page2, page3 of the same site.

Similar to log into Gmail, and then browse mails 1,2,3....X

Any ideas, examples? Thanks in advance

If you know you are going to pull a bunch of items down one after another, why on earth do you open a new connection for each one instead of just pulling the whole page down once and then parsing it for each piece you need? — azurefrog, Jan 05 '15 at 21:42
Well those items are not in the same page. They are in different pages, in page site.com/item1 , site.com/item2, site.com/item3. The solution I think is just to authenticate in site.com and the just to browse the different pages, site1, site2, site3 etc — Marcopolo Ramos, Jan 05 '15 at 21:43
http://stackoverflow.com/questions/2793150/using-java-net-urlconnection-to-fire-and-handle-http-requests I think what you want here is to just manage your auth cookie better? Check the "Maintaining the session" section a bit. I'm no expert on this though, just thought that guide might help a bit — RHok, Jan 05 '15 at 22:04

score 0 · Answer 1 · edited May 23 '17 at 10:33

0

Yes HTTPURLConnection supports persistent connections. Read http://docs.oracle.com/javase/7/docs/technotes/guides/net/http-keepalive.html

Also read Persistent HttpURLConnection in Java and TCP connection is not reused for HTTP requests with HttpURLConnection

edited May 23 '17 at 10:33

Community

1
1

answered Jan 05 '15 at 22:05

Jean-Baptiste Yunès

30,872
2
40
66

How to "browse" a URL connection in Java?

1 Answers1