2

I'm trying to connect to a web page through a HttpURLConnection but it's not responding the same as a regular browser (firefox, chrome). I'm getting an error 500.

With the same code I can get "anyother" page (google, for example). My code is posted below, but I'm pretty sure it is ok.

Using "Live HTTP Headers" firefox addon I sent the same request and received a valid response.

//...
String urlConsulta = "myURL";
URL url = new URL(urlConsulta);
HttpURLConnection uConsulta = (HttpURLConnection)consulta.openConnection();
uConsulta.setDoOutput(true);
uConsulta.setRequestMethod("POST");
uConsulta.addRequestProperty("Content-Type", "application/x-www-form-urlencoded");
uConsulta.addRequestProperty("Host", "host");
//...
user470610
  • 21
  • 1
  • 2
  • 1
    @user470610: I've crawled webpages using Java and there is **one** thing I can tell you: you **MUST** fake your user agent pretending to be some very common browser or you'll have a lot of issues. Really a lot. It is amazing the number of webservers out there which are configured (either out of malice or out of incompetence) in a way that will prevent you from accessing them from Java until you fake your user agent. – SyntaxT3rr0r Oct 09 '10 at 08:25
  • @SyntaxT3rr0r: Interesting. As a professional programmer I must assert that HttpURLConnection is not a mature class if that were the case. Do you have access to any good resources which can be used to fake a user agent or are those illegal due to child robots? Seems like a slippery slope. I wonder if perhaps the class is intended to be used in conjunction with other objects... – Jesse Ivy Jan 03 '18 at 18:34

1 Answers1

2

This can be caused by everything. The server may rely on the presence of certain request parameters. The server may rely on a valid session. The server may sniff the user agent. Etc..etc.. Read the server logs for the cause of the error. Or if you don't have access to server logs, read the body of the error response by HttpURLConnection#getErrorStream(), it may contain the error details.

See also:

Community
  • 1
  • 1
BalusC
  • 992,635
  • 352
  • 3,478
  • 3,452
  • Thanks for the quick answer. Sorry about the lack of information, I tried to put just the essential and might have forgotten other important things. There is no need of valid session, I've tried several different user-agents. I am passing all needed params. Actually you can type exactly the same web address used as "myURL" in the small piece of code I posted in any regular browser and get a valid answer. – user470610 Oct 08 '10 at 20:01
  • The actual answer is in the server logs and/or the error response body. Go read it. If the `500` is coming from a Servlet (you tagged `servlets`), then it simply means that it has thrown an `Exception`. In other words, it's possibly a plain vanilla bug in the Servlet code. If you have full control over it, read the logs and fix it accordingly. – BalusC Oct 08 '10 at 20:08
  • As in my above comment: some documentation regarding resources to handle "everything", including user agent, would improve this answer. I'm going to hit the dock of the bay. – Jesse Ivy Jan 03 '18 at 18:38