I want to use Jsoup to screen scrape contents of a website. But I have to login first into the site. On browsing to the main page I get a dialog for username and password. Since it is not a form, I am getting the "not authorized" page as the response in in Jsoup. I tried to look for the url using firebug but I guess the dialog is appearing before the other page components are loaded. Hence I don't know what are the parameters I need to pass for the username and password fields nor do I get to know the service where I need to post.
This is a C# based website. I have seen this authentication mechanism in several Sharepoint sites. How should I go ahead with this kind of login mechanism ?