I need to capture all hosts a webpage requests resources from to during its loading process. Currently I am achieving this using Selenium with PhantomJSDriver and Browsermob Proxy to generate a har file. After generating the file, I can parse all HTTP-Requests the page did during its loading process, from the har-logs:
public static void main(String[] args) throws IOException {
// BrowserMobProxy
BrowserMobProxy server = new BrowserMobProxyServer();
server.start(0);
server.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());
server.enableHarCaptureTypes(CaptureType.RESPONSE_COOKIES, CaptureType.REQUEST_COOKIES,
CaptureType.REQUEST_HEADERS, CaptureType.RESPONSE_HEADERS, CaptureType.REQUEST_CONTENT,
CaptureType.RESPONSE_CONTENT);
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(server);
// PHANTOMJS_CLI_ARGS
ArrayList<String> cliArgsCap = new ArrayList<>();
cliArgsCap.add("--proxy=localhost:" + server.getPort());
cliArgsCap.add("--ignore-ssl-errors=yes");
// DesiredCapabilities
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,
"...");
//connect to website using webdriver
Set<String> hosts = new HashSet<>();
WebDriver driver = new PhantomJSDriver(capabilities);
//generate har-file
server.newHar();
String site = "...";
driver.get(site);
//parse information from har-file
Har har = server.getHar();
for (HarEntry entry : har.getLog().getEntries()) {
if (!entry.getRequest().getUrl().contains(new URL(site).getHost())) {
for (HarNameValuePair h : entry.getRequest().getHeaders()) {
if(h.getName().equals("Host"))
{
if(!hosts.contains(h.getValue()))
{
hosts.add(h.getValue());
}
}
}
}
}
server.stop();
driver.close();
It bothers me that using selenium webdriver is very slow and memory intense. Unfortunally I'm not very experienced with Selenium and BMP (or webdevelopment in general). Is there maybe a way to generate the har-file with BMP without using Selenium? Or maybe a better approach to get the information I need? Thanks in advance.