1

I need to capture all hosts a webpage requests resources from to during its loading process. Currently I am achieving this using Selenium with PhantomJSDriver and Browsermob Proxy to generate a har file. After generating the file, I can parse all HTTP-Requests the page did during its loading process, from the har-logs:

    public static void main(String[] args) throws IOException {

    // BrowserMobProxy
    BrowserMobProxy server = new BrowserMobProxyServer();
    server.start(0);
    server.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());
    server.enableHarCaptureTypes(CaptureType.RESPONSE_COOKIES, CaptureType.REQUEST_COOKIES,
            CaptureType.REQUEST_HEADERS, CaptureType.RESPONSE_HEADERS, CaptureType.REQUEST_CONTENT,
            CaptureType.RESPONSE_CONTENT);

    Proxy seleniumProxy = ClientUtil.createSeleniumProxy(server);

    // PHANTOMJS_CLI_ARGS
    ArrayList<String> cliArgsCap = new ArrayList<>();
    cliArgsCap.add("--proxy=localhost:" + server.getPort());
    cliArgsCap.add("--ignore-ssl-errors=yes");

    // DesiredCapabilities
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
    capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
    capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
    capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
    capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,
            "...");

    //connect to website using webdriver
    Set<String> hosts = new HashSet<>();
    WebDriver driver = new PhantomJSDriver(capabilities);

    //generate har-file
    server.newHar();
    String site = "...";
    driver.get(site);

    //parse information from har-file
    Har har = server.getHar();
    for (HarEntry entry : har.getLog().getEntries()) {
        if (!entry.getRequest().getUrl().contains(new URL(site).getHost())) {
            for (HarNameValuePair h : entry.getRequest().getHeaders()) {
                if(h.getName().equals("Host"))
                {
                    if(!hosts.contains(h.getValue()))
                    {
                        hosts.add(h.getValue());
                    }
                }
            }

        }

    }
    server.stop();
    driver.close();

It bothers me that using selenium webdriver is very slow and memory intense. Unfortunally I'm not very experienced with Selenium and BMP (or webdevelopment in general). Is there maybe a way to generate the har-file with BMP without using Selenium? Or maybe a better approach to get the information I need? Thanks in advance.

P.Raber
  • 68
  • 5

0 Answers0