HAR incomplete when using BrowserMob Proxy in an Extension for Arquillian Drone

Question

I wrote an extension for Arquillian-Drone that is supposed to record the HTTP-Requests of the Test and create a HAR-File for each method. The HAR Files are created ( well most of the time ... it seems to be a bit unstable at the moment), but they do not contain the URLs of the HTTP Requests

I have found another Question here on stackoverflow, that describes the same result, but the answer there was not a solution in my case.

(Ticket: BrowserMob Proxy + Selenium: Not receiving any HTTP responses )

Instantiator:

@Override
public FirefoxDriver createInstance(WebDriverConfiguration arg0) {
    server.setTrustAllServers(true);
    server.setHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
    server.start();

    System.err.println("BrowserMob Proxy running on port: " + server.getPort());

    seleniumProxy = ClientUtil.createSeleniumProxy(server);

    try {
        hostIp = Inet4Address.getLocalHost().getHostAddress();
        seleniumProxy.setHttpProxy(hostIp + ":" + server.getPort());
        seleniumProxy.setSslProxy(hostIp + ":" + server.getPort());
    } catch (UnknownHostException e1) {
        e1.printStackTrace();
        System.err.println("invalid Host Address");
    }

    options.setCapability(CapabilityType.PROXY, seleniumProxy);
    options.setAcceptInsecureCerts(true);

    geckoService = new GeckoDriverService.Builder()
            .usingDriverExecutable(new File("C:/Program Files/GeckoDriver/geckodriver.exe")).usingAnyFreePort()
            .build();

    try {
        geckoService.start();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }


    return new FirefoxDriver(geckoService, options);

}

Observers:

public void prepareHAR(@Observes EventContext<Test> context) throws IOException {
    harFileName = "arquilliantest" + System.currentTimeMillis() + context.getEvent().getTestMethod().getName();
    server.newHar(harFileName);
    if (server.getHar() != null) {
        System.err.print(harFileName + " is prepared");
    } else {
        throw new RuntimeException("HAR was not created!");
    }
    context.proceed();

}

public void writeHAR(@Observes EventContext<After> context) throws IOException {
    Har har = server.getHar();
    String pfad = System.getProperty("user.dir") + File.separator + harFileName + ".har";
    if (!server.getHar().getLog().getEntries().isEmpty()) {
        File harFile = new File(pfad);
        har.writeTo(harFile);
        System.err.print(pfad + " is saved");

        List<HarEntry> entries = har.getLog().getEntries();
        for (HarEntry entry : entries) {
            System.err.println("Request URL: " + entry.getRequest().getUrl());
            System.err.println("Entry response status: " + entry.getResponse().getStatus());
            System.err.println("Entry response text: " + entry.getResponse().getStatusText());

        }

    } else {
        throw new RuntimeException("HAR is empty!");
    }

    // server.newHar(harFileName);

    context.proceed();

}

public void closeServer(@Observes EventContext<AfterClass> context) throws IOException {
    server.endHar();
    server.stop();
    geckoService.stop();

    context.proceed();

}

Snipped of my HAR:

{"log":{"version":"1.2","creator":{"name":"BrowserMob Proxy","version":"2.1.5","comment":""},

"pages": [{"id":"arquilliantest1561033479096minimaltestAnlegenKrankheit", "startedDateTime":"2019-06-20T12:24:39.316Z", "title":"arquilliantest1561033479096minimaltestAnlegenKrankheit", "pageTimings":{"comment":""},"comment":""}],

"entries": [{"pageref":"arquilliantest1561033479096minimaltestAnlegenKrankheit", "startedDateTime":"2019-06-20T12:24:40.032Z",

"request": {"method":"POST", "url":"https://shavar.services.mozilla.com/downloads?client=navclient-auto- ffox&appver=67.0&pver=2.2", "httpVersion":"HTTP/1.1", "cookies":[], "headers":[],

so it appears that the urls are somehow replaced with the HAR-name (like in the other ticket)

I already use BrowserMob like it was suggested in the other ticket:

compile group: 'net.lightbody.bmp', name: 'browsermob-core', version: '2.1.5'

any ideas would be much appreciated!

edit: I use the following versions:

• Browsermob Core: 2.1.5
• Arquillian : 1.4.1-Final
• Drone: 2.5.1
• Selenium: 3.14.0
• Firefox 67.0
• GeckoDriver: 0.24.0

edit

I think i know the root of the problem: the tracking protection of Firefox. I tried to implement a Firefox Profile, that has the Preference for Tracking protection set to false, but somehow the changes won´t show in my browser. Is there anything that needs to be considered when changing preferences?

Answer 1

i have found a solution to my problem now: 1. the BrowserMob (or BrowserUp) ProxyServer needs to be initialized and started at an Event before the Drone Initializer creates the Driver (I used ManagerStarted) and 2. the Firefox Browser turns off the proxy for requests to the localhost (at least when it is in automation mode). the fix to this was adding this preference: options.addPreference("network.proxy.allow_hijacking_localhost", true);

hope this will help others :)

Answer 2

I would call newPage so you don't get the weird title. Requests are grouped into pages. The URL's are going to be in the entries section and will have the pageref you pass to newpage. It looks like you're not capturing the traffic yet, so that's another issue to debug.