0

I am using BrowserMob-Proxy inside a Selenium test suite. I would like to change the Referer for a few tests. I have added requestInterceptor from the 2.0 docs into our MyProxy class and while it does not generate an error the Referer is not changed.

For now, I am trying to get the requestInterceptor to work in the MyProxy class where the proxy is created. In the end, I would like to be able to specify the Referer in each test.

If anyone has suggestions on getting the requestInterceptor to work please let me know. Here is the MyProxy class. Please let me know if other code examples would be helpful to troubleshoot this.

import org.openqa.selenium.Proxy;

import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.proxy.ProxyServer;
import net.lightbody.bmp.proxy.http.BrowserMobHttpRequest;
import net.lightbody.bmp.proxy.http.RequestInterceptor;

public class MyProxy {
    private ProxyServer proxy;
    private boolean initialized;

    public Har endCapture() throws Exception {
        Thread.sleep(15000);
        return this.proxy.getHar();
    }

    public Proxy getSeleniumProxy() {
        return this.proxy.seleniumProxy();
    }

    public boolean isInitialized() throws Exception {
        return this.initialized;
    }

    public void start() throws Exception {
        int proxyPort = Integer.parseInt(System.getProperty("proxyPort"));
        this.proxy = new ProxyServer(proxyPort);
        this.proxy.start();
        this.proxy.setCaptureHeaders(true);
        this.proxy.setCaptureContent(true);

        this.proxy.addRequestInterceptor(new RequestInterceptor() {
            @Override
            public void process(BrowserMobHttpRequest request, Har har) {
                request.getMethod().removeHeaders("Referer");
                request.getMethod().addHeader("Referer", "http://www.google.com");
            }
        });

        this.initialized = true;
    }

    public void startCapture() throws Exception{
        this.proxy.newHar("MyHar");
    }

    public void stop() throws Exception {
        this.proxy.stop();
        this.initialized = false;
    }
}
analyticsPierce
  • 2,677
  • 8
  • 44
  • 77
  • Please note that I am not able to upgrade to BrowserMob-Proxy 2.1.0 beta 5 as I am getting a conflict that I can't resolve. Could I only use the littleProxy library? – analyticsPierce Apr 05 '16 at 22:38
  • I have much the same code running OK, so there's no need to do any unnecessary upgrades. Have you confirmed `process()` is getting called? If so, you could log the request object (the result of getMethod() is actually an HttpClient obj, not a Browsermob one) after the add to check its state. Syntax looks fine. Finally, how/where are you checking Referer: could that be the problem? – Andrew Regan Apr 05 '16 at 23:51
  • Also you could replace remove/add with simply 'set', as I don't think multiple Referer headers makes any sense. – Andrew Regan Apr 06 '16 at 00:12
  • @AndrewRegan Yes, `process()` is getting called. However, the request objects (after `addHeader()` or `setHeader()` ) shows a localhost Referer, not the google.com Referer I specified. – analyticsPierce Apr 06 '16 at 04:47
  • @AndrewRegan I am checking the Referer in the request in Chrome dev tools while the test is running and also checking the request object as you suggested. – analyticsPierce Apr 06 '16 at 04:51
  • Bear in mind that `req.getMethod()` is a `org.apache.http.client.methods.HttpRequestBase`, so not BMP code. Also you could try running this test (beta 4) that tests setting UserAgent very much as you do: https://github.com/lightbody/browsermob-proxy/blob/browsermob-proxy-2.1.0-beta-4/browsermob-core/src/test/java/net/lightbody/bmp/proxy/MailingListIssuesTest.java#L70 – Andrew Regan Apr 06 '16 at 09:01
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/108423/discussion-between-analyticspierce-and-andrew-regan). – analyticsPierce Apr 06 '16 at 16:01

2 Answers2

1

I think the key here is how to test the newly-added header, which is tricky to do manually.

I chose as a test-site: http://headers.cloxy.net/request.php, which simply logs the names and values of all request headers. Having first set up my proxy, I arranged a screenshot to be written after page request completed.

I was able to determine that:

 @Override
 public void process(BrowserMobHttpRequest req, Har har) {
     req.getMethod().removeHeaders("Referer");
     req.getMethod().addHeader("Referer", "http://www.google.xyz");

     // Some extras
     req.getMethod().addHeader("Foo_" + System.currentTimeMillis(), "Bar_" + new java.util.Date());
     req.getMethod().setHeader("Lorem_" + System.currentTimeMillis(), "Ipsum_" + new java.util.Date());
 }

... successfully adds all the specified headers in both BrowserMob 2.0.0 and 2.1 beta 5. I've confirmed this for each version in Firefox (45), Chrome (49), and PhantomJS.

enter image description here

So, in short:

  • The OP's header-adding syntax is perfectly fine.
  • setHeader also works as expected.
  • BMP version numbers do not affect this (but by all means upgrade to 2.1 as/when it is released)
  • Browsers do not affect this
Andrew Regan
  • 4,887
  • 6
  • 35
  • 67
0

Review this issue and see if it describes your problem.

Suggestion is to move to the latest version of BrowserMobProxy which is 2.1.0-beta-5.

MikeJRamsey56
  • 2,534
  • 1
  • 16
  • 32
  • Seems similar. I agree that 2.1.0 would be much better. As I mentioned above though, it will not work inside my corporate network. – analyticsPierce Apr 06 '16 at 04:15
  • 2.0 had problems. The BrowserMobProxy developers are recommending that people move to version 2.1.0-beta-5. Earlier 2.1's had issues as well. I suggest that you discuss this with the powers that be. – MikeJRamsey56 Apr 06 '16 at 13:41
  • BMP has a dedicated test almost identical to OP's code, and I have virtually identical code in 2.0 betas, so it just seems premature to say that the version is unusable (not debuggable) and must be updated. – Andrew Regan Apr 06 '16 at 15:44
  • @AndrewRegan I will let you and Pierce work through it then. – MikeJRamsey56 Apr 06 '16 at 16:07