3

I am trying to use browsermob-proxy to monitor all requests and responses for selenium tests. In my case, I am running the selenium tests with the py.test framework inside a docker image on jenkins. I am using the following versions:

  • selenium==3.8.0
  • pyvirtualdisplay==0.2.1
  • pytest==3.4.0
  • browsermob-proxy==0.8.0

also java openjdk-8-jdk.

Inside the py.test code I have the following line to create and start the server:

proxyserver =  Server(
           path="/root/tests/bsp_usecase_tests/bin/browsermob-proxy",
           options={'port': 8090}
       )
proxyserver.start()
driverproxy = proxyserver.create_proxy() # line 127

but in the last line I get the following error:

conftest.py:127: in basedriver
    driverproxy = proxyserver.create_proxy()
/usr/local/lib/python2.7/dist-packages/browsermobproxy/server.py:40: in create_proxy
    client = Client(self.url[7:], params)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <browsermobproxy.client.Client object at 0x7f1a5f73f1d0>
url = 'localhost:8090', params = {}, options = {}

    def __init__(self, url, params=None, options=None):
        """
            Initialises a new Client object


            :param url: This is where the BrowserMob Proxy lives
            :param params: URL query (for example httpProxy and httpsProxy vars)
            :param options: Dictionary that can contain the port of an existing
                            proxy to use (for example 'existing_proxy_port_to_use')
            """
        params = params if params is not None else {}
        options = options if options is not None else {}
        self.host = "http://" + url
        if params:
            urlparams = "?" + unquote(urlencode(params))
        else:
            urlparams = ""
        if 'existing_proxy_port_to_use' in options:
            self.port = options['existing_proxy_port_to_use']
        else:
            resp = requests.post('%s/proxy' % self.host + urlparams)
            content = resp.content.decode('utf-8')
            try:
                jcontent = json.loads(content)
            except Exception as e:
                raise Exception("Could not read Browsermob-Proxy json\n"
>                               "Another server running on this port?\n%s..." % content[:512])
E               Exception: Could not read Browsermob-Proxy json
E               Another server running on this port?
E               <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
E               <html><head>
E               <meta type="copyright" content="Copyright (C) 1996-2016 The Squid Software Foundation and contributors">
E               <meta http-equiv="Content-Type" CONTENT="text/html; charset=utf-8">
E               <title>ERROR: The requested URL could not be retrieved</title>
E               <style type="text/css"><!-- 
E                /*
E                * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
E                *
E                * Squid software is distributed under GPLv2+ license and incl...

/usr/local/lib/python2.7/dist-packages/browsermobproxy/client.py:37: Exception

Maybe I forgot to copy important files into the docker image?

I have the following setup for the browsermob-proxy (besides the pip install):

bsp_usecase_tests/bin:
browsermob-proxy

bsp_usecase_tests/lib:
browsermob-dist-2.1.4.jar

i.e. I have copied the file browsermob-proxy into bin and the file browsermob-dist-2.1.4.jar into the lib folder. Also, the server itself seem to be starting and running. But I get this weird error when I try to get the proxy information I can use for the selenium driver...

Any ideas what I have done wrong, or what is missing?

Alex
  • 34,021
  • 64
  • 178
  • 371

1 Answers1

0

Seems to me that the port is already in use. You can check if the port is being used using lsof and then kill it, if it is in use. You can check using lsof -i:8090

EDIT : And you have already done that in this thread - How to fix 'Address already in use' error with browsermob-proxy?

I would initialize server with

proxyserver =  Server(
       path="/root/tests/bsp_usecase_tests/bin/browsermob-proxy",
       options={'existing_proxy_port_to_use': 8090})

This should do it.

The internal code in Browsermob-Proxy is looking for key in 'options' dictionary named 'existing_proxy_port_to_use', else it uses default i.e. empty dictionary.

AnkDasCo
  • 869
  • 6
  • 16
  • I have tried both options, `options={'existing_proxy_port_to_use': 8090}` and `options={'port':8090}`, and in both cases I get the exact same error message... The problem is not that easy I am afraid... – Alex Aug 20 '18 at 14:58
  • In your error console, can you share the next line to this self = after you tried the existing_proxy_port_to_use – AnkDasCo Aug 20 '18 at 15:04
  • I am not sure what you are asking. What error console? Remember that I am running all of the code inside a docker image... – Alex Aug 20 '18 at 15:16
  • Ha, I mean the error that you get when using existing_proxy_port_to_use': 8090', can you post that, it feels redundant but I would like to see if it made any difference – AnkDasCo Aug 20 '18 at 15:38
  • I just looked at the source code for broswermob-proxy (the python wrapper), and the only thing the implementation is checking in the options is a possible key `port`, nothing else (https://github.com/AutomatedTester/browsermob-proxy-py/blob/master/browsermobproxy/server.py). Also it looks like I have an somewhat older installation. – Alex Aug 21 '18 at 05:11
  • Otherwise the complete error here: https://gist.github.com/alex4200/bab5063b68c9e2362b4d683d50fa3159 – Alex Aug 21 '18 at 05:15
  • I see, the option you mentioned is in the `Client` package. I will try to call the Client directly... – Alex Aug 21 '18 at 05:19
  • I think the problem was really that processes used these ports. I am not sure I can reproduce the problem anymore. Suggestion: Either restart computer or investigate the processes that use the ports... – Alex Sep 10 '18 at 12:59