Trying to get HAR files for a bunch of urls using browsermob-proxy with selenium in Python. For the basic implementation, I'm using the sample code from Browsermob documentation. My code below
from browsermobproxy import Server
import psutil
import time
server = Server(“/path/to/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har(“google”)
driver.get("http://www.google.com")
print(proxy.har) #ISSUE IN THIS LINE
server.stop()
driver.quit()
I'm able to initialize browsermob-proxy, and get selenium to open the page on firefox (and chrome). When it gets to the 'proxy.har' line, it throws a JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Error trace below
JSONDecodeError Traceback (most recent call last)
<ipython-input-2-f690bb4c2c08> in <module>()
----> 1 proxy.har
~/anaconda3/lib/python3.6/site-packages/browsermobproxy/client.py in har(self)
102 r = requests.get('%s/proxy/%s/har' % (self.host, self.port))
103
--> 104 return r.json()
105
106 def new_har(self, ref=None, options=None, title=None):
~/anaconda3/lib/python3.6/site-packages/requests/models.py in json(self, **kwargs)
890 # used.
891 pass
--> 892 return complexjson.loads(self.text, **kwargs)
893
894 @property
~/anaconda3/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
352 parse_int is None and parse_float is None and
353 parse_constant is None and object_pairs_hook is None and not kw):
--> 354 return _default_decoder.decode(s)
355 if cls is None:
356 cls = JSONDecoder
~/anaconda3/lib/python3.6/json/decoder.py in decode(self, s, _w)
337
338 """
--> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
340 end = _w(s, end).end()
341 if end != len(s):
~/anaconda3/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
355 obj, end = self.scan_once(s, idx)
356 except StopIteration as err:
--> 357 raise JSONDecodeError("Expecting value", s, err.value) from None
358 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I've tried a bunch of things to resolve this, but can't get it to work
- The GET request returns 200
- Tried some json functions to load or parse to utf-8 or strip invalid characters
- Tried simplejson
- Tried changing browsermob port
- Tried on different networks in case the block was being caused by proxy settings
I get the same error no matter what I try. I initially thought the issue was with my installation. This is how I did it:
- Installed selenium using pip3 (using python 3.6 with Jupyter/iPython)
- Downloaded browsermob binary from https://bmp.lightbody.net/
- Downloaded geckodriver
- Started the proxy using ./browsermob-proxy within /bin, and also tried selecting the port using -port
Anyone had similar issues, which they resolved using anything other than the ones described above?
