Browsermob-Proxy - Problem to read XHR over SSL

Question

This is my setup:

Raspberry Pi 3 Model B Plus Rev 1.3
Linux 4.19.66-v7+ (RaspbianGNU/Linux 9 (stretch))
Selenium 3.141.0
Browsermob-Proxy 2.1.4
Chromium 72.0.3626.121
ChromeDriver 72.0.3626.121
Python 3.5.3

I would like to record the network traffic when I visit an https page. So far, it actually works quite well. The problem is, the content of the packages that browsermob proxy records are encrypted.

Here my code

import pprint
import time
from selenium import webdriver
from pyvirtualdisplay import Display
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from browsermobproxy import Server

# Source: https://github.com/ArturSpirin/YouTube-WebDriver-Tutorials/blob/master/proxy/BmpProxy.py
class ProxyManger:

    __BMP = "/usr/local/bin/browsermob-proxy-2.1.4/bin/browsermob-proxy"

    def __init__(self):
        self.__server = Server(ProxyManger.__BMP, options={'port': 8089})
        self.__client = None

    def start_server(self):
        self.__server.start()
        return self.__server

    def start_client(self):
        self.__client = self.__server.create_proxy(params={"trustAllServers": "true"})
        return self.__client

    @property
    def client(self):                                                                                                                                                                                                                                return self.__client                                                                                                                                                                                                                                                                                                                                                                                                                                                                  @property
    def server(self):
        return self.__server

# set virtual dispaly
display = Display(visible=0, size=(800, 600))
display.start() 

# set browsermob-proxy
proxy = ProxyManger()                                                                                                                                                                                                                        server = proxy.start_server() 
client = proxy.start_client()
client.new_har(url)

# set chrome options 
opts = webdriver.ChromeOptions()
opts.add_argument("--proxy-server={}".format(client.proxy))
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--no-sandbox")                                                                                                                                                                                           
opts.add_argument("--ignore-certificate-errors")

browser = webdriver.Chrome(options=opts)

browser.get(url)
time.sleep(10)
pprint.pprint(client.har)

browser.quit()
server.stop()
display.stop()

The code works quite well so far. I receive the packages i want.

The problem is the encrypted content. It is clear to me that the browsermob-proxy acts as a MITM and cannot read the contents of these packages due to the end-to-end encryption.

...
'content': {'comment': '',
                                               'encoding': 'base64',
                                               'mimeType': 'application/json',
                                               'size': 10493,
                                               'text': 'IUQHACBHdln10z6SWSgCD9DkLZ0OUL9H9+NwllhRXLaI+7nOI023mVdkr5uCJV115AeolXUwyJUgklGU8z/0tYu/n/iuQCnAQJIG8JwmwaOcwRRLTheZ8abRSDFM/gQTqc6nP03QiSiJ/ZuxVZTkH/6SKKpir/SsMAt5+RMiPU+eJ3fN+U8JBjguGdWoNCGCrSqOw9gBeKORKcY4Ek014310aXl3BUqBnJ01VqPyeaJQasKY1hxRkkYTfFGAefuYQ5pbF1588ghm1VDPrdoKB1lERMVl/j0Y2HWEt+tbdHYe3t9fCrtSN+5Nq++ejmp/pg9UUuyVF8FlWvJiA6YB'},
...

I run the Raspberry Pi headless. That means I only have access via ssh and no x. According to the github page of Browsermob-proxy, it is possible to add a certificate to my browser. According to some internet research, this usually works in chrome via the GUI.

After doing some more research, I found this:

https://github.com/ThomasLeister/root-certificate-deployment

I ran linux-browser-import.sh, but unfortunately this had no effect on that.

Where is my mistake? Does someone have a solution to my problem? How is it possible to read packages decrypted from an ssl connection?

Is there any other method known how I can read xhr packages?

Thanks, Mike

Browsermob-Proxy - Problem to read XHR over SSL

0 Answers0