How to know if my Custom HTTP headers are being passed?

Question

So, I have been struggling with passing custom HTTP header for some time now.

I am creating a script (Python) to Open a URL with Custom headers like

    {'Referer': 'https://google.com', 'X-Forwarded-For': '47.29.76.109', 

'User-Agent': 'Mozilla/5.0 (Linux; Android 7.1.1; CPH1723 Build/N6F26Q; 

wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 

Mobile Safari/537.36', 'existing_proxy_port_to_use': '8090'}

I have been using BrowserMob-Proxy for this but I am unable to see the effect when I try checking Network field in Inspect of Google Chrome.

CODE:

def automation():
    headers = pd.read_excel('Database/header.xlsx')
    for i in range(0,headers.shape[0]):
        dict = {}
        header = headers.loc[i]
        dict['Referer'] = header['Referrer']
        dict[header['Option']] = header['IP']
        dict['User-Agent'] = header['USERAGENT']
        dict['existing_proxy_port_to_use'] = "8090"
        print(dict)

        URL = 'xyz'
        data = pd.read_csv('Database/data.csv')
        server = Server(path="./browsermob-proxy/bin/browsermob-proxy", options=dict)
        server.start()
        proxy = server.create_proxy()
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy)) #Configure chrome options
        driver = webdriver.Chrome(chrome_options=chrome_options,executable_path='/home/.../chromedriver')
        proxy.new_har("google")
        for j in range(0,data.shape[0]):
            datum = data.loc[j]
            print(datum)
            driver.get(URL)
        driver.quit()
        server.stop()   
    return None    

automation()

I am reading these Parameters from header file and using Selenium to fill the Google Form.

So, Please help me in knowing how to pass the headers correctly and how to know if they are working.

Would using httpbin.org, specifically [this return header method](https://httpbin.org/#/Request_inspection/get_headers) be of use to you? — Chillie, Jan 31 '19 at 09:26
It is a nice utility but I want to use Custom HTTP header in python script with selenium and check if they are being used. — Kartikey Singh, Jan 31 '19 at 09:32
I just want to make sure I worded this right. When you request https://httpbin.org/headers the response content contains the headers you sent it with your request. This would be a simple way to see if your custom headers are actually used. — Chillie, Jan 31 '19 at 09:44
Thanks for the help, go to know how to use something new. But it still doesn't help my problem :( — Kartikey Singh, Jan 31 '19 at 15:02

score 2 · Accepted Answer · edited May 23 '19 at 06:44

I solved the problem of passing the header by removing the Browsermob-proxy and instead using seleniumwire and use its driver._client.set_header_overrides(headers=dict_headers) to override the default HTTP headers.

def automation():

    headers = pd.read_excel('Database/header.xlsx')
    data = pd.read_csv('Database/data.csv')

    for i in range(0,headers.shape[0]):
        dict_headers = {}
        header = headers.loc[i]
        dict_headers['Referer'] = header['Referrer']
        dict_headers[header['Option']] = header['IP']
        dict_headers['User-Agent'] = header['USERAGENT']

        URL = 'xyz'

        user_agent = "user-agent="+header['USERAGENT']
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument(user_agent) 
        driver = webdriver.Chrome(
                chrome_options=chrome_options,
                executable_path='/home/.../chromedriver')

        driver._client.set_header_overrides(headers=dict_headers)
        datum = data.loc[i]
        driver.get(URL)
        driver.quit()
    return None    
automation()

How to know if my Custom HTTP headers are being passed?

1 Answers1

Linked