I parse website 'http://ok.ru'. To get data from the post request I need to send a specific token that is generated by Javascript on the website and this token is contained in headers.
So I thought maybe one solution would be to open the website, let it generate token, grab headers and that's it.
One tool that can implement Java scripts is Selenium, however, to get headers I need to use brosermob-proxy (or equivalent). That is where I'm stuck.
There's no headers in response and I can't figure it out. Maybe someone who worked with browsermob can see what's wrong? I would also be glad to hear another solutions to my task. The code itself is below:
from browsermobproxy import Server
from selenium import webdriver
from ast import literal_eval
import json, os
os.chdir('C:/browsermob-proxy-2.1.0-beta-2/bin')
server = Server()
server.start()
proxy = server.create_proxy()
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har('test')
driver.get('http://ok.ru')
driver.find_element_by_xpath('//input[@name="st.email"]').send_keys('****@****.com')
driver.find_element_by_xpath('//input[@name="st.password"]').send_keys('****')
driver.find_element_by_xpath(u'//input[contains(@value,"Log in")]').click()
result = literal_eval(json.dumps(proxy.har, ensure_ascii=False))
driver.close()
for entry in result['log']['entries']:
if len(entry['response']['headers']) > 0:
print entry['response']['headers']
