Capturing JSON data from intermediate events using Selenium

Question

Below I have setup a script which simply executes a search on a website. The goal is to capture JSON data utilizing Selenium from an event that is fired from an intermediate script, namely the POST request to "https://www.botoxcosmetic.com/sc/api/findclinic/FindSpecialists" as seen in the included image, but without directly sending a request to that URL using Selenium or the requests library. What is the best way to do this, preferably in Python but open to any language?

from selenium import webdriver
base_url = 'https://www.botoxcosmetic.com/women/find-a-botox-cosmetic-specialist'
driver = webdriver.Chrome()
driver.find_element_by_class_name('normalZip').send_keys('10022')
driver.find_element_by_class_name('normalSearch').click()

I think you may be out of luck. As it's not even found in the html source even after the page is rendered — chitown88, Apr 19 '19 at 15:39
Why do you need to use Selenium? I feel like you are using the wrong tool for the job. — Greg Burghardt, Apr 19 '19 at 16:03
Don't know if [this](https://stackoverflow.com/questions/36744627/network-capturing-with-selenium-phantomjs) is a starting point? — QHarr, Apr 19 '19 at 19:56
It sounds like you want to intercept the xhr. This is actually pretty convoluted in Selenium. — pguardiario, Apr 19 '19 at 22:52
@pguardiario if there is another tool that would be easier than Selenium, I'm all for it. — ikemblem, Apr 20 '19 at 12:33
@ikemblem If I understand your query correctly, you don't want to use the direct api call using request module. I have the solution to get the information with request module, but wanted to check if that's what something you are interested in. — supputuri, Apr 27 '19 at 23:57
You could inject javascript and override xhr methods [see there](https://stackoverflow.com/a/26917962/2498790) — Alex, Apr 28 '19 at 10:40

Ardesco · Accepted Answer · 2019-04-28T11:48:44.690

You will need to use a proxy, my suggestion would be to use the BrowserMob Proxy.

First of all install the BrowserMob Proxy libraries:

pip install browsermob-proxy

You will then need to download the latest release (2.1.4 at the time of writing this), extract it and then place it in your project directory. This is going to be a location you need to pass in when setting up the BrowserMob Proxy server (See below where Server("browsermob-proxy-2.1.4/bin/browsermob-proxy") is defined)

I've then updated your script to the following:

import json

from browsermobproxy import Server
from haralyzer import HarParser
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

base_url = 'https://www.botoxcosmetic.com'
server = Server("browsermob-proxy-2.1.4/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy))

driver = webdriver.Chrome(options=chrome_options)
driver.get("{0}/women/find-a-botox-cosmetic-specialist".format(base_url))

proxy.new_har(options={"captureContent": "true"})
driver.find_element_by_class_name('normalZip').send_keys('10022')
driver.find_element_by_class_name('normalSearch').click()

WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#specialist-results > div")))

har_parser = HarParser(proxy.har)
for entry in har_parser.har_data["entries"]:
    if entry["request"]["url"] == "{0}/sc/api/findclinic/FindSpecialists".format(base_url):
        result = json.loads(entry["response"]["content"]["text"])

driver.quit()
server.stop()

This will start up a BrowserMob Proxy instance and capture the response for the FindSpecialists network call and store it as JSON in the result variable.

You can then use that to do whatever you want to do with the response. Apologies if the code is not as clean as you would expect, I'm not a native Pythonista.

Useful references are:

The BrowserMob Proxy website
The BroswerMob proxy source code on Github
The Python documentation for BrowserMob Proxy
The haralyser website
The ChromeDriver website

if you can please update your code to answer the specific example stated in the question, I can accept your answer — ikemblem, Apr 27 '19 at 13:54
Thanks. Excellent answer and a very good reference for anyone else attempting something similar — ikemblem, Apr 28 '19 at 13:29

Capturing JSON data from intermediate events using Selenium

1 Answers1