Unable to click on different results to parse the title from inner pages

Question

I've written a script in python to populate results after initiating a serach in the search box in google map. My script is doing this flawlessly. Now, I'm trying to click on each of the results to go one layer deep and parse the title from there.

When I run the script, I get one title successfully but then the script throws the same common error element is not attached to the dom, although I've taken all the measures to get rid of that.

website address

I used here this keyword motels in new jersey as search.

I've tried with:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://www.google.com/maps/search/")
wait = WebDriverWait(driver, 10)

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input#searchboxinput"))).send_keys("motels in new jersey")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#searchbox-searchbutton"))).click()

while True:
    try:
        for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "[class='section-result'] h3[class='section-result-title']"))):
            # click on each of the results
            item.click()
            # the script now in inner page to parse the title
            name = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"h1.section-hero-header-title-title"))).text
            print(name)
            # click on the "Back to results" link located on the top left to get back to the results page
            wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"button[class^='section-back-to-list-button']"))).click()
            # wait for the spinner to be invisible
            wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "#searchbox[class*='loading']")))
            # tried to get rid of staleness condition
            wait.until(EC.staleness_of(item))
    except Exception:
        break

How can I click on different results to parse the title from inner pages?

Answer 1

To continue looping over a list that has been affected by any manipulation of the DOM, you have to refresh your list of elements after each .click() event, or any event that requires loading of elements within your current scope. Please try the below code to solve you issue:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://www.google.com/maps/search/")
wait = WebDriverWait(driver, 10)

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input#searchboxinput"))).send_keys("motels in new jersey")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#searchbox-searchbutton"))).click()

while True:
    try:
        for count, item in enumerate(wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "[class='section-result'] h3[class='section-result-title']")))):
            refreshList = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "[class='section-result'] h3[class='section-result-title']")))
            # click on each of the results
            refreshList[count].click()
            # the script now in inner page to parse the title
            name = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"h1.section-hero-header-title-title"))).text
            print(name)
            # click on the "Back to results" link located on the top left to get back to the results page
            wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,"button[class^='section-back-to-list-button']"))).click()
            # wait for the spinner to be invisible
            wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "#searchbox[class*='loading']")))
            # tried to get rid of staleness condition
            wait.until(EC.staleness_of(refreshList[count]))
    except Exception:
        break

Answer 2

To parse the title of the search results e.g. Holiday Inn Express, Hyatt House Mt Laurel, etc you don't have to click() on each of the results and go one layer deep. Instead you can induce WebDriverWait for the desired visibility_of_all_elements_located() and you can use the following Locator Strategy:

• Code Block:

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  chrome_options = webdriver.ChromeOptions() 
  chrome_options.add_argument("start-maximized")
  chrome_options.add_argument('disable-infobars')
  driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://www.google.com/maps/search/")
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#searchboxinput"))).send_keys("motels in new jersey")
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#searchbox-searchbutton"))).click()
  print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.section-result-title-container h3.section-result-title > span")))])
  

• Console Output:

  ['Holiday Inn Express & Suites Philadelphia - Mt. Laurel', 'Hyatt House Mt Laurel', 'Motel 6 East Brunswick', 'Motel 6 New Brunswick NJ', 'Skyview Motel', 'Anchor Motel', 'Motel 6 Philadelphia - MT Laurel NJ', 'Motel 6 Piscataway', 'Motel 6 Elizabeth - Newark Liberty Intl Airport', 'Twin Oaks Motel', 'Shore Hills Motel', 'Franklin Terrace Motel', 'Loop Inn Motel', 'Hershey Motel', 'Royal Motel', 'Ala Moana Motel', 'Bird of Paradise Motel', 'Appalachian Motel', 'Hudson Plaza Motel', 'Fair Motel', 'Days Inn & Suites by Wyndham Cherry Hill - Philadelphia', 'Holly Hill Motel']