Extract contents after click() with Python Selenium

Question

I would like to extract the bio of a guy Herbert W. Gullquist. It is from "Manager Timeline" after clicking on his name ("Gullquist is chief investment officer and a general partner with Lazard Asset Management...") from this web page: https://www.morningstar.com/funds/xnas/lziex/people

The code couldn't find the guy. Was it because the code clicked on the wrong place?

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options)
driver.get("https://www.morningstar.com/funds/xnas/lziex/people")

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='sal-component-ctn sal-component-manager-timeline-chart']//text[text()='Gullquist']/.."))).click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.sal-modal-biography.ng-binding.ng-scope"))).text.strip())

In addition, what if I want everyone's bio (9 people in total) from the Manager Timeline instead of a certain person? Any help is much appreciated.

Answer 1

Wrong xpath value at the line:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='sal-component-ctn sal-component-manager-timeline-chart']//text[text()='Gullquist']/.."))).click()

Change with the below value:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[text()='Herbert W. Gullquist']"))).click()

Answer 2

There are 6 current managers and 3 past managers. To get both links

current= driver.find_elements(By.XPATH, "//*[@class='current-manager-timeline']//*[name()='g']//*[name()='text']") past = driver.find_elements(By.XPATH, "//*[@class='past-manager-timeline']//*[name()='g']//*[name()='text']") total_links = current+past

After getting the links, you can cycle over the links and extract the bio with

bio = driver.find_element(By.XPATH, "//div[contains(@class, 'biography')]").text

Answer 3

To extract the bio of "Herbert W. Gullquist" from the web page https://www.morningstar.com/funds/xnas/lziex/people and close the popup, as the element is within a <tag> tag, you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategies:

• Code Block:

  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://www.morningstar.com/funds/xnas/lziex/people")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='sal-component-ctn sal-component-manager-timeline-chart']//*[name()='svg']//*[name()='g' and @class='past-manager-timeline']//*[text()='Herbert W. Gullquist']"))).click()
  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.sal-modal-biography.ng-binding.ng-scope"))).text.strip())
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='sal-component-ctn sal-modal-scrollable']//div[@class='sal-manager-modal__modalHeader']//button[@class='sal-icons sal-icons--close mds-button mds-button--icon-only']"))).click()
  

• Console Output:

  Gullquist is chief investment officer and a general partner with Lazard Asset Management, his employer since 1982. Previously, he spent 12 years as general partner, managing director, and chief investment officer of Oppenheimer & Company. Prior to that, from 1970 to 1971, he served as the director of Stuyvesant Asset Management, a company he founded. He has also worked at First National Bank of Chicago.
  

You can find a relevant discussion in How to click on SVG elements using XPath and Selenium WebDriver through Java