1

I am building a web scraper that scrapes citation data from the Web of Science. On Friday, I finalized that part of the program, but today it has stopped working and when I try to access the citation data via Selenium I get a NoSuchElementException. I have tried using different ways (id, class, xpath, css selector) of clicking on this one specific element, but it always throws the error. Here is my code, which was working and now is not:

url = 'https://apps.webofknowledge.com/Search.do?product=UA&SID=8F2pCcE8ApJDSKZLHfF&search_mode=GeneralSearch&prID=acd62bc2-0ee0-47a1-a85d-12009db3c2f5'
driver.get(url)
citers_num = driver.find_element_by_class_name('snowplow-citation-network-times-cited-count-link')
citers_num.click()

Here is the html:

<div class="search-results-data-cite">Times Cited: <a class="snowplow-    times-cited-link" title="View all of the articles that cite this one"     href="/CitingArticles.do     product=WOS&amp;SID=5FAYgZP1cYhuG9LGN3I&amp;search_mode=CitingArticles&amp;parentProduct=WOS&amp;parentQid=18&amp;parentDoc=12&amp;REFID=84460199&amp;excludeEventConfig=ExcludeIfFromNonInterProduct">313</a>

Here is the error I've been getting as of today:

citers_num = driver.find_element_by_class_name('snowplow-citation-network-all-times-cited')
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 564, in find_element_by_class_name
    return self.find_element(by=By.CLASS_NAME, value=name)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"class name","selector":"snowplow-citation-network-all-times-cited"}
  (Session info: chrome=75.0.3770.80)
  (Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Mac OS X 10.14.1 x86_64)

I know that there are other similar questions on here, but none of them have been able to help me. Thank you!

DebanjanB
  • 118,661
  • 30
  • 168
  • 217
Firestarter
  • 86
  • 10
  • I suggest using the ```find_element_by_xpath()``` function for more precision(but less readability). find the xpath by right clicking the source in the "inspect element" section of your browser. – Xosrov Jun 10 '19 at 18:04
  • I have tried that and the same thing happens. – Firestarter Jun 10 '19 at 18:19
  • According to your own post, the class name is `snowplow- times-cited-link`, not `snowplow-citation-network-all-times-cited`.... – John Gordon Jun 10 '19 at 18:20

2 Answers2

0

Use Webdriverwait and following xpath to click on that.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,"//div[@class='search-results-data-cite'][contains(.,'Times Cited:')]/a"))).click()

OR css following css selector.

WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"div.search-results-data-cite a"))).click()

make sure you have imported above imports before use the code.

KunduK
  • 26,790
  • 2
  • 10
  • 32
0

There are exactly two issues out there. As you are attempting a click() you need to to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.search-results-data-cite a[class*='times-cited-link'][href^='/CitingArticles']"))).click()

  • Using XPATH:

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='search-results-data-cite']//a[@title='View all of the articles that cite this one' and starts-with(@href, '/CitingArticles')]"))).click()

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

The next issue is the incompatibility between the version of the binaries you are using as follows:

  • You are using chromedriver=74.0
  • Release Notes of chromedriver=74.0 clearly mentions the following :

Supports Chrome v74

  • You are using chrome=75.0 which have got updated by the recent push.
  • Release Notes of ChromeDriver v75.0 clearly mentions the following :

Supports Chrome v75

So there is a clear mismatch between ChromeDriver v74.0 and the Chrome Browser v75.0 and to address it you need to either:

  • Keep Chrome v75 and upgrade ChromeDriver to ChromeDriver v75 level.
  • Keep ChromeDriver v74 and downgrade Chrome to Chrome v74 level.
DebanjanB
  • 118,661
  • 30
  • 168
  • 217
  • Thank you, but my chrome and chrome driver were already compatible and neither of those approaches work. I think I may have gotten myself banned from Web of Science and that might be the root of the problem, because this was working fine a few days ago and the HTML for the site hasn't changed. – Firestarter Jun 11 '19 at 14:32