I am scraping a site using selenium. The site uses infinite scrolling, so I am trying to use selenium to scroll down and get more search results than those served after the first request.
I adapted the following, excellent script:
crawl site that has infinite scrolling using python
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.implicitly_wait(30)
verificationErrors = []
accept_next_alert = True
driver.get(my_url)
driver.find_element_by_link_text("All").click()
for i in range(1,100):
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(4)
html_source = driver.page_source
data = html_source.encode('utf-8')
The web browser is opened using the url above. However, I get the following error message:
C:\Users\xxxx\Anaconda3\python.exe
C:/Users/xxxx/Documents/IdenSoft/unittest.py
Traceback (most recent call last):
File "C:/Users/xxxx/unittest.py", line 23, in <module>
driver.find_element_by_link_text("All").click()
File C:\Users\xxxx\Anaconda3\lib\sitepackages\selenium\webdriver\remote\webelement.py", line 77, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\xxxx\Anaconda3\lib\site
packages\selenium\webdriver\remote\webelement.py", line 494, in _execute
return self._parent.execute(command, params)
File "C:\xxxx\xxxx\Anaconda3\lib\site- packages\selenium\webdriver\remote\webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "C:\Users\xxxx\Anaconda3\lib\site packages\selenium\webdriver\remote\errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message:
Process finished with exit code 1
How should I change the script for it to work?
