How to scroll specified infinite scroll element in Selenium with Python

Question

I'm new to Selenium and not familiar with JavaScript. I'd like to get news links from Top Stories in this link. therefore, I have to scroll only that container which includes headlines. I am using Selenium with Python. My code is

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome('C:\\Program Files (x86)\\Google\\Chrome\\chromedriver.exe')
driver.get("http://www.marketwatch.com/newsviewer")
element = driver.find_element_by_tag_name('html')
element.send_keys(Keys.END)
time.sleep(8)
element.send_keys(Keys.HOME)

I get an error that says: "WebDriverException: Message: unknown error: cannot focus element". I read how to scrape an infinite scroll in a page, but scrolling a specific infinite scroll within page is a challenge to me. Any help would be appreciated.

Answer 1

What about something like this:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Don't forget to include sufficient sleep time to ensure that the content has loaded properly. Moreover, since you are not looking for any specific text, you will need to decide when to stop. So perhaps checking somehow that the data you are scraping stopped changing.

Saying that, make sure that you need to scroll at all. In some cases the whole data on a list is available directly, even if it is partially invisible on the screen. The scroll is only required when you need to load additional data on AJAX page.