How to scroll a web page down until specific button to click?

Question

I am working with Python and Selenium but I got stuck scraping a page trying to scroll down until selenium finds the classic "Show more" button (that seems to be hidden).

What I have done 'til now is the following:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


from selenium.webdriver.common.keys import Keys


element = browser.find_element_by_xpath('//*[@id="ember371"]/button') # you can use ANY way to locate element
coordinates = element.location_once_scrolled_into_view # returns dict of X, Y coordinates
browser.execute_script('window.scrollTo({}, {});'.format(coordinates['x'], coordinates['y']))

However, whatever I try to do Python throws me several kind of errors, for example the below.

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="ember371"]/button"}
  (Session info: chrome=80.0.3987.122)

Could it be due to the fact that the button is hidden?

Below is the HTML code for the button I am looking for and that I would like to click on.

<button class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid" aria-controls="skill-categories-expanded" aria-expanded="false" data-control-name="skill_details" data-ember-action="" data-ember-action-2182="2182">
                    <span aria-hidden="true">
                    Show more
                  </span>
                  <span class="visually-hidden">
                    BLA BLA BLA
                  </span>
                  <li-icon aria-hidden="true" type="chevron-down-icon" class="pv-skills-section__chevron-icon" size="small"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" data-supported-dps="16x16" fill="currentColor" width="16" height="16" focusable="false">
  <path d="M8 9l5.93-4L15 6.54l-6.15 4.2a1.5 1.5 0 01-1.69 0L1 6.54 2.07 5z"></path>
</svg></li-icon>

</button>

#################################################################################

It worked with the following:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


# scroll down smoothly
scheight = .1
while scheight < 1.0:
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight*%s);" % scheight)
    scheight += .1
    time.sleep(1)
try:
    browser.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))
    browser.execute_script("arguments[0].click();", WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))
except:
    print('NO')

add wait for getting element, might be page loading and element could not present in HTML content. — Manali Kagathara, Mar 05 '20 at 05:43

DebanjanB · Accepted Answer · 2020-03-06T12:05:50.207

1

The element Show more within linkedin website is EmberJS enabled element. So to scrollIntoView the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button[aria-controls='skill-categories-expanded'][data-control-name='skill_details']>span"))))
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-controls='skill-categories-expanded'][data-control-name='skill_details']>span"))))

Using XPATH:

driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))
driver.execute_script("arguments[0].click();", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-controls='skill-categories-expanded' and @data-control-name='skill_details']/span[normalize-space()='Show more']"))))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Mar 06 '20 at 12:05

answered Mar 05 '20 at 11:23

DebanjanB

118,661
30
168
217

@DaniB This answer was the solution to the _NoSuchElementException_where as _chrome not reachable_ is a all together a different issue pertaining to the version of the _ChromeDriver_ and _Chrome_ you are using. – DebanjanB Mar 05 '20 at 21:05
1

Thanks @DebanjanB! What if I would like to click on the button? – DaniB Mar 05 '20 at 21:10
@DaniB Checkout the updated answer and let me know the status. – DebanjanB Mar 05 '20 at 21:14
1

nope, I don't know why it doesn't work. (Code updated) – DaniB Mar 05 '20 at 21:26
@DaniB You are _scrollingIntoView_ `normalize-space()='Show more'`, how can `normalize-space()='Mostra altro'` be clickable? – DebanjanB Mar 05 '20 at 21:29
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/209114/discussion-between-danib-and-debanjanb). – DaniB Mar 05 '20 at 21:35
@DaniB Checkout the updated answer and let me know the status. – DebanjanB Mar 05 '20 at 21:59

score 0 · Answer 2 · answered Mar 04 '20 at 23:22

You can try using a while loop here, check for the existence of the button as the condition, and put the browser.execute_script in the body. In this way, you would start by checking if the button exists, if not then scroll for 200 px or so (just make sure you don't scroll past the button) until the button appears.

score 0 · Answer 3 · answered Mar 05 '20 at 05:51

You can first try to find if the "show more" button is enabled by using the isEnabled() method. This method returns “true” value if the specified web element is enabled on the web page otherwise returns “false” value if the web element is disabled on the web page.

If isEnabled() method returns False, you can then try scrolling down the page till the element is visible and use the isEnabled() method again.

Also, try replacing id with XPath. It may work.

How to scroll a web page down until specific button to click?

3 Answers3