1

I am trying to crawl the reviews on this website: https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425897/.

However, I have to click a button ( Toon meer) to show all the reviews.

<div class="load-more load-more--divider load-more--reviews js-review-load-more-container">
  <a data-href="/nl/rnwy/productPage/reviews?productId=9200000118425896&amp;offset=5&amp;limit=10&amp;loadMore=true" class="review-load-more__button js-review-load-more-button" data-test="review-load-more"><div class="css-loader css-loader--reviews"></div>
      Toon meer</a>
</div>

I use the below code :

import requests
import pandas as pd
from selenium import webdriver
from bs4 import BeautifulSoup
from datetime import datetime

start_time = datetime.now()
data = []
link = "https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425897/"
op = webdriver.ChromeOptions()
op.add_argument('--ignore-certificate-errors')
op.add_argument('--incognito')
op.add_argument('--headless')
driver = webdriver.Chrome(executable_path='D:/Desktop/work/real/chromedriver.exe',options=op)
driver.get(link)
driver.find_element_by_css_selector('div.review-load-more__button js-review-load-more-button').click()

However, it throws an error:

No such element: Unable to locate element: {"method":"css selector","selector":"div.review-load-more__button js-review-load-more-button"} .

Is there any solution?

mht
  • 91
  • 7

3 Answers3

2

Css selectors cannot select an element by containing text.

Try using xpath. The last line of your script should look something like:

wait = WebDriverWait(driver, 10)
wait.until(expected_conditions.element_to_be_clickable((By.XPATH, "//a[contains(., 'Toon meer')]")).click()
Mate Mrše
  • 6,305
  • 6
  • 26
  • 53
  • I tried your solution and it threw an error : element click intercepted: Element is not clickable at point (202, 1865). What does it mean ? – mht Dec 09 '20 at 15:27
  • that means another element (invisible) is blocking the click. Add a wait condition as in the updated answer. – Mate Mrše Dec 09 '20 at 15:34
1

When you get the page a popup comes with an accept button click it and then proceed with clicking your element.

driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[@class='js-confirm-button']"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@data-test='review-load-more']"))).click()

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
Arundeep Chohan
  • 6,219
  • 4
  • 6
  • 22
0

To click on Toon meer you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-test='consent-modal-confirm-btn']>span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.review-load-more__button.js-review-load-more-button"))).click()

  • Using XPATH:

    driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@data-test='consent-modal-confirm-btn']/span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='review-load-more__button js-review-load-more-button' and contains(., 'Toon meer')]"))).click()

  • Note: You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
DebanjanB
  • 118,661
  • 30
  • 168
  • 217
  • Thanks for your comprehensive solution. I tried to run by using css_selector. However, sometimes it can click the button, sometimes it does not. I dont know why. Should I extend the waiting time ? – mht Dec 10 '20 at 19:08
  • 1
    @mht Yes, the is a dependency on the network connectivity. I was able to reduce the waiting time to `20` secs. You may need to adjust. – DebanjanB Dec 10 '20 at 19:10
  • How can I determine the waiting time ? How can I get the network connectivity ( I guess it is a number ?) ? – mht Dec 10 '20 at 19:13
  • @mht Apologies, instead of _network connectivity_ we should consider it as the time required for the desired element to render within the HTML and this specific value should be part of your _Test Plan_ / _testcase_. – DebanjanB Dec 10 '20 at 19:58