0

i am trying to scrape the price from this page : url = https://www.renodepot.com/en/steph-round-base-shower-kit-69375118

the price information is given in the span tag and I am not able to scrape it. the simple code which I am using for this is 

from requests import get
from bs4 import BeautifulSoup
response = get(url)
html_soup = BeautifulSoup(response.text, 'html.parser')
ProductPrice = html_soup.find('div',class_ = 'product_price_wrapper')

but this returns nothing, I think 

 BEGIN RenoProdDetailPriceSnippet.jsp

which appears just above the price div tab is causing the information to be protected.

I even tried doing it with selenium but was not successful. I tried many other combination to get the price but was not able to get the same.

So, I am looking for some ideas to solve this. Thanks

Jaskaran Singh
  • 1
  • You cannot scrape dynamically generated pages with `requests`. Use `selenium` or a similar web driver. – DYZ Sep 16 '18 at 23:25
  • Possible dupe of https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python – DYZ Sep 16 '18 at 23:28
  • 1
    Possible duplicate of [Parse the JavaScript returned from BeautifulSoup](https://stackoverflow.com/questions/21069294/parse-the-javascript-returned-from-beautifulsoup) – Chris Sep 16 '18 at 23:30
  • I tried doing this with selenium too, but was not able to get the required information. I even tried headless web drivers but was not successful. – Jaskaran Singh Sep 17 '18 at 05:07

1 Answers1

0

You cannot scrape the page because it requires the completion of a reCAPTCHA to access. This is specifically designed to stop bots.

If you examine html_soup you will find that you are actually searching the reCAPTCHA page, not the desired product page.

Joon-Ho Son
  • 75
  • 8