1

I am new to learning Python and trying to make a web scraper app but ran into the following error. This is not a complete code but I cannot go ahead if this issue is not resolved first.

Any help will be highly appreciated!

AttributeError: 'NoneType' object has no attribute 'get_text'

import requests
from bs4 import BeautifulSoup

url = "https://www.amazon.co.uk/b?node=13978643031&pf_rd_r=7WY9X56GFTSX0ZTD0VQQ&pf_rd_p=7510143e-2d7f-4e64-a435-f4e242b0abc4"
headers = {
    "user-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36"}
price = 389


def getPrice():
    page = requests.get(url, headers=headers)
    soup = BeautifulSoup(page.content, 'html.parser')
    title = soup.find(id="productTitle").get_text().strip()
    print(title)


if __name__ == "__main__":
    getPrice()

Error:

Traceback (most recent call last): File "/Users/sumeet/vs_code_py/app.py", line 18, in getPrice() File "/Users/sumeet/vs_code_py/app.py", line 13, in getPrice title = soup.find(id="productTitle").get_text().strip() AttributeError: 'NoneType' object has no attribute 'get_text'

Suumeet Singh
  • 21
  • 2
  • 1
    Does this answer your question? [Web-scraping JavaScript page with Python](https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python) – ggorlen Apr 12 '20 at 21:24
  • Do you know which line is raising the error ? – Marco Apr 12 '20 at 21:26
  • 1
    Do you know how to use the Python debugger? That's the most important skill you have to lean right now! – Peter Apr 12 '20 at 21:29
  • @ggorlen Thanks for the suggestion but I'm a noob and it is quite hard to understand that solution. – Suumeet Singh Apr 12 '20 at 22:04
  • @Marco Yes, here's the info. Traceback (most recent call last): File "/Users/sumeet/vs_code_py/app.py", line 18, in getPrice() File "/Users/sumeet/vs_code_py/app.py", line 13, in getPrice title = soup.find(id="productTitle").get_text().strip() AttributeError: 'NoneType' object has no attribute 'get_text' – Suumeet Singh Apr 12 '20 at 22:04
  • @Peter I'm afraid I don't but thank you for pointing out to me. I must spend some time learning that too. – Suumeet Singh Apr 12 '20 at 22:04
  • What do you understand from the error message? What do you think caused this? have you done any debugging? – AMC Apr 12 '20 at 23:15

1 Answers1

1

The issue is probably that the BeautifulSoup.find() method has not found anything with id='productTitle' so you are trying to get the text of a non-existent element, represented in Python as the primitive None. You need to add some logic to accomodate this possibility, either using if/else or try/except (assuming of course that this actually works on other pages? If not perhaps you aren't searching for the right thing)

Also, as flagged in the comments if the element you are searching for is created by Javascript (which it looks like it is) then the reason you aren't finding it is because it doesn't exist at the stage you are scraping. You need something like Selenium that will actually execute the Javascript.

SimonN
  • 1,832
  • 2
  • 8
  • 16
  • Thank you for the detailed explanation, Simon! It seems like I need to strengthen my fundamentals a bit more. I was following a tutorial on YouTube and ran into this issue. I admit I am at a stage where if I run into an issue I cannot find a solution on my own. – Suumeet Singh Apr 12 '20 at 22:09