Wait Before Scraping using Beatifulsoup

Question

I'm trying to scrape data from this review site. It first go through first page, check if there's a 2nd page then go to it too. Problem is when getting to 2nd page. Page takes time to update and I still get the first page's data instead of 2nd

For example, if you go here, you will see how it takes time to load page 2 data

I tried to put a timeout or sleep but didn't work. Prefer a solution with minimal package/browser dependency (like webdriver.PhantomJS()) as I need to run this code on my employer's environment and not sure if I can use it. Thank you!!

from urllib.request import Request, urlopen
from time import sleep
from socket import timeout
    
req = Request(softwareadvice, headers={'User-Agent': 'Mozilla/5.0'})
    
web_byte = urlopen(req, timeout=10).read()
    
webpage = web_byte.decode('utf-8')
parsed_html = BeautifulSoup(webpage, features="lxml")
    
true=parsed_html.find('div', {'class':['Grid-cell--1of12 pagination-arrows pagination-arrows-right']})
    
while(true):
                                       
    true = parsed_html.find('div', {'class':['Grid-cell--1of12 pagination-arrows pagination-arrows-right']})

    if(not True):
       true=False
    else: 
       req = Request(softwareadvice+'?review.page=2', headers=hdr)
       sleep(10)
       webpage = urlopen(req, timeout=10)
       sleep(10)
       webpage = webpage.read().decode('utf-8')
       parsed_html = BeautifulSoup(webpage, features="lxml")

Answer 1

The reviews are loaded from external source via Ajax request. You can use this example how to load them:

import re
import json
import requests
from bs4 import BeautifulSoup


url = "https://www.softwareadvice.com/sms-marketing/twilio-profile/reviews/"
api_url = (
    "https://pkvwzofxkc.execute-api.us-east-1.amazonaws.com/production/reviews"
)

params = {
    "q": "s*|-s*",
    "facet.gdm_industry_id": '{"sort":"bucket","size":200}',
    "fq": "(and product_id: '{}' listed:1)",
    "q.options": '{"fields":["pros^5","cons^5","advice^5","review^5","review_title^5","vendor_response^5"]}',
    "size": "50",
    "start": "50",
    "sort": "completeness_score desc,date_submitted desc",
}

# get product id
soup = BeautifulSoup(requests.get(url).content, "html.parser")
a = soup.select_one('a[href^="https://reviews.softwareadvice.com/new/"]')
id_ = int("".join(re.findall(r"\d+", a["href"])))

params["fq"] = params["fq"].format(id_)

for start in range(0, 3):  # <-- increase the number of pages here
    params["start"] = 50 * start

    data = requests.get(api_url, params=params).json()

    # uncomment this to print all data:
    # print(json.dumps(data, indent=4))

    # print some data:
    for h in data["hits"]["hit"]:
        if "review" in h["fields"]:
            print(h["fields"]["review"])
            print("-" * 80)

Prints:

After 2 years using Twilio services, mainly phone and messages, I can say I am so happy I found this solution to handle my communications. It is so flexible,  Although it has been a little bit complicated sometimes to self-learn about online phoning systems it saved me from a lot of hassles I wanted to avoid. The best benefit you get is the ultra efficient support service
--------------------------------------------------------------------------------
An amazingly well built product -- we rarely if ever had reliability issues -- the Twilio Functions were an especially useful post-purchase feature discovery -- so much so that we still use that even though we don't do any texting.  We also sometimes use FracTEL, since they beat Twilio on pricing 3:1 for 1-800 texts *and* had MMS 1-800 support long before Twilio. 
--------------------------------------------------------------------------------
I absolutely love using Twilio, have had zero issues in using the SIP and text messaging on the platform.
--------------------------------------------------------------------------------
Authy by Twilio is a run-of-the-mill 2FA app. There's nothing special about it. It works when you're not switching your hardware.
--------------------------------------------------------------------------------
We've had great experience with Twilio. Our users sign up for text notification and we use Twilio to deliver them information. That experience has been well-received by customers. There's more to Twilio than that but texting is what we use it for. The system barely ever goes down and always shows us accurate information of our usage.
--------------------------------------------------------------------------------

...and so on.

Answer 2

I have been scraping many types of websites and I think in the world of scraping, there are roughly 2 types of websites.

The first one is "URL-based" websites (i.e. you send request with URL, the server responds with HTML tags from which elements can be directly extracted), and the second one is "JavaScript-rendered" websites (i.e. the response you only get is the javascript and you can only see HTML tags after it is run).

In former's cases, you can freely navigate through the website with bs4. But in the latter's cases, you cannot always use URLs as a rule of thumb.

The site you are going to scrape is built with Angular.js, which is based on client-side rendering. So, the response you get is the JavaScript code, not HTML tags with page content in it. You have to run the code to get the content.

About the code you introduced:

req = Request(softwareadvice, headers={'User-Agent': 'Mozilla/5.0'})
    
web_byte = urlopen(req, timeout=10).read() # response is javascript, not page content you want...
    
webpage = web_byte.decode('utf-8')

All you can get is the JavaScript code that must be run to get HTML elements. That is why you get the same pages(response) every time.

So, what to do? Is there any way to run JavaScript within bs4? I guess there aren't any appropriate ways to do this. You can use selenium for this one. You can literally wait until the page fully loads, you can click buttons and anchors, or get page content at any time.

Headless browsers in selenium might work, which means you don't have to see the controlled browser opening on your computer.

Here are some links that might be of help to you.

scrape html generated by javascript with python

https://sadesmith.com/2018/06/15/blog/scraping-client-side-rendered-data-with-python-and-selenium

Thanks for reading.