div Web Scrape on site output returns None

Question

I was trying to web scrape the past multipliers on https://roobet.com/crash . But When I try to run the program there is no results. What's the problem? Code is below

from bs4 import BeautifulSoup
import requests

source = requests.get('https://roobet.com/crash').text

soup = BeautifulSoup(source, 'lxml')

title = soup.find('title').text

results = soup.find_all('div', attrs={'class': 'jss75'})

for i in results:
    multi = i.find('span', attrs={"class":"jss75"})
    if multi is not None:
        print('multi:', multi).text

Thanks for the help!

Does this answer your question? [scrape html generated by javascript with python](https://stackoverflow.com/questions/2148493/scrape-html-generated-by-javascript-with-python) — esqew, Oct 21 '20 at 11:27
I was trying to get it done with only python. not using javascript too so.. — Lakshan Costa, Oct 21 '20 at 11:36
Did you even read the linked duplicate? The suggestion isn’t to use JavaScript instead - the issue is that the content on the page is probably generated by JavaScript - something that `requests` nor `BeautifulSoup` has the ability to interpret, render, and then query against. — esqew, Oct 21 '20 at 11:40

score 1 · Accepted Answer · edited Oct 21 '20 at 11:48

Take a look at the returned source and you may understand why you cannot find the result you are looking for.

<!DOCTYPE html>
<html lang="en">

  <head>
    <!-- Google Tag Manager -->
    <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
    new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
    j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
    'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
  })(window,document,'script','dataLayer','GTM-563FCQS');</script>
    <!-- End Google Tag Manager -->
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="preconnect" href="https://fonts.googleapis.com/" crossorigin>
    <title>Roobet | Crypto's Fastest Growing Casino</title>
    <meta name="description" content="Roobet, crypto's fastest growing casino. Hop on in, chat to others and play exciting games - Come and join the fun!">
    <base href="/">
    <meta name="theme-color" content="#191b31" />
    <link rel="icon" type="image/png" href="images/favicon.png">
    <link rel="manifest" href="/manifest.json" />
    <script src="https://cdn.onesignal.com/sdks/OneSignalSDK.js" async ></script>
    <script src="https://maps.googleapis.com/maps/api/js?key=AIzaSyCXI19SE-ZWv_ZyW7gGMzCTf4TGfOA3Sdk&libraries=places"></script>
    <script src="https://tekhou5-dk2.pragmaticplay.net/gs2c/common/js/lobby/GameLib.js" />
    <script>
      var OneSignal = window.OneSignal || [];
      OneSignal.push(function() {
        OneSignal.init({
          appId: "29c72f64-e7e6-408c-99b2-d86a84c6a9cb",
          notifyButton: {
            enable: false,
            autoResubscribe: true,
          },
          welcomeNotification: {
            disable: true
          }
        });
      });
    </script>
  <link href="0.aafac69fdc9eee2864e9.css" rel="stylesheet"><link href="app.aafac69fdc9eee2864e9.css" rel="stylesheet"></head>
  <body>
    <!-- Google Tag Manager (noscript) -->
    <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-563FCQS"
    height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
    <!-- End Google Tag Manager (noscript) -->
    <div id="root"></div>
    <div id="modalRoot"></div>
    <div id="loader">
      <div class="loaderLogo">
        <img src="/images/logo.svg" />
      </div>
    </div>
  <script type="text/javascript" src="vendors.37e373e3e07a018e2e49.bundle.js"></script><script type="text/javascript" src="locale.9c51b6a88780f5e87cd3.bundle.js"></script><script type="text/javascript" src="app.7bee5f919f764925b254.bundle.js"></script></body>
  <script>(function(){var w=window;var ic=w.Intercom;if(typeof ic==="function"){ic('reattach_activator');ic('update',intercomSettings);}else{var d=document;var i=function(){i.c(arguments)};i.q=[];i.c=function(args){i.q.push(args)};w.Intercom=i;function l(){var s=d.createElement('script');s.type='text/javascript';s.async=true;s.src='https://widget.intercom.io/widget/gcr7bzde';var x=d.getElementsByTagName('script')[0];x.parentNode.insertBefore(s,x);}if(w.attachEvent){w.attachEvent('onload',l);}else{w.addEventListener('load',l,false);}}})()</script>
  <script src="https://intaggr.softswiss.net/public/sg.js"></script>
  <script type="text/javascript" src="https://www.google.com/recaptcha/api.js?render=6LdG97YUAAAAAHMcbX2hlyxQiHsWu5bY8_tU-2Y_"></script>
  <script type="text/javascript">
    if (typeof window.grecaptcha !== 'undefined') {
      grecaptcha.ready(function() {
        grecaptcha.execute('6LdG97YUAAAAAHMcbX2hlyxQiHsWu5bY8_tU-2Y_', {action: 'homepage'});
      })
    }
  </script>
</html>

When you inspect element on then website the div containing the multipliers that your looking for is there. <div class="jss75"> however in the above source you can see the body of the HTML file contains is script imports which generates the HTML you are looking for.

Some of the data you are looking for might be contained in the other files retrieved by the website (open dev tools, go to the network tab and reload). The recentNumbers file looks like it might contain what you need (I'm not familiar with the website) it contains many data points ladled as crashPoint which look like they are the multipliers you are looking for.

https://api.roobet.com/crash/recentNumbers

If this isn't what your looking for i can take a deeper look, or as i say checkout the network tab and all the data it pulls in.

div Web Scrape on site output returns None

1 Answers1