Wait element to be present with selenium on scores website

Question

I want to scrape this page https://www.sofascore.com/betting-tips-today I created code with selenium but don't work:

from bs4 import BeautifulSoup
import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome(executable_path=r"C:/chromedriver.exe") #, options=options)
url = "https://www.sofascore.com/betting-tips-today"

driver.get(url)

WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.CSS_SELECTOR,'.fsFjfS'))) #wait until the element be present


time.sleep(5)
soup = BeautifulSoup(driver.page_source, 'html.parser')

It return this error:

raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message:

I tried without Wait but don't return anything.

what are you trying to pull? Theres an api so it's possible you don't need selenium, — chitown88, Feb 17 '21 at 13:04

score 0 · Answer 1 · answered Feb 17 '21 at 12:18

You were close enough. Instead of presence_of_element_located() you need to induce WebDriverWait for the visibility_of_element_located() of a visible element and you can use the following Locator Strategy:

Code Block:

driver.get("https://www.sofascore.com/betting-tips-today")
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[class^='Content__PageContainer-sc-']")))
soup = BeautifulSoup(driver.page_source, 'html.parser')
print(soup)

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Console Output:

<html data-triggered="true" lang="en">

<head>
  <meta content="width=device-width, initial-scale=1, shrink-to-fit=no" name="viewport" />
  <meta content="#0057a3" name="theme-color" />
  <link href="/static/favicon.ico" rel="icon" />
  <meta content="SofaScore.com LiveScore" name="author" />
  <meta content="app-id=1176147574/" name="apple-itunes-app" />
  <meta content="53e30a9a6b4c2d9e" name="yandex-verification" />
  <link href="https://api.sofascore.com" rel="preconnect dns-prefetch" />
  <link href="https://fonts.googleapis.com" rel="preconnect dns-prefetch" />
  <link href="https://fonts.gstatic.com" rel="preconnect dns-prefetch" />
  <link href="https://www.googletagmanager.com" rel="preconnect dns-prefetch" />
  <link href="https://securepubads.g.doubleclick.net" rel="preconnect dns-prefetch" />
  <link href="https://polyfill.io" rel="preconnect dns-prefetch" />
  <link href="/static/manifest.json" rel="manifest" />
  <link href="/static/favicon.ico" rel="shortcut icon" />
  <link href="/static/favicon.ico" rel="mask-icon" />
  <link href="/static/images/apple-icon-180x180.png" rel="apple-touch-icon-precomposed" sizes="180x180" />
  <link href="/static/images/apple-icon-152x152.png" rel="apple-touch-icon-precomposed" sizes="152x152" />
  <link href="/static/images/apple-icon-144x144.png" rel="apple-touch-icon-precomposed" sizes="144x144" />
  <link href="/static/images/apple-icon-120x120.png" rel="apple-touch-icon-precomposed" sizes="120x120" />
  <link href="/static/images/apple-icon-114x114.png" rel="apple-touch-icon-precomposed" sizes="114x114" />
  <link href="/static/images/apple-icon-76x76.png" rel="apple-touch-icon-precomposed" sizes="76x76" />
  <link href="/static/images/apple-icon-72x72.png" rel="apple-touch-icon-precomposed" sizes="72x72" />
  <link href="/static/images/apple-icon-57x57.png" rel="apple-touch-icon-precomposed" />
  <script async="" src="https://www.googletagmanager.com/gtag/js?id=G-HNQ9P9MGZR&amp;l=dataLayer&amp;cx=c" type="text/javascript"></script>
  <script async="" src="https://www.google-analytics.com/analytics.js" type="text/javascript"></script>
  <script async="" src="https://www.google-analytics.com/gtm/js?id=GTM-M8Z5Q48&amp;t=gtm5&amp;cid=640338494.1613564014&amp;aip=true" type="text/javascript"></script>
  <script async="" src="https://www.google-analytics.com/analytics.js" type="text/javascript"></script>
  <script async="" src="https://connect.facebook.net/signals/config/759407694841706?v=2.9.33&amp;r=stable"></script>
  <script async="" src="https://connect.facebook.net/en_US/fbevents.js"></script>
  <script async="" crossorigin="anonymous" src="https://connect.facebook.net/en_US/sdk.js?hash=434141e6be23d41ce5b145e37bc66356&amp;ua=modern_es6"></script>
  <script async="" src="https://www.googletagmanager.com/gtm.js?id=GTM-54Q2FDL"></script>
  <script charset="UTF-8" data-document-language="true" data-domain-script="bbfc9ddf-194b-411a-8caa-3d06175a3e3e" src="https://cookie-cdn.cookiepro.com/scripttemplates/otSDKStub.js" type="text/javascript"></script>
  <script type="text/javascript">
    function OptanonWrapper() {}
  </script>
  <script>
    window.dataLayer = window.dataLayer || [];
    (function(w, d, s, l, i) {
      w[l] = w[l] || []
      w[l].push({
        'gtm.start': new Date().getTime(),
        event: 'gtm.js'
      })
      var f = d.getElementsByTagName(s)[0],
        j = d.createElement(s),
        dl = l != 'dataLayer' ? '&l=' + l : ''
      j.async = true
      j.src = 'https://www.googletagmanager.com/gtm.js?id=' + i + dl
      f.parentNode.insertBefore(j, f)
    })(window, document, 'script', 'dataLayer', 'GTM-54Q2FDL')

    function gtag() {
      window.dataLayer.push(arguments)
    }
  </script><noscript><iframe height="0" src="https://www.googletagmanager.com/ns.html?id=GTM-54Q2FDL" style="display:none;visibility:hidden" width="0"></iframe></noscript>
  <script defer="" src="/static/scripts/isMobileDevice.js"></script>
  <script defer="" src="/static/scripts/deferredStyles.js"></script>

@khaledkoubaa Your main issue was **TimeoutException** which I have addressed within this answer. You haven't mentioned about any data and it sounds like a different requirement all together. Can you raise a new question as per your new requirement? Stackoverflow contributors will be happy to help you out. — DebanjanB, Feb 17 '21 at 12:51

Wait element to be present with selenium on scores website

1 Answers1