0

Hello I am trying to run a python script on a Apache2 server based on Ubuntu. Here is the configuration of the server from the file 000-default.conf:

<Directory /usr/lib/cgi-bin/>
    Options Indexes FollowSymLinks ExecCGI
    AddHandler cgi-script .cgi .py
    AllowOverride None
    Require all granted
    </Directory>

test.cgi is the name of the python script. It is located in /usr/lib/cgi-bin/

First I run this script and it runs perfectly:

#!/usr/bin/env python3

from math import sqrt
print("Content-type: text/html\n\n")
print("Hello World!\n")
a = sqrt(4)
print(a)

I import math library and it is ok.

Then I tried to add webdriver to navigate through url

#!/usr/bin/env python3

#from selenium import webdriver
print("Content-type: text/html\n\n")
print("Hello World!\n")

driver = webdriver.Firefox()
url = 'https://www.coteur.com/cotes-foot.php'
driver.get(url)

url_links = []
for i in driver.find_elements_by_xpath('//a[contains(@href, "match/cotes-")]'):
    url_links.append(i.get_attribute('href'))

print(len(url_links), '\n')
print(url_links[0], '\n')

The goal is to go to https://www.coteur.com/cotes-foot.php and print the first url of soccer games list.

But the output is bad :

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 webmaster@localhost to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 127.0.0.1 Port 80</address>
</body></html>

I guess that the problem comes from webdriver but I do not know exactly why

DebanjanB
  • 118,661
  • 30
  • 168
  • 217
ahmedaao
  • 321
  • 1
  • 8

2 Answers2

0

The desired element is a dynamic element so to click on the element you have to induce WebDriverWait for the visibility_of_all_elements_located() and print the element with index 0 and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    driver.get('https://www.coteur.com/cotes-foot.php')
print(WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[href^='match/cotes-']")))[0].get_attribute("innerHTML"))

  • Using XPATH:

    driver.get('https://www.coteur.com/cotes-foot.php')
print(WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[contains(@href, 'match/cotes-')]")))[0].get_attribute("innerHTML"))

  • Console Output:

    St. Gallen - Sion

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
DebanjanB
  • 118,661
  • 30
  • 168
  • 217
0

I implement your feedback on my script :

Here is the new code :

#!/usr/bin/env python3

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
print("Content-type: text/html\n\n")
print("Hello World!\n")

driver = webdriver.Firefox()
driver.get('https://www.coteur.com/cotes-foot.php')
print(WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a[href^='match/cotes-']")))[0].get_attribute("innerHTML"))

url_links = []
for i in driver.find_elements_by_xpath('//a[contains(@href, "match/cotes-")]'):
    url_links.append(i.get_attribute('href'))

print(len(url_links), '\n')
print(url_links[0], '\n')

It does not work. I also tried with Xpath same result It doesn't work I have exactly the same output that above

ahmedaao
  • 321
  • 1
  • 8