Learning to scrape with Selenium and Python

Question

I'm learning to scrape with selenium, but I'm having trouble connecting to this site 'http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720'

it does not load the content of the site

I would like to learn how to connect to this site to request images and data

my code is simple because I'm learning, I looked for ways to make the connection but without success

from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile

ff_profile = FirefoxProfile()
ff_profile.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36")
driver = webdriver.Firefox(firefox_profile = ff_profile)
driver.get('http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720')
time.sleep(5)
campo_busca = driver.find_elements_by_id('of132')
print(campo_busca)

apply the path of gecko driver for initialize browser – Manali Kagathara Jan 20 '20 at 13:19 — Manali Kagathara, Jan 20 '20 at 13:19

DebanjanB · Accepted Answer · 2020-01-20T14:08:19.483

As the the desired element is within an <iframe> so to invoke extract the src attribute of the desired element you have to:

Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired visibility_of_element_located().

You can use the following Locator Strategies:

driver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get('http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720')
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='CamosIFId' and @name='CamosIF']")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//img[@id='of132']"))).get_attribute("src"))

However as in one of the comments @google mentioned, it seems the browsing experiance is better with ChromeDriver / Chrome and you can use the following solution:

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720')
WWebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#CamosIFId[name='CamosIF']")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "img#of132"))).get_attribute("src"))

Note : You have to add the following imports :

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Console Output:

https://www.festo.com/cfp/camosHtml/i?SIG=0020e295a546f45d9acb6844231fd8ff31ca817a_64_64.png

Here you can find a relevant discussion on Ways to deal with #document under iframe

I already gave a positive vote, but because I am new to the community, says that my vote does not calculate on the screen — jefferson lopes da silva, Jan 23 '20 at 14:05
@jeffersonlopesdasilva Thanks, your upvote is calculated/registered now :) Happy learning — DebanjanB, Jan 23 '20 at 14:06

score 0 · Answer 2 · answered Jan 20 '20 at 13:26

try this for more information here

 FIREFOX_DRIVER_PATH = "your_geckodriver_path"
 firefox_options = FirefoxOptions()
 firefox_options.headless = True

 # set options as per requirement for firefox
 firefox_options.add_argument("--no-sandbox")
 firefox_options.add_argument("--disable-setuid-sandbox")
 firefox_options.add_argument('--disable-dev-shm-usage')
 firefox_options.add_argument("--window-size=1920,1080")
 driver = webdriver.Firefox(firefox_options=firefox_options, executable_path=FIREFOX_DRIVER_PATH)
 driver.get('http://www.festo.com/cat/it_it/products_VUVG_SCurrentPartNo=8043720')
 time.sleep(5)

 campo_busca = driver.find_elements_by_id('of132')

 print(campo_busca)

score 0 · Answer 3 · answered Jan 20 '20 at 13:37

download the driver from this link and place it a folder and copy the complete path and paste below

 FIREFOX_DRIVER_PATH = "driver_path"

 firefox_options = FirefoxOptions()

 #only if you dont want to see the gui else make is false or comment
 firefox_options.headless = True

 driver = webdriver.Firefox(firefox_options=firefox_options, executable_path=FIREFOX_DRIVER_PATH)
 driver.get('http://www.festo.com/cat/it_it/products_VUVG_SCurrentPartNo=8043720')
 time.sleep(3)

 campo_busca = driver.find_elements_by_id('of132')

 print(campo_busca)

in my opinion and personal preference is to use chrome instead of firefox. — i_am_deesh, Jan 20 '20 at 13:38

Learning to scrape with Selenium and Python

3 Answers3