Extracting a date from webpage using selenium webdriver

Question

For one of my projects, i am trying to run a looped script (for and while) which would run until a particular predefined date value is reached. My code is aimed at extracting the flight data of an aircraft between two specified dates. My source of information is the publicly available flight tracker, <flightradar24.com>, for which i also have a business subscription. For example - https://www.flightradar24.com/data/aircraft/d-abyt When trying to collect a list of flights from the page, i want to be able to read a date and stop the loop if it is over the specified date.

The html source looks like the following: In this case the 22 Mar 2020 is what I want to read and use to compare.

So far I've tried the following to try and extract the date.

element = driver.find_element_by_class_name('w40 hidden-xs hidden-sm')

print(driver.find_element_by_xpath("//time[@class='hidden-xs hidden-sm']").text)

print(driver.find_element_by_xpath("//time[@class='w40 hidden-xs hidden-sm']").get_attribute("data-time-format"))

and

element = driverfox.find_element_by_xpath('// time[ @class ="data-time-format"] / @ datetime'.__getattribute__("data-time-format"))

Thank you in advance for your advice !!

Error when executing Debanjan's CSS suggestion.

score 0 · Answer 1 · answered Nov 28 '20 at 17:44

0

Use the following to print out the element's text.

print(driver.find_element_by_css_selector('td.hidden-xs.hidden-sm').text)

Also induce webdriver waits

elem=WebDriverWait(driver, 10).until(EC.presence_of_element_located(((By.CSS_SELECTOR, "td.hidden-xs.hidden-sm"))).text
print(elem)

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

answered Nov 28 '20 at 17:44

Arundeep Chohan

6,219
4
6
22

Thank you for your answer. I tried your first suggestion and it doesn't print anything. For your second suggestion, it results in an error "'presence_of_element_located' object has no attribute 'text'". How can we extract the date if it says no text attribute found? – Void Dec 02 '20 at 17:04

score 0 · Answer 2 · answered Nov 28 '20 at 21:05

0

To print the date i.e. 22 Mar 2020 you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and text attribute:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tr.data-row[data-timestamp] td[data-timestamp][data-time-format='DD MMM YYYY']"))).text)

Using XPATH and get_attribute():

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//tr[@class='data-row']//td[@data-timestamp and @data-time-format='DD MMM YYYY']"))).get_attribute("innerHTML"))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

answered Nov 28 '20 at 21:05

DebanjanB

118,661
30
168
217

Thank you Debanjan ! I tried it but it doesn't work. – Void Dec 01 '20 at 08:36
@Void Update the main question with the error you are seeing. – DebanjanB Dec 01 '20 at 08:44
I have updated it with the error. Thank you ! – Void Dec 02 '20 at 15:44
@Void I am still unable to find the relevant string `22 Mar 2020` within https://www.flightradar24.com/data/aircraft/d-abyt I can find **2020** only in _Camtasia®️ 2020 Makes it Simple to Records and Create Professional Videos. Buy it Today!_ and _© 2020 Flightradar24 AB_ – DebanjanB Dec 02 '20 at 15:50
I have added another screenshot. Did you try to attach something? I see something from Camtasia – Void Dec 03 '20 at 17:40
@Void The _Camtasia®️ 2020 Makes it Simple to Records and Create Professional Videos. Buy it Today! and © 2020 Flightradar24 AB_ was from a Camtasia ad within the website. – DebanjanB Dec 03 '20 at 19:18

score 0 · Answer 3 · answered Dec 02 '20 at 05:37

0

The problem is you're tryng to match with multiple class names. Xpath isn't great with this. Css selectors are which is why this works well:

driver.find_element_by_css_selector('td.hidden-xs.hidden-sm')

But you could also accomplish this with xpath such as:

driver.find_element_by_xpath("//td[contains(@class,'hidden-xs') and contains(@class, 'hidden-sm')]")

And, as always it's best to introduce a wait before locating an element:

element = WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.XPATH, "//td[contains(@class,'hidden-xs') and contains(@class, 'hidden-sm')]"))
    )

answered Dec 02 '20 at 05:37

DMart

2,198
1
11
17

As suggested, i tried this-----element = WebDriverWait(driver, 10).until( EC.visibility_of_element_located((By.XPATH, "//td[contains(@class,'hidden-xs') and contains(@class, 'hidden-sm')]")) ) print(element.text) ----It gives me no output. Without the element.text, it gives me some selenium web element – Void Dec 04 '20 at 18:13
and you're sure there is text in that field? My quick look at the site shows that sometimes it's an empty placeholder. – DMart Dec 04 '20 at 22:22
Examining/inspecting the date element on the page shows that there is a date value there. How it is stored and how it can be extracted is what i am trying to do ? – Void Dec 06 '20 at 17:04

Extracting a date from webpage using selenium webdriver

3 Answers3