How to get item url after finding with css selector using selenium

Question

I'm trying to extract a url(link) from a webpage, I used "find_element_by_css_selector" to get the item i want. This item has a url in it. How do I extract this url.

I have tried:

prod_item = browser.find_elements_by_css_selector('div.col-lg-2')
print(prod_item[0].get_attribute('href'))

But I'm getting "None" as output. I would love to use the css_selector because there are many similar items on the page and 'div.col-lg-2' is the attribute that is common to them all. How do solve this problem and get the link?

Here is the full code now:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

url = 'https://auctionmaxx.com/Browse?page=0'

browser = webdriver.Firefox()
browser.get(url)


prod_item = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.col-lg-2[href]")))

print(prod_item[4].get_attribute('href'))

DebanjanB · Accepted Answer · 2020-07-28T21:40:06.027

1

To print the value of the href attribute you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

browser.get("https://auctionmaxx.com/Browse?page=0")
prod_item = WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.col-lg-2>div a")))
print(prod_item[0].get_attribute('href'))

Using CSS_SELECTOR in a single line:

browser.get("https://auctionmaxx.com/Browse?page=0")
print(WebDriverWait(browser, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.col-lg-2>div a")))[0].get_attribute('href'))

Console Output:

https://auctionmaxx.com/Listing/Details/321939965/NEW-PUREX-LAUNDRY-DETERGENT-924L

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Jul 28 '20 at 21:40

answered Jul 27 '20 at 23:40

DebanjanB

118,661
30
168
217

Thank you. I have edited my code but it still does not work. I have edited the question with the full code. Can you take a look at it? – ebere Jul 28 '20 at 12:59
1

@ebere Generally `
` won't contain `href` attribute, but still my answer was based on your research. Now as you shared the [link](https://auctionmaxx.com/Browse?page=0) it seems the grand children `` tag contains the `href` attribute. Checkout the updated answer and let me know the status.
– DebanjanB Jul 28 '20 at 21:35
1

Yeah, this is it. This works perfectly. Thank you very much – ebere Jul 29 '20 at 12:41
@ebere Glad to be able to help you. [Vote up questions and answers](https://stackoverflow.com/help/privileges/vote-up) that you found helpful. See [Why is voting important](https://stackoverflow.com/help/why-vote). – DebanjanB Jul 29 '20 at 12:42

score 0 · Answer 2 · answered Jul 27 '20 at 23:12

0

This must work,

  browser.find_elements_by_css_selector('a').get_attribute('href')

answered Jul 27 '20 at 23:12

Induraj PR

212
2
7

There are other elements on the page that are links and also have selector 'a'. I'm only looking to get the 'href' in elements that have selector 'div.col-lg-2' – ebere Jul 27 '20 at 23:17
could you write explicitly how to full line/block looks like? – Induraj PR Jul 27 '20 at 23:38

icaine · Answer 3 · 2020-07-27T23:39:39.193

The code seems legit so at first i would try to check raw html source (with curl or browser with disabled JS). Maybe href attribute does not contain any url in the moment you are trying to get it's value and the value is in other attribute (e.g. in data-href) or it is loaded dynamically via ajax. Anyway, check docs for Waits or this link where you can find some tips how to wait for particular content until it's available.

How to get item url after finding with css selector using selenium

3 Answers3