1

I am trying to scrape a website using selenium. My goal is to give an input string like "youtube" and my script will go ahead and click the links in that website that contain that string or similar strings. To elaborate on that, if the input string is "youtube" but the link has a friendly name "Youtube" or "You.tube" or "Youtube Go", I want the script to select/click it.

I have tried to use

  1. driver.find_element_by_link_text("youtube").click()

but it gives error for links that are not an exact match (its also case sensitive so even if the link is "Youtube" it will give error)

  1. driver.find_element_by_partial_link_text("youtube").click()

but this also gives error and the only difference with the first option is if I provide a shorter input than the link name and even then, the shorter input needs to be an exact match with a part of the link name.

Any advice is much appreciated!

DebanjanB
  • 118,661
  • 30
  • 168
  • 217
Pericles Faliagas
  • 606
  • 7
  • 21

2 Answers2

1

You can use either of the following Locator Strategies:

  • Using xpath and contains(translate()) v1:

    element = driver.find_element_by_xpath("//*[contains(translate(., 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz'), 'youtube')]")

  • Using xpath and contains(translate()) v2:

    element = driver.find_element_by_xpath("//*[contains(translate('youtube','abcdefghijklmnopqrstuvwxyz','ABCDEFGHIJKLMNOPQRSTUVWXYZ')]")
DebanjanB
  • 118,661
  • 30
  • 168
  • 217
  • Thank you for the answer! The only problem with this solution is that I will have to manually input alternatives every time. And I will want to use different inputs. So, this mapping process will require maintenance effort. Unless I misunderstood the proposed solution. – Pericles Faliagas Dec 16 '20 at 12:24
  • 1
    @PericlesFaliagas Creating a function and passing the required text as an argument would do the job for you. – DebanjanB Dec 16 '20 at 12:27
  • 1
    You can use '{}'.format(text) to make it use any value. – Arundeep Chohan Dec 16 '20 at 12:27
  • @arundeepchohan Yeah, that's true. OP can use, `%s` or `{}` anything within the xpath. – DebanjanB Dec 16 '20 at 12:29
1

The first thing I notice here is that you want to go through all matching links found in a page, so I'll help you do that correctly first.

The find_element_by_partial_link_text() method will only return the first link that is found to match your search term. Instead, you can store the results of the find_elements_by_partial_link_text() method in a list. Selenium Documentation on locating elements can be found here

For example:

links = []
for link in driver.find_elements_by_partial_link_text("Youtube"):
    links.append(link) # This will append the object itself
    # To append the actual link:
    # links.append(link.get_attribute('href'))

Code Source

Now, onto your main question. How can you find all links with similar partial text? From what I can gather, this isn't possible with Selenium by default. You'd have to iterate through all the possible variations of your term (eg. using string.lower(), string.upper() and other various methods to maximise the results returned) and run the find_elements code each time.

Another option would be to rewrite the method in the Selenium package to check the lowercase version of the link text it has found against the argument you pass it. Personally, I can't come up with any other strategies but I'm only an intermediate programmer, and have only used Selenium a couple of times. Hopefully someone will come along with a much better answer to do exactly what you want, however I hope the information I have provided will be of some use for now.

4RJ
  • 60
  • 6