1

I am trying to web scrape coinmarketcap.com using Selenium, but I can only retrieve the first 10 altcoins on the list. I read that //div[contains(concat(' ', normalize-space(@class), ' '), 'class name')] should do the trick, but it is not working. Can someone help me? I am also aware that coinmarketcap as an api, but I just wanted to try another way.


driver = webdriver.Chrome(r'C:\Users\Ejer\PycharmProjects\pythonProject\chromedriver')
driver.get('https://coinmarketcap.com/')

Crypto = driver.find_elements_by_xpath("//div[contains(concat(' ', normalize-space(@class), ' '), 'sc-16r8icm-0 sc-1teo54s-1 lgwUsc')]")
#price = driver.find_elements_by_xpath('//td[@class="cmc-link"]')
#coincap = driver.find_elements_by_xpath('//td[@class="DAY"]')

CMC_list = []
for c in range(len(Crypto)):
    CMC_list.append(Crypto[c].text)
print(CMC_list)

driver.close()
DebanjanB
  • 118,661
  • 30
  • 168
  • 217
Dfhaa_DK
  • 111
  • 7

1 Answers1

0

To retrieve the first 10 altcoins on the list you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR and get_attribute("innerHTML"):

    driver.get('https://coinmarketcap.com/')
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table.cmc-table tbody tr td > a p[color='text']")))[:10]])

  • Using XPATH and text attribute:

    driver.get('https://coinmarketcap.com/')
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[contains(@class, 'cmc-table')]//tbody//tr//td/a//p[@color='text']")))[:10]])

  • Console Output:

    ['Bitcoin', 'Ethereum', 'XRP', 'Tether', 'Litecoin', 'Bitcoin Cash', 'Chainlink', 'Cardano', 'Polkadot', 'Binance Coin']

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
DebanjanB
  • 118,661
  • 30
  • 168
  • 217
  • Thank you again for your input, but I have tried to retrieve more than the first 10 alt coins. When I use your code, and change the 10 to for example 20, I only get 11 coins. Another thing, how will I set it up if I want to store the values in an empty list. I also want to retrieve other data like coin market cap, price, etc... – Dfhaa_DK Dec 13 '20 at 13:11
  • @Dfhaa_DK Sounds to be a different requirement all together. Can you raise a new question for your new requirement please? – DebanjanB Dec 13 '20 at 16:18
  • I made a new post ifyou can find it – Dfhaa_DK Dec 13 '20 at 19:13