I am currently working on scraping similarweb datas using Scrapy Python in a virtual environment.
I have this example site : "https://www.similarweb.com/fr/website/golfgenius.com", and I want to scrape the number of visitors.
Unfortunately, divs and span are relatives so they won't have specific name you can scrape easily.
It means that if you wanna do a Xpath request you are going to have multiple answers everytime, unless you get a little more accurate.
So the span/text() I want is inside some span with relatives class names. (see screenshot)
The yellow highlighted expression is what i want, here is my Xpath expression yet :
//span[@class='engagementInfo-value engagementInfo-value--large u-text-ellipsis']/span[@class='engagementInfo-valueNumber js-countValue']/text()
On browser it highlight only one span(which is good) and the right one, so everything cool. but if I try this expression inside scrapy shell for example, it returns me an empty list.
What am I doing wrong ? Is it not the way to get a text inside a span?
thank you if you wanna help !