Diffbot docs suggest that to set custom headers, including Cookies
,I simply add the X-Forward
prefix to the header.
For example I do the following
cookie='SportsDirect_AnonymousUserCurrency=CNY'
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36'
headers = {'X-Forward-User-Agent': user_agent,'X-Forward-Cookie': cookie}
I then make the request with my client to the URL https://www.sportsdirect.com/adidas-ace-17-1-mens-fg-football-boots-203193?colcode=20319312, and I can see in Wireshark that the client correctly passed the X-Forward
headers:
However the target site responds in USD still, not CNY.
This is contrast to a simple cURL:
curl 'https://www.sportsdirect.com/adidas-ace-17-1-mens-fg-football-boots-203193?colcode=20319312' -H 'Cookie: SportsDirect_AnonymousUserCurrency=CNY' --compressed | grep -A 5 pdpPrice
which shows the price in accord with the "SportsDirect_AnonymousUserCurrency" as it should.
However, I finally tested using diffbot to make a request to https://webhook.site, which records any requests along with headers made to a special address it provides you.
clearly Diffbot did indeed forward my headers, so it's baffling why it would not get the right currency, when cURL does.