0

Diffbot docs suggest that to set custom headers, including Cookies,I simply add the X-Forward prefix to the header.

For example I do the following

cookie='SportsDirect_AnonymousUserCurrency=CNY'
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36' 
headers = {'X-Forward-User-Agent': user_agent,'X-Forward-Cookie': cookie}

I then make the request with my client to the URL https://www.sportsdirect.com/adidas-ace-17-1-mens-fg-football-boots-203193?colcode=20319312, and I can see in Wireshark that the client correctly passed the X-Forward headers:

enter image description here

However the target site responds in USD still, not CNY.

This is contrast to a simple cURL:

curl 'https://www.sportsdirect.com/adidas-ace-17-1-mens-fg-football-boots-203193?colcode=20319312' -H 'Cookie: SportsDirect_AnonymousUserCurrency=CNY'  --compressed | grep -A 5 pdpPrice

which shows the price in accord with the "SportsDirect_AnonymousUserCurrency" as it should.

However, I finally tested using diffbot to make a request to https://webhook.site, which records any requests along with headers made to a special address it provides you.

enter image description here

clearly Diffbot did indeed forward my headers, so it's baffling why it would not get the right currency, when cURL does.

fpghost
  • 2,462
  • 2
  • 26
  • 46
  • Diffbot uses servers in the USA, so it's possible that your IP is playing a major factor and that's why you're getting defaulted to CNY, not the header itself. Does the site return USA or CNY when trying with curl without a header? – Swader Apr 19 '18 at 05:31
  • My IP is UK. If I cURL without a header I get GBP. If I diffbot without a header I get USD. For me, cURL with the single, `SportsDirect_AnonymousUserCurrency` suffices to change the currency as desired, but as the answer below suggests I need 2 cookies to have diffbot change the currency. – fpghost Apr 19 '18 at 11:35

1 Answers1

1

Unable to reproduce with your curl, does not work that way either. To make it work with both approaches you need to add another cookie value because of a redirect on their end. The full header is:

ChosenSite=www; SportsDirect_AnonymousUserCurrency=CNY;
Swader
  • 10,807
  • 14
  • 46
  • 82
  • Strange that you don't reproduce the cURL, I just tried again and it works fine for me to change between currencies like that with the single currency. Nevertheless, adding the extra cookie, `ChosenSite` does indeed stop the redirect with diffbot and gives me the desired response. Thanks! – fpghost Apr 19 '18 at 11:32