Using 'Requests' python module for POST request, receiving response as if it were GET

Question

So I am trying to make a script that checks a reservation availability of a bus. The starting link for this is https://reservation.pc.gc.ca/.

In the reserve box the following needs to be selected:

Reservation: Day Use (Guided Hikes, Lake O’Hara Bus)
Park: Yoho-Lake O'Hara
Arrival Date: Jun 16
Party Size: 2

When those options are entered, it takes you to the following page: https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar

It is my understanding that if I send a POST request to that second link with the correct data it should return the page I'm looking for

If I look in the dev tools network info when I select the correct parameters the form data is:

__EVENTTARGET:

__EVENTARGUMENT: __VIEWSTATE: -reallllly long string-

__VIEWSTATEGENERATOR: 8D0E13E6

ctl00$MainContentPlaceHolder$rdbListReservationType: Events

ddlLocations: 213a1bc9-9218-4e98-9a7f-0f209008e437**

ddlArrivalMonth: 2017-06-16

ddlArrivalDay: 19

ddlNights: 1

ddlDepartureMonth:

ddlDepartureDay:

ddlEquipment:

ddlEquipmentSub:

ddlPartySize:2

ctl00$MainContentPlaceHolder$chkExcludeAccessible: on

ctl00$MainContentPlaceHolder$imageButtonCalendar.x: 64

ctl00$MainContentPlaceHolder$imageButtonCalendar.y: 56

So the code I wrote is:

import requests

payload = {
        '__EVENTTARGET': '',
        '__EVENTARGUMENT': '',
        '__VIEWSTATE':-reallly long string-,
        '__VIEWSTATEGENERATOR': '8D0E13E6',
        'ctl00$MainContentPlaceHolder$rdbListReservationType': 'Events',
        'ddlLocations': '213a1bc9-9218-4e98-9a7f-0f209008e437',
        'ddlArrivalMonth': 2017-06-16,
        'ddlArrivalDay': 19,
        'ddlNights': 1,
        'ddlDepartureMonth': '',
        'ddlDepartureDay': '',
        'ddlEquipment': '',
        'ddlEquipmentSub': '',
        'ddlPartySize': 2,
        'ctl00$MainContentPlaceHolder$chkExcludeAccessible': 'on',
        'ctl00$MainContentPlaceHolder$imageButtonCalendar.x': 64,
        'ctl00$MainContentPlaceHolder$imageButtonCalendar.y': 56
        }

r = requests.get(r"https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar", data=payload)

print r.text

r.text ends up just being the second link as if no parameters were entered - as if I just sent a normal GET request to the link. I tried turning the payload values that are integers into strings, I tried removing the empty key:value pairs. No luck. Trying to figure out what I'm missing.

With the indentions the way they are, the code you posted would not run. Try to create a proper [minimal working example](https://stackoverflow.com/help/mcve) that can be executed without errors. — Arne, May 17 '17 at 13:23
Sorry, this is my first SO post, that is not the indentation in my python script. Let me try to edit it to display correctly. — Tennyx, May 17 '17 at 13:28

score 1 · Accepted Answer · edited May 23 '17 at 12:26

It looks to me like there are 2 things going on:

@errata was correct, and this should be a POST request. You're about halfway there.
What I noticed though is that it seems to post the form data to Home.aspx and the URL that you see after submitting the form is the result of that processing and subsequent redirect.

You might try posting the form data as json to ./Home.aspx.

I found through Postman that this nearly worked, but I had to specify the content-type in order to get the proper results.

If you need to know how to add header and body instructions to the .post() method, it looks like there is a good example (though perhaps slightly outdated) here: adding header to python request module

Also, fwiw, check out Postman. If you're both inexperienced with requests and with doing it in Python, at least this may lesson some of the burden of trial and error.

This is it, thanks so much! Included headers in the request and it worked like a charm. — Tennyx, May 18 '17 at 15:55

errata · Answer 2 · 2017-05-17T14:30:36.597

You are using

r = requests.get(r"https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar", data=payload)

instead of

r = requests.post(r"https://reservation.pc.gc.ca/Yoho-LakeO'Hara?Calendar", data=payload)

Digging a bit deeper in your problem, I found out that the URL you are calling is actually redirecting to a different URL (returning HTTP response 302):

$ curl -I "https://reservation.pc.gc.ca/Yoho-LakeO'Hara"
HTTP/1.1 302 Found
Cache-Control: private
Content-Length: 77273
Content-Type: text/html; charset=utf-8
Location: https://reservation-pc.fjgc-gccf.gc.ca/GccfLanguage.aspx?lang=eng&ret=https%3a%2f%2freservation.pc.gc.ca%3a443%2fYoho-LakeO%27Hara
Server: Microsoft-IIS/8.0
Set-Cookie: ASP.NET_SessionId=qw4p4e2zxjxx0c2zyq014p45; path=/; secure; HttpOnly
Set-Cookie: CookieLocaleName=en-CA; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
X-Frame-Options: SAMEORIGIN
Date: Wed, 17 May 2017 14:22:53 GMT

However, following the Location from response results also in 302:

$ curl -I "https://reservation-pc.fjgc-gccf.gc.ca/GccfLanguage.aspx?lang=eng&ret=https%3a%2f%2freservation.pc.gc.ca%3a443%2fYoho-LakeO%27Hara"
HTTP/1.1 302 Found
Cache-Control: private
Content-Length: 179
Content-Type: text/html; charset=utf-8
Location: https://reservation.pc.gc.ca:443/Yoho-LakeO'Hara?gccf=true
Server: Microsoft-IIS/8.0
Set-Cookie: ASP.NET_SessionId=rbcuvexfg4fb340ixtcjd1qy; path=/; secure; HttpOnly
Set-Cookie: _gc_lang=eng; domain=.fjgc-gccf.gc.ca; path=/; secure; HttpOnly
X-Powered-By: ASP.NET
X-Frame-Options: SAMEORIGIN
Date: Wed, 17 May 2017 14:24:55 GMT

All this probably results in Requests transforming your POST into GET in the end...

This did not do the trick unfortunately, still returning the page as if I was visiting the link directly. — Tennyx, May 17 '17 at 13:35
Thanks for digging further. So is there any way, in your opinion, to receive the correct POST response I'm looking for? Or does the redirect screw everything up? — Tennyx, May 17 '17 at 15:55
To be perfectly hones, I never came across such problem in my life, so I cannot tell you at this particular moment :) I can try to check it out a bit later today... — errata, May 17 '17 at 16:00
Ok, if I add allow_redirects=False to the .post function it gets me one step closer. It now gets me to the scheduling page, just at the wrong dates! — Tennyx, May 17 '17 at 17:38

Using 'Requests' python module for POST request, receiving response as if it were GET

2 Answers2