Python and mechanize login script

Question

Hi fellow programmers!

I am trying to write a script to login into my universities "food balance" page using python and the mechanize module...

This is the page I am trying to log into: http://www.wcu.edu/11407.asp The website has the following form to login:

<FORM method=post action=https://itapp.wcu.edu/BanAuthRedirector/Default.aspx><INPUT value=https://cf.wcu.edu/busafrs/catcard/idsearch.cfm type=hidden name=wcuirs_uri> 
<P><B>WCU ID Number<BR></B><INPUT maxLength=12 size=12 type=password name=id> </P>
<P><B>PIN<BR></B><INPUT maxLength=20 type=password name=PIN> </P>
<P></P>
<P><INPUT value="Request Access" type=submit name=submit> </P></FORM>

From this we know that I need to fill in the following fields: 1. name=id 2. name=PIN

With the action: action=https://itapp.wcu.edu/BanAuthRedirector/Default.aspx

This is the script I have written thus far:

#!/usr/bin/python2 -W ignore

import mechanize, cookielib
from time import sleep

url   = 'http://www.wcu.edu/11407.asp'
myId  = '11111111111'
myPin = '22222222222'

# Browser
#br = mechanize.Browser()
#br = mechanize.Browser(factory=mechanize.DefaultFactory(i_want_broken_xhtml_support=True))
br = mechanize.Browser(factory=mechanize.RobustFactory()) # Use this because of bad html tags in the html...

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# User-Agent (fake agent to google-chrome linux x86_64)
br.addheaders = [('User-agent','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11'),
                 ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
                 ('Accept-Encoding', 'gzip,deflate,sdch'),                  
                 ('Accept-Language', 'en-US,en;q=0.8'),                     
                 ('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.3')]

# The site we will navigate into
br.open(url)

# Go though all the forms (for debugging only)
for f in br.forms():
    print f


# Select the first (index two) form
br.select_form(nr=2)

# User credentials
br.form['id']  = myId
br.form['PIN'] = myPin

br.form.action = 'https://itapp.wcu.edu/BanAuthRedirector/Default.aspx'

# Login
br.submit()

# Wait 10 seconds
sleep(10)

# Save to a file
f = file('mycatpage.html', 'w')
f.write(br.response().read())
f.close()

Now the problem...

For some odd reason the page I get back (in mycatpage.html) is the login page and not the expected page that displays my "cat cash balance" and "number of block meals" left...

Does anyone have any idea why? Keep in mind that everything is correct with the header files and while the id and pass are not really 111111111 and 222222222, the correct values do work with the website (using a browser...)

Thanks in advance

EDIT

Another script I tried:

from urllib import urlopen, urlencode                                           
import urllib2                                                                  
import httplib                                                                  

url = 'https://itapp.wcu.edu/BanAuthRedirector/Default.aspx'                    

myId = 'xxxxxxxx'                                                               
myPin = 'xxxxxxxx'                                                              

data = {                                                                        
            'id':myId,                                                          
            'PIN':myPin,                                                        
            'submit':'Request Access',                                          
            'wcuirs_uri':'https://cf.wcu.edu/busafrs/catcard/idsearch.cfm'      
        }                                                                       

opener = urllib2.build_opener()                                                 
opener.addheaders = [('User-agent','Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11'),
                     ('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
                     ('Accept-Encoding', 'gzip,deflate,sdch'),                  
                     ('Accept-Language', 'en-US,en;q=0.8'),                     
                     ('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.3')]      

request = urllib2.Request(url, urlencode(data))                                 
open("mycatpage.html", 'w').write(opener.open(request))

This has the same behavior...

Nolen Royalty · Answer 1 · 2012-04-13T03:10:11.627

1

# User credentials
br.form['id']  = myId
br.form['PIN'] = myPin

I believe this is the problem line.

Try changing it to

br['id'] = myId
br['PIN'] = myPin

I'm also pretty sure that you don't need br.form.action = 'https://itapp.wcu.edu/BanAuthRedirector/Default.aspx' because you have already selected the form so just calling submit should work, but I could be wrong.

Additionally, I have done a similar task just using urllib and urllib2, so if this doesn't work I will post that code.

Edit: here is the the technique that I used with urllib and urllib2:

import urllib2, urllib

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
urllib2.install_opener(opener)
encoded = urllib.urlencode({"PIN":my_pin, "id":my_id})
f = opener.open('http://www.wcu.edu/11407.asp', encoded)
data = f.read()
f.close()

Edit 2:

>>> b = mechanize.Browser(factory=mechanize.RobustFactory())
>>> b.open('http://www.wcu.edu/11407.asp')
<response_seek_wrapper at 0x10acfa248 whose wrapped object = <closeable_response at 0x10aca32d8 whose fp = <socket._fileobject object at 0x10aaf45d0>>>
>>> b.select_form(nr=2)
>>> b.form
<mechanize._form.HTMLForm instance at 0x10ad0dbd8>
>>> b.form.attrs
{'action': 'https://itapp.wcu.edu/BanAuthRedirector/Default.aspx', 'method': 'post'}

This could be your problem? Not sure.

Edit 3:

Used an html inspector, I think there's a decent chance you need to set 'wcuirs_uir' to 'https://cf.wcu.edu/busafrs/catcard/idsearch.cfm'. I'm 95% sure that will work.

edited Apr 13 '12 at 03:10

answered Apr 13 '12 at 01:17

Nolen Royalty

16,989
4
37
46

Changing br.form['id'] = myId br.form['PIN'] = myPin did not work.... Iv'e tried urllib as well with no luck.... – Perun Apr 13 '12 at 01:38
The script you gave me gives me: urllib2.HTTPError: HTTP Error 500: Internal Server Error – Perun Apr 13 '12 at 01:51
So every time I try using urllib2 (both the one I posted and the one you suggested) I get internal server error (500)... When I use mechanize, I just get the login page back... – Perun Apr 13 '12 at 01:57
@UkrNIXakep also the original url in the second script I posted was wrong(I believe), I have updated it with the url you are using. I think that + the url I just posted might be worth trying. – Nolen Royalty Apr 13 '12 at 01:57
How would I make it submit to cf.wcu.edu/busafrs/catcard/idsearch.cfm in my mech script? – Perun Apr 13 '12 at 01:58
http://stackoverflow.com/questions/4815033/asp-in-python-in-iis-7-is-giving-http-1-1-500-server-error Might be of help on the 500 error. – Nolen Royalty Apr 13 '12 at 01:58
when I send to cf.wcu.edu/busafrs/catcard/idsearch.cfm instead of https://itapp.wcu.edu/BanAuthRedirector/Default.aspx, I get the login page back..... again..... – Perun Apr 13 '12 at 02:02
I did a little reading on submitting to asp stuff like this and it seems pretty complicated, I'm not quite sure yet. – Nolen Royalty Apr 13 '12 at 02:31
Still get 500 error, notice that thats is exactly what I did in my urllib2 script (original post bottom) – Perun Apr 13 '12 at 03:36
not sure what to say, idk if you need to encode that submit value in the first script but I think the problem is with the website. – Nolen Royalty Apr 13 '12 at 03:38
Yeah the website is pretty horrible... But there should be a way to login with a script, right? maybe use perl? I dont understand what is failing.... – Perun Apr 13 '12 at 03:45

score 0 · Answer 2 · answered Sep 23 '14 at 16:10

I suggest the following library: http://docs.python-requests.org/en/latest/

It is a nice and easy library. It has a good documentation. I have used this library to do different kind of scripting, just like the one you are doing.

You need to do something like this:

import requests 

s = requests.Session()
url = 'https://itapp.wcu.edu/BanAuthRedirector/Default.aspx'                    
myId = 'xxxxxxxx'                                                               
myPin = 'xxxxxxxx'  
headers = {'User-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11',
           'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
           'Accept-Encoding': 'gzip,deflate,sdch',                  
           'Accept-Language': 'en-US,en;q=0.8',                     
           'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3'}
data = {'id':myId,                                                          
        'PIN':myPin,                                                        
        'submit':'Request Access',                                          
        'wcuirs_uri':'https://cf.wcu.edu/busafrs/catcard/idsearch.cfm'} 
response = s.post(url, headers = headers, data=date)

if response.status_code == 200: #Maybe add another constraint to be sure we are logged in
    #Now do any call you want
    response = s.get(another_url)
    print response.text

You can get more info here

score 0 · Answer 3 · edited May 23 '17 at 10:24

Another solution I've used in messing w/ ASPX is robobrowser.

For example:

def auth(mailbox, password):
    browser = RoboBrowser(history=False)
    browser.open(oc_auth_uri)

    signin = browser.get_form(id='aspnetForm')
    signin['SubLoginControl:mailbox'].value = mailbox
    signin['SubLoginControl:password'].value = password
    signin['SubLoginControl:javascriptTest'].value = 'true'
    signin['SubLoginControl:btnLogOn'].value = 'Logon'
    signin['SubLoginControl:webLanguage'].value = 'en-US'
    signin['SubLoginControl:initialLanguage'].value = 'en-US'
    signin['SubLoginControl:errorCallBackNumber'].value = 'Entered+telephone+number+contains+non-dialable+characters.'
    signin['SubLoginControl:cookieMailbox'].value = 'mailbox'
    signin['SubLoginControl:cookieCallbackNumber'].value = 'callbackNumber'
    signin['SubLoginControl:serverDomain'].value = ''

    browser.submit_form(signin)
    return browser

Note: You may need to update the form to add hidden form fields such as __VIEWSTATE and friends to the form prior to submitting. See this post for further info.

Python and mechanize login script

3 Answers3

Linked