10

I am trying to get the HTML page back from sending a POST request:

import httplib 
import urllib 
import urllib2 
from BeautifulSoup import BeautifulSoup


headers = {
    'Host': 'digitalvita.pitt.edu',
    'Connection': 'keep-alive',
    'Content-Length': '325', 
    'Origin': 'https://digitalvita.pitt.edu',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1',
    'Content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'Accept': 'text/javascript, text/html, application/xml, text/xml, */*',
    'Referer': 'https://digitalvita.pitt.edu/index.php',
    'Accept-Encoding': 'gzip,deflate,sdch',
    'Accept-Language': 'en-US,en;q=0.8',
    'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
    'Cookie': 'PHPSESSID=lvetilatpgs9okgrntk1nvn595'
}

data = {
    'action': 'search',
    'xdata': '<search id="1"><context type="all" /><results><ordering>familyName</ordering><pagesize>100000</pagesize><page>1</page></results><terms><name>d</name><school>All</school></terms></search>',
    'request': 'search'
}

data = urllib.urlencode(data) 
print data 
req = urllib2.Request('https://digitalvita.pitt.edu/dispatcher.php', data, headers) 
response = urllib2.urlopen(req)
the_page = response.read()

soup=BeautifulSoup(the_page)
print soup

Can anyone tell me how to make it work?

Martijn Pieters
  • 889,049
  • 245
  • 3,507
  • 2,997
user1652287
  • 237
  • 2
  • 3
  • 9
  • 3
    Have you checked out the [requests](http://docs.python-requests.org/en/latest/index.html) library? It makes http requests much easier. Also your missing a few carriage returns, e.g. between your import statements, and before your `print` statements. Are these also existent in your original code? – Maus Sep 20 '12 at 19:04
  • 1
    it would be helpful to see the error message – Will Sep 20 '12 at 19:12
  • Using pycurl is much simpler and cleaner: http://stackoverflow.com/questions/22799648/convert-curl-example-to-pycurl – johannzhaojohann Aug 01 '16 at 18:36

1 Answers1

7

Do not specify a Content-Length header, urllib2 calculates it for you. As it is, your header specifies the wrong length:

>>> data = urllib.urlencode(data) 
>>> len(data)
319

Without that header the rest of the posted code works fine for me.

Martijn Pieters
  • 889,049
  • 245
  • 3,507
  • 2,997