I am trying to write a webscraping program in python. Howevere, the pages I want to scrape are behind a login. I have an account and have been trying to follow the help posted here . I think I have done everything right, but I cannot get past the login. My code is posted below:
#!/usr/bin/env python
import requests, sys, lxml.html
#logging in
s = requests.Session()
login_url = 'https://login.fidelity.com/ftgw/Fas/Fidelity/RtlCust/Login/'
payload = {
'ssn' : 'USERNAME',
'pin' : 'PASSWORD'
}
s.post(login_url, data=payload, headers=dict(referer='https://login.fidelity.com'))
#page to scrape
response = s.get('https://fixedincome.fidelity.com/ftgw/fi/FIBondDetails?requestType=&displayFormat=TABLE&cusip=30382LDK1&ordersystem=TORD&preferenceName=')
print response.content #redirected to the login page