Python Request HTTP Page and Use The Source Information

Question

I'm looking to request a webpage with python and to get the source code to string (my page has only text for exmaple Year2012 with no html or other codes so if you do view source you see Year2012 this is what I need)

for example I have this C# code what I want in Python:

string WebClient = new WebClient().DownloadString(WebPage);

If I create myscript.php and enter Year2012 string WebClient will be = "Year2012" If I change it later on and my program check the webpage it will be what ever I change. I want the same thing In python for windows any idea how?

Thanks!

prehistoricpenguin · Accepted Answer · 2013-05-14T02:24:51.450

3

I suggest you using the package requests.Here is a code snipet that may meet your need:

import requests
r = requests.get('https://github.com/timeline.json')

EDIT:

Or you may use the default package in python: urllib2, a simple example:

import urllib2
f = urllib2.urlopen('http://www.python.org/')
print f.read(100)

The output:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<?xml-stylesheet href="./css/ht2html

edited May 14 '13 at 02:24

answered May 14 '13 at 01:17

prehistoricpenguin

3,842
1
21
34

@prehistoicpenguin thanks! how can I install the package on windows? Didn't find it :( – Mark May 14 '13 at 02:06
@Mark Here is the installation page:http://docs.python-requests.org/en/latest/user/install/ – prehistoricpenguin May 14 '13 at 02:11
@Mark Follow this tips to install pip:http://stackoverflow.com/questions/4750806/how-to-install-pip-on-windows – prehistoricpenguin May 14 '13 at 02:13
@Mark Then install requests with pip. – prehistoricpenguin May 14 '13 at 02:14
thanks I will install it now with pip for windows does requests have any dependencies or will it work on any PC? Any chance a way to accomplish this without external packages? – Mark May 14 '13 at 02:16
Any PC with the package `requests` will be ok. – prehistoricpenguin May 14 '13 at 02:18
@I will be creating it with py2exe will it work without installing `requests` ? – Mark May 14 '13 at 02:19
I have not try that, I think it will work.And the default package `urllibs` with python will also work. – prehistoricpenguin May 14 '13 at 02:22
do you have a solution with urllibs? I have tried `myyear = urllib2.urlopen("http://website.com/year.php").read()` but it didn't seem to work – Mark May 14 '13 at 02:23
`TypeError: cannot concatenate 'str' and 'instance' objects` trying to use it later in the script I'm doing it wrong? – Mark May 14 '13 at 02:28
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/29880/discussion-between-prehistoricpenguin-and-mark) – prehistoricpenguin May 14 '13 at 02:30
seems like urllib2 works fine on my PC but where ever there is no installed python it will not work, Just wanted to let you know. I will try `requests` way now and see if it will work. – Mark May 14 '13 at 12:07
@If i change `urllib2` to `urllib` will it still works? I have read that urllib is included in the exe and 2 isn't – Mark May 14 '13 at 12:11
@Mark Look at this page: http://www.hacksparrow.com/python-difference-between-urllib-and-urllib2.html – prehistoricpenguin Jun 22 '13 at 11:42

Python Request HTTP Page and Use The Source Information

1 Answers1