Questions tagged [urllib]

Python module providing a high-level interface for fetching data across the World Wide Web. Predecessor to urllib2. In Python 3, urllib2 and urllib have been reorganized and merged into urllib.

For Python 2, the urllib module is the predecessor of the urllib2 module, while the latter still uses some functionality of the former.

For Python 3, the urllib package was reorganized. It now has no content of its own. All methods and classes are in several submodules:

Note that doesn't exist in Python 3 anymore.

3637 questions
944
votes
25 answers

How to download a file over HTTP?

I have a small utility that I use to download an MP3 file from a website on a schedule and then builds/updates a podcast XML file which I've added to iTunes. The text processing that creates/updates the XML file is written in Python. However, I use…
Owen
  • 19,917
  • 13
  • 38
  • 47
847
votes
11 answers

What are the differences between the urllib, urllib2, urllib3 and requests module?

In Python, what are the differences between the urllib, urllib2, urllib3 and requests modules? Why are there three? They seem to do the same thing...
Paul Biggar
  • 24,430
  • 19
  • 91
  • 139
605
votes
13 answers

How to urlencode a querystring in Python?

I am trying to urlencode this string before I submit. queryString = 'eventName=' + evt.fields["eventName"] + '&' + 'eventDescription=' + evt.fields["eventDescription"];
James
  • 7,872
  • 5
  • 28
  • 36
337
votes
5 answers

How to percent-encode URL parameters in Python?

If I do url = "http://example.com?p=" + urllib.quote(query) It doesn't encode / to %2F (breaks OAuth normalization) It doesn't handle Unicode (it throws an exception) Is there a better library?
Paul Tarjan
  • 44,540
  • 54
  • 163
  • 207
325
votes
39 answers

urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error

I am getting the following error: Exception in thread Thread-3: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File…
user3724476
  • 3,599
  • 3
  • 10
  • 16
301
votes
8 answers

UnicodeEncodeError: 'charmap' codec can't encode characters

I'm trying to scrape a website, but it gives me an error. I'm using the following code: import urllib.request from bs4 import BeautifulSoup get = urllib.request.urlopen("https://www.website.com/") html = get.read() soup =…
SstrykerR
  • 5,326
  • 3
  • 10
  • 10
284
votes
6 answers

How to send POST request?

I found this script online: import httplib, urllib params = urllib.urlencode({'number': 12524, 'type': 'issue', 'action': 'show'}) headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"} conn =…
user1113569
  • 2,979
  • 2
  • 12
  • 7
201
votes
19 answers

Downloading a picture via urllib and python

So I'm trying to make a Python script that downloads webcomics and puts them in a folder on my desktop. I've found a few similar programs on here that do something similar, but nothing quite like what I need. The one that I found most similar is…
Mike
  • 2,021
  • 2
  • 13
  • 5
165
votes
13 answers

AttributeError: 'module' object has no attribute 'urlopen'

I'm trying to use Python to download the HTML source code of a website but I'm receiving this error. Traceback (most recent call last): File "C:\Users\Sergio.Tapia\Documents\NetBeansProjects\DICParser\src\WebDownload.py", line 3, in
delete
129
votes
4 answers

How to convert a dictionary to query string in Python?

After using cgi.parse_qs(), how to convert the result (dictionary) back to query string? Looking for something similar to urllib.urlencode().
Sam
  • 4,107
  • 8
  • 20
  • 17
128
votes
5 answers

Python: Importing urllib.quote

I would like to use urllib.quote(). But python (python3) is not finding the module. Suppose, I have this line of code: print(urllib.quote("châteu", safe='')) How do I import urllib.quote? import urllib or import urllib.quote both…
imrek
  • 2,489
  • 3
  • 14
  • 33
123
votes
4 answers

In Python, how do I use urllib to see if a website is 404 or 200?

How to get the code of the headers through urllib?
TIMEX
  • 217,272
  • 324
  • 727
  • 1,038
122
votes
10 answers

can we use XPath with BeautifulSoup?

I am using BeautifulSoup to scrape an URL and I had the following code import urllib import urllib2 from BeautifulSoup import BeautifulSoup url = "http://www.example.com/servlet/av/ResultTemplate=AVResult.html" req = urllib2.Request(url) response…
Shiva Krishna Bavandla
  • 20,872
  • 61
  • 174
  • 298
119
votes
3 answers

'module' has no attribute 'urlencode'

When I try to follow the Python Wiki's example related to URL encoding: >>> import urllib >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params) >>> print…
Croll
  • 3,213
  • 6
  • 24
  • 54
110
votes
6 answers

urllib2.HTTPError: HTTP Error 403: Forbidden

I am trying to automate download of historic stock data using python. The URL I am trying to open responds with a CSV file, but I am unable to open using urllib2. I have tried changing user agent as specified in few questions earlier, I even tried…
kumar
  • 2,340
  • 2
  • 15
  • 18
1
2 3
99 100