This article is based on a presentation I gave at PyMNtos in June, 2011.
Working with HTTP in Python is needlessly cumbersome. The included
module is comprehensive at the expense of great complexity. Kenneth Reitz’s
Requests module, by contrast, eschewes completeness in favor of simplifying
common use cases.
A Simple Example
Imagine we’re trying to
GET a resource at
http://example.test/ and look up
the response code, its
content-type header, and the contents of the
response. This is pretty simple with both
urllib2 and Requests:
>>> import urllib2 >>> url = 'http://example.test/' >>> response = urllib2.urlopen(url) >>> response.getcode() 200 >>> response.headers.getheader('content-type') 'text/html; charset=utf-8' >>> response.read() 'Hello, world!'
>>> import requests >>> url = 'http://example.test/' >>> response = requests.get(url) >>> response.status_code 200 >>> response.headers['content-type'] 'text/html; charset=utf-8' >>> response.content u'Hello, world!'
The two are very similar, with Requests offering access to the response’s
properties via attributes, versus the method calls required by
There are two other subtle but important differences illustrated above:
Responses automatically decoded the response into Unicode.
Responses automatically saved the content, so you can access it multiple times, unlike the read-once file-like object returned by
The second issue is particularly annoying when sketching out code in the Python REPL.
A More Complex Example
Now let’s try something slightly more complex:
GET a resource
http://foo.test/secret, which requires HTTP Basic authentication. Using
the code above as a template, it seems like we just need to replace the call
urllib2.urlopen() or to
requests.get() with something that sends a
username and password along with the request.
This is where
urllib2 goes off the rails.
>>> import urllib2 >>> url = 'http://example.test/secret' >>> password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm() >>> password_manager.add_password(None, url, 'dan', 'h0tdish') >>> auth_handler = urllib2.HTTPBasicAuthHandler(password_manager) >>> opener = urllib2.build_opener(auth_handler) >>> urllib2.install_opener(opener) >>> response = urllib2.urlopen(url) >>> response.getcode() 200 >>> response.read() 'Welcome to the secret page!'
What was one simple function call ballooned into instantiating two classes,
building a third, installing that one into the global urllib2 module, and then
finally making the
Confused? Here’s a link to the urllib2 documentation.
So how does Requests handle the same situation?
>>> import requests >>> url = 'http://example.test/secret' >>> response = requests.get(url, auth=('dan', 'h0tdish')) >>> response.status_code 200 >>> response.content u'Welcome to the secret page!'
By simply adding an
auth keyword parameter to the function call.
I bet you can remember how to do that without consulting the docs, too.
Requests also has far more convenient error handling. If you used an invalid
username and password above,
urllib2 would raise a
Requests would return a normal response object, as expected. All it takes to
determine if the request was successful is a quick peek at the boolean
>>> response = requests.get(url, auth=('dan', 'wrongPass')) >>> response.ok False
The same goes for the remote server being down:
urllib2 raises a
urllib2.URLError, while Requests returns a normal response object, which you
can introspect to learn more about the reason for failure.
Requests exposes a similarly simple API for
It handles multipart file uploads, as well as automatically form-encoding dictionaries.
The documentation is great.
And much more.
Requests is fantastic. Give it a shot the next time you need to communicate over HTTP.