Python HTTP Libraries: urllib, urllib2, and requests Comparison

Understanding urllib and urllib2 Differences

Both urllib and urllib2 modules handle URL operations, but they serve different purposes. The key distinctions include:

  • urllib2.urlopen can accept either a Request object or a URL, while urllib.urlopen only accepts URLs
  • When using a Request object with urllib2, you can set custom headers for the URL
  • urllib provides the urlencode function, which urllib2 lacks - this is why they're often used together

Example with urllib2 and Request Object


# Creating a request with custom headers and POST data
from urllib2 import Request, urlopen
from urllib import urlencode

web_request = Request(url='http://www.example.com')
web_request.add_header('User-Agent', 'Python Web Scraper')
web_request.add_data(urlencode({'key': 'value'}))

# Sending the request
response = urlopen(web_request)

urllib Module Features

Handling Unicode in urlencode

urlencode doesn't directly process unicode objects, so unicode needs to be encoded first:


# Encoding unicode to utf-8 before URL encoding
from urllib import urlencode

encoded_data = urlencode(u'unicode_string'.encode('utf-8'))

Basic URL Fetching


# Fetching a webpage and saving it to a file
from urllib import urlopen

web_url = 'http://m.example.com/some-page'
response = urlopen(web_url)
page_content = response.read()

# Save to file
with open('webpage.html', 'w') as output_file:
    output_file.write(page_content)

# Display response object attributes
print dir(response)

URL Encoding and Decoding

The urllib module provides several functions for URL encoding and decoding:


from urllib import quote, unquote, quote_plus, unquote_plus, urlencode

# Basic URL encoding
encoded_string = quote('This is a test')
print 'Encoded:', encoded_string  # Spaces become %20
decoded_string = unquote(encoded_string)
print 'Decoded:', decoded_string

# URL encoding with plus for spaces
encoded_with_plus = quote_plus('This is a test')
print 'Encoded with plus:', encoded_with_plus
decoded_from_plus = unquote_plus(encoded_with_plus)
print 'Decoded from plus:', decoded_from_plus

# Encoding dictionary data
form_data = {'username': 'johndoe', 'password': 'secure123'}
encoded_form = urlencode(form_data)
print 'Encoded form:', encoded_form

File Download with urlretrieve

The urlretrieve function is useful for simple file downloads:


from urllib import urlretrieve

file_url = 'http://m.example.com/some-resource'
urlretrieve(file_url, 'downloaded_file.html')

urllib2 Module Features

urllib2 provides advanced features for handling HTTP requests, including:

  • Basic authentication
  • Redirect handling
  • Cookie management
  • Custom headers

Basic Request Handling


# Simple URL request
from urllib2 import urlopen

web_url = 'http://m.example.com/some-page'
response = urlopen(web_url)
page_content = response.read()

Using Request Objects


# Creating and using a Request object
from urllib2 import Request, urlopen

web_url = 'http://m.example.com/some-page'
request = Request(web_url)
response = urlopen(request)
page_content = response.read()

Request Class Parameters

The Request class accepts several parameters:

  • URL - The target URL as a string
  • data - Additional data to send to the server (for POST requests)
  • headers - Dictionary of HTTP headers
  • origin_req_host - The originating request host (optional)
  • unverifiable - Whether the request is unverifiable (optional)

POST Request Example


# Sending POST data
from urllib2 import Request, urlopen
from urllib import urlencode

target_url = 'http://www.example.com/submit-form'
form_values = {
    'name': 'John Doe',
    'location': 'New York',
    'language': 'Python'
}
post_data = urlencode(form_values)
request = Request(target_url, post_data)
response = urlopen(request)
page_content = response.read()

Adding Custom Headers


# Request with custom headers
from urllib2 import Request, urlopen
from urllib import urlencode

target_url = 'http://www.example.com/submit-form'
browser_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
form_values = {
    'name': 'Jane Smith',
    'email': 'jane@example.com'
}
headers = {'User-Agent': browser_agent}
post_data = urlencode(form_values)
request = Request(target_url, post_data, headers)
response = urlopen(request)
page_content = response.read()

Adding Headers to Requests


# Adding headers to a request
from urllib2 import Request, urlopen

request = Request('http://www.example.com/')
request.add_header('Referer', 'http://www.python.org/')
response = urlopen(request)

Global Opener Configuration


# Configuring global opener
from urllib2 import Request, urlopen, build_opener, install_opener

request = Request('http://www.python.org/')
opener = build_opener()
install_opener(opener)
response = urlopen(request)

Error Handling

When making HTTP requests, various errors can occur. urllib2 provides specific exception classes:

  • URLError - For general URL-related errors
  • HTTPError - For HTTP-specific errors (404, 403, etc.)

# Handling HTTP errors
from urllib2 import urlopen, URLError, HTTPError

try:
    response = urlopen('http://www.example.com/nonexistent-page')
    page_content = response.read()
except HTTPError as e:
    print 'HTTP Error:', e.code, e.reason
except URLError as e:
    print 'URL Error:', e.reason

Summary of When to Use Each Library

  • urllib.urlretrieve() - Best for simple file downloads where you don't need to process the content
  • urllib2 with Request objects - Ideal for forms, authentication, and requests requiring custom headers
  • urllib.urlencode() - Use when encoding dictionary data for URL parameters

requests Library

requests is a third-party library that builds on urllib3, inheriting all its features while adding convenience:

  • HTTP connection keep-alive and connection pooling
  • Cookie session support
  • File upload capabilities
  • Automatic response encoding detection
  • International URL support
  • Automatic POST data encoding

Basic requests Usage


# Basic requests usage
import requests

# GET request
response = requests.get('http://www.example.com/users')

# POST request
data = {'first_name': 'John', 'last_name': 'Doe', 'pwd': 'password123'}
response = requests.post('http://www.example.com/users', data=data)

# Other HTTP methods
response = requests.put('http://www.example.com/users/update')
response = requests.delete('http://www.example.com/users/delete')

# Accessing response data
if response.headers['content-type'] == 'application/':
    _data = response.()
else:
    text_data = response.text

requests Features

  • International domain and URL support
  • Keep-Alive and connection pooling
  • Persistent cookie sessions
  • Browser-style SSL verification
  • Basic/Digest authentication
  • Cookie handling
  • Automatic decompression
  • Unicode response bodies
  • Multipart file uploads
  • Connection timeouts
  • .netrc support
  • Thread safety

Installation

requests is not part of Python's standard library and must be installed separately:


# Using pip
pip install requests

# Using easy_install
easy_install requests

Limitations

While requests offers many conveniences, it has some limitations:

  • Not designed for asynchronous calls
  • Can be slower than urllib for certain operations
  • Standard urllib can be used as an alternative

Tags: python HTTP urllib urllib2 Requests

Posted on Fri, 25 Sep 2026 16:54:16 +0000 by myanavrin

Hot Tags

©2026 Freaks City