Introduction to HTTPX for Web Requests

HTTPX is a modern HTTP client library for Python that offers many similarities with the popular requests library. This article explores how to utilize HTTPX for web scraping tasks with practical examples.

Basic HTTP Requests

Making HTTP requests with HTTPX follows a straightforward approach similar to requests:

import httpx

# Perform a GET request
web_response = httpx.get('https://example.com/api/data')
print(web_response.status_code)
print(web_response.headers)
print(web_response.text)

Customizing Request Headers

You can easily add custom headers to your requests:

import httpx

# Define custom headers
request_headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36'
}

# Send request with headers
api_response = httpx.get('https://example.com/api/endpoint', headers=request_headers)
print(api_response.text)

Enabling HTTP/2 Support

By default, HTTPX uses HTTP/1.1. To enable HTTP/2 support:

import httpx

# Create client with HTTP/2 enabled
http2_client = httpx.Client(http2=True)

# Make request using HTTP/2
modern_response = http2_client.get('https://http2.example.com')
print(modern_response.text)

The Client Object

The Client object in HTTPX is comparable to the Session object in the requests library, providing connection pooling and persistent parameters:

import httpx

# Using context manager for automatic cleanup
with httpx.Client() as session:
    api_data = session.get('https://api.example.com/data')
    print(api_data)

The context manager approach above is equivalent to manually managing the client lifecycle:

# Manual client management
session = httpx.Client()
try:
    response = session.get('https://api.example.com/data')
finally:
    session.close()

Client with Persistent Parameters

When using a Client object, you can set parameters that persist across requests:

import httpx

# Define base URL and headers
base_url = 'https://api.example.com/headers'
custom_headers = {'User-Agent': 'my-application/1.0.0'}

# Create client with persistent headers
with httpx.Client(headers=custom_headers) as session:
    # All requests will use the same headers
    response = session.get(base_url)
    # Extract and print the User-Agent from response
    print(response.json()['headers']['User-Agent'])

Further Exploration

For more advanced features and capabilities of HTTPX, refer to the official documentation:

HTTPX Advanced Docuemntation