Concurrency is essential for efficient web scraping—especially when dealing with I/O-bound tasks like HTTP requests. Python offers two primary approaches: threading for lightweight, shared-memory concurrency, and multiprocessing for CPU-bound or truly isolated workloads. Understanding when and how to aply each is key to building robust scrapers.

Processes vs. Threads: Core Concepts

A process is an independent instance of a running program, with its own memory space, file handles, and system resources. An thread, by contrast, is a lightweight execution unit within a process—sharing memory and global state but scheduled independently by the OS.

Think of a process as a separate application sandbox; threads are cooperative workers inside that sendbox. Due to Python’s Global Interpreter Lock (GIL), threads cannot execute Python bytecode in parallel—but they excel at overlapping I/O waits (e.g., network latency), making them ideal for scraping.

Threading for I/O-Intensive Scraping

For fetching many web pages concurrently, threading reduces total wall-clock time significantly—even if CPU usage remains low.

Manual Thread Management

import threading
import time

def task(name):
    for i in range(3):
        print(f"[{name}] {i}")
        time.sleep(0.5)

# Sequential execution
start = time.time()
task("A")
task("B")
print(f"Sequential: {time.time() - start:.2f}s")

# Concurrent execution
start = time.time()
t1 = threading.Thread(target=task, args=("A",))
t2 = threading.Thread(target=task, args=("B",))
t1.start(); t2.start()
t1.join(); t2.join()
print(f"Threaded: {time.time() - start:.2f}s")

Using ThreadPoolExecutor

The concurrent.futures.ThreadPoolExecutor abstracts thread lifecycle management and simplifies result handling:

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def fetch_url(url):
    time.sleep(1)  # Simulate network delay
    return f"Response from {url}"

urls = ["https://httpbin.org/delay/1"] * 5

# Submit and collect results as they complete
with ThreadPoolExecutor(max_workers=3) as executor:
    futures = {executor.submit(fetch_url, u): u for u in urls}
    for future in as_completed(futures):
        print(future.result())

Real-World Example: Box Office Data Aggregation

This script scrapes annual box office tables from a static site using concurrent requests and structured parsing:

import csv
import requests
from lxml import html
from concurrent.futures import ThreadPoolExecutor

def fetch_page(year):
    url = f"http://www.boxofficecn.com/boxoffice{year}"
    try:
        resp = requests.get(url, timeout=10)
        resp.raise_for_status()
        resp.encoding = "utf-8"
        return resp.text
    except Exception as e:
        print(f"Failed to fetch {year}: {e}")
        return None

def parse_page(html_content, year):
    if not html_content:
        return []
    tree = html.fromstring(html_content)
    rows = tree.xpath("//table/tbody/tr[position() > 1]")
    results = []
    for row in rows:
        try:
            year_text = (row.xpath("./td[2]//text()") or [""])[0].strip()
            title = (row.xpath("./td[3]//text()") or [""])[0].strip()
            gross = (row.xpath("./td[4]//text()") or [""])[0].strip()
            if title:
                results.append([year_text or str(year), title, gross])
        except Exception:
            continue
    return results

def scrape_year(year, writer):
    content = fetch_page(year)
    data = parse_page(content, year)
    for record in data:
        writer.writerow(record)

def main():
    with open("box_office.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow(["Year", "Title", "BoxOffice"])
        
        years = list(range(1994, 2022))
        with ThreadPoolExecutor(max_workers=8) as pool:
            pool.map(lambda y: scrape_year(y, writer), years)

if __name__ == "__main__":
    main()

Multiprocessing for Isolated or CPU-Heavy Tasks

When tasks must be fully isolated (e.g., avoiding shared-state bugs), involve heavy local processing (like image resizing), or require true parallelism beyond the GIL’s limits, multiprocessing is the right choice.

Producer–Consumer Pipeline for Image Downloading

This example separates URL discovery and download logic into two processes, communicating via a multiprocessing.Queue:

import requests
from multiprocessing import Process, Queue
from concurrent.futures import ThreadPoolExecutor
from lxml import html

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
}

def discover_images(page_queue, base_url="https://www.pkdoutu.com/photo/list/"):
    """Producer: Extract image URLs across paginated results."""
    for page_num in range(1, 6):
        try:
            resp = requests.get(f"{base_url}?page={page_num}", headers=HEADERS, timeout=10)
            tree = html.fromstring(resp.content)
            srcs = tree.xpath("//img[@data-original]/@data-original")
            for src in srcs:
                if src.startswith("http"):
                    page_queue.put(src.strip())
        except Exception as e:
            print(f"Discovery error on page {page_num}: {e}")
    page_queue.put(None)  # Sentinel

def download_images(queue, max_workers=10):
    """Consumer: Download images using a thread pool per process."""
    def save_image(url):
        try:
            resp = requests.get(url, headers=HEADERS, timeout=15)
            filename = url.split("/")[-1] or f"img_{hash(url) % 10000}.jpg"
            with open(f"images/{filename}", "wb") as f:
                f.write(resp.content)
            print(f"Downloaded: {filename}")
        except Exception as e:
            print(f"Failed to download {url}: {e}")

    with ThreadPoolExecutor(max_workers=max_workers) as pool:
        while True:
            url = queue.get()
            if url is None:
                break
            pool.submit(save_image, url)

if __name__ == "__main__":
    q = Queue()
    producer = Process(target=discover_images, args=(q,))
    consumer = Process(target=download_images, args=(q,))

    producer.start()
    consumer.start()
    producer.join()
    consumer.join()

Note: Always ensure the images/ directory exists before running this script.

Selecting the Right Model

• Use threading for high-volume, I/O-bound scraping (HTTP, database reads) where shared memory is acceptable and startup overhead must be minimal.
• Prefer multiprocessing when tasks need strict isolation, involve CPU-bound post-processing, or must survive failures without affecting others.
• Avoid mixing both unnecessarily: Over-engineering concurrency adds complexity without benefit unless bottlenecks are clearly identified.