Overview

ThreadPoolExecutor from the concurrent.futures module provides a simple way to run multiple tasks concurrently in Python. It manages a pool of worker threads and distributes tasks across them efficiently.

Three-Step Pattern

# Step 1: Import the executor
from concurrent.futures import ThreadPoolExecutor

# Step 2: Create an executor instance with specified worker count
executor = ThreadPoolExecutor(max_workers=20)

# Step 3: Submit tasks to the executor
future = executor.submit(worker_function, arg1, arg2)

Practical Example

import time
from concurrent.futures import ThreadPoolExecutor

pool = ThreadPoolExecutor(max_workers=5)
completed = []

def process_item(delay_seconds):
    time.sleep(delay_seconds)
    print('Task completed')
    completed.append('done')

# Launch 5 concurrent tasks
for i in range(5):
    pool.submit(process_item, 2)

print('Main thread continues executing')

# Wait for all tasks to finish by checking completion count
while len(completed) < 5:
    pass

print(f'All {len(completed)} tasks finished')
print(completed)

Waiting for Task Completion

The shutdown() method blocks until all submitted tasks finish exeuction:

import time
from concurrent.futures import ThreadPoolExecutor

pool = ThreadPoolExecutor(max_workers=3)
results = []

def compute(value):
    time.sleep(value)
    print(f'Processed: {value}')
    results.append(value)

# Submit multiple tasks
for num in [3, 1, 2]:
    pool.submit(compute, num)

print('Main thread executing')

# Block until all tasks complete
pool.shutdown(wait=True)

print('All workers finished')
print(results)

Key Points

• max_workers determines how many threads run simultaneously
• submit() returns a Future object for tracking indivdiual tasks
• shutdown(wait=True) ensures the main thread waits for all workers to complete
• Tasks submitted after shutdown() is called will be rejected

The executor automatically reuses threads, making it memory-efficient for long-running applications with many short-lived tasks.