Before diving into the three fundamental Python constructs, it's essential to understand the distinction between containers and iterable objects.

Containers

A container is a data structure that groups multiple elements together. Elements within a container can be iterated over one by one, and you can use the in and not in operators to check whether an element exists within the container. Most container implementations store all elements in memory, though exceptions exist such as iterators and generators that compute values on demand.

Common container types in Python include:

• Lists and deques
• Sets and frozen sets
• Dictionaries, including specialized variants like defaultdict, OrderedDict, and Counter
• Tuples and named tuples
• Strings

The container concept resembles a box that holds items. When an object can answer the question "is element X contained within me?" it qualifies as a container:

>>> 1 in [1, 2, 3]
True
>>> 4 not in [1, 2, 3]
True
>>> 1 in {1, 2, 3}
True
>>> 4 not in {1, 2, 3}
True
>>> 1 in (1, 2, 3)
True

For dictionaries, the in operator checks against keys, not values:

>>> d = {'a': 1, 'b': 2, 'c': 3}
>>> 'a' in d
True
>>> 1 in d
False

For strings, the in operator works for both characters and substrings:

>>> 'foobar'
>>> 'b' in 'foobar'
True
>>> 'x' not in 'foobar'
True
>>> 'foo' in 'foobar'
True

Not all containers are iterable. For example, a Bloom filter can test membership but cannot iterate over elements because it doesn't store elements directly—it uses hash functions to map elements to array positions.

Iterable Objects

An object is iterable if it implements the __iter__ method, which returns an iterator for that object:

>>> numbers = [1, 2, 3]
>>> numbers.__iter__()
<list_iterator object at 0x7f97c549aa50>

Python provides several mechanisms for accessing elements from iterable objects, including for loops, list comprehensions, and membership operators.

To verify whether an object is iterable:

>>> from collections.abc import Iterable
>>> isinstance('hello', Iterable)
True
>>> isinstance(42, Iterable)
False
>>> isinstance([], Iterable)
True

Any object that can be used in a for loop qualifies as iterable. This includes lists, tuples, sets, dictionaries, and strings.

Iterators

The iterator protocol requires an object to provide a __next__ method that returns the next item in the sequence or raises StopIteration when no items remain. Iterators move forward only—there is no backward movement.

Objects implementing the iterator protocol also define an __iter__ method that returns the iterator itself. Python's built-in constructs like for loops, sum(), min(), and max() all operate based on this protocol.

Using iterators offers two primary advantages. First, unlike lists that load all values into memory simultaneously, iterators compute values one at a time, reducing memory consumption. Second, iterator-based code tends to be more generic and simpler.

To verify whether an object is an iterator:

>>> from collections.abc import Iterator
>>> d = {'a': 1, 'b': 2, 'c': 3}
>>> isinstance(d, Iterator)
False
>>> isinstance(iter(d), Iterator)
True

The __next__ method retrieves items sequentially:

>>> d = {'a': 1, 'b': 2, 'c': 3}
>>> it = iter(d)
>>> next(it)
('a', 1)
>>> next(it)
('b', 2)
>>> next(it)
('c', 3)

Understanding iterator behavior requires examining how the protocol works internally:

# Demonstrating iterator protocol manually
items = ['apple', 'banana', 'cherry']
iterator = items.__iter__()

print(iterator.__next__())  # apple
print(iterator.__next__())  # banana
print(iterator.__next__())  # cherry
# print(iterator.__next__())  # StopIteration

# The same sequence using index-based access
print(items[0])  # apple
print(items[1])  # banana
print(items[2])  # cherry
# print(items[3])  # IndexError

Simulating the for loop mechanism demonstrates how Python handles iteration internally:

items = ['x', 'y', 'z']
iterator = items.__iter__()

while True:
    try:
        value = iterator.__next__()
        print(value)
    except StopIteration:
        print("Iteration complete")
        break

The for loop follows the iterator protocol: it calls iter() to obtain an iterator, repeatedly calls next(), and terminates when StopIteration is raised.

Generators

A generator is a specialized type that automatically implemants the iterator protocol. When a generator function executes and encounters a yield statement, it pauses and saves its current state, returning the yielded value. When next() or send() is called again, execution resumes from where it left off.

Generator Functions

A generator function is any function containing the yield keyword. Unlike regular functions that return once, generators can pause and resume multiple times:

def sequence_generator():
    print("Starting sequence")
    first = yield
    print(f"First value received: {first}")
    yield 2
    print("Second yield reached")
    yield 3

seq = sequence_generator()
next(seq)                      # Advance to first yield
seq.send("hello")              # Send value to first yield
next(seq)                      # Advance to second yield
next(seq)                      # Advance to third yield

The yield keyword serves dual purposes: it returns a value to the caller and preserves the function's execution state for resumption.

Generator Expressions

Generator expressions provide a concise syntax similar to list comprehensions but with lazy evaluation:

total = sum(x for x in range(10000))

This approach memory-efficiently generates values on demand rather than creating an intermediate list.

Decorators

A decorator extends a function's capabilities without modifying its source code or changing how it's called. The underlying pattern combines higher-order functions, nested functions, and closures.

The basic structure follows this pattern:

import time

def timing_decorator(func):
    def wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start_time
        print(f"Execution time: {elapsed:.4f} seconds")
        return result
    return wrapper

@timing_decorator
def process_data(data):
    time.sleep(0.5)
    return f"Processed: {data}"

result = process_data("example")

When @timing_decorator is applied to process_data, Python calls timing_decorator(process_data) and assigns the returned wrapper function back to process_data. Subsequent calls to process_data actually invoke wrapper, which executes the timing logic before and after calling the original function.

The decorator pattern enables cross-cutting concerns like logging, authentication, caching, and metrics to be added to functions transparently.

Closures

A closure forms when an inner function references variables from an enclosing function's scope. Three conditions must be satisfied for a closure to exist:

1. A nested function must be defined within another function
2. The inner function must reference at least one variable from the outer function's scope
3. The outer function must return the inner function

def counter_factory():
    count = 0
    def increment():
        nonlocal count
        count += 1
        return count
    return increment

counter = counter_factory()
print(counter())  # 1
print(counter())  # 2
print(counter())  # 3

The increment function forms a closure over the count variable, preserving access to it even after counter_factory has returned.

Closures offer several advantages in Python development. They eliminate the need for global variables by encapsulating state within functions. They provide data hiding by preventing external code from directly accessing enclosed variables. They also enable more elegant object-oriented implementations without requiring full class definitions for simple scenarios.

The decorator pattern fundamentally relies on closures—when the outer function accepts a function as an argument and returns an inner function that references the outer function's parameters, a closure is created that maintains access to the original function throughout the decorator's lifetime.