Iteration in Python refers to the process of traversing elements within a collection sequentially. This is most commonly achieved using a for loop, which abstracts the underlying mechanics of accessing each item.

# Basic iteration over a sequence
for element in (10, 20, 30):
    print(element)

Iterable Objects

Technically, an object is considered iterable if it implements the __iter__ method. Practically, any object that can be used on the right-hand side of a for loop falls into this category. Common examples include lists, tuples, dictionaries, sets, strings, and range objects.

To verify if an object is iterable, one can inspect its attributes. While many container types share attributes like __len__ or __contains__, the defining characteristic of an iterable is the presence of __iter__.

import inspect

data_samples = [
    "text",             # String
    [1, 2, 3],          # List
    {1: 'a', 2: 'b'},   # Dictionary
    {1, 2, 3},          # Set
]

# Check for the __iter__ method
for sample in data_samples:
    if hasattr(sample, '__iter__'):
        print(f"{type(sample).__name__} is iterable")

When the built-in iter() function is called on these objects, it returns an iterator object specific to that type.

for sample in data_samples:
    iterator_obj = iter(sample)
    print(type(iterator_obj))

Iterator Objects

An iterator is an object that represents a stream of data. It must implement two methods: __iter__ and __next__. The __next__ method returns the next item in the sequence and raises a StopIteration exception when no further items are available.

The for loop internally handles this protocol. It calls iter() on the target object to get an iterator, then repeatedly calls next() until StopIteration is caught. This process can be manually replicated using a while loop.

dataset = ['start', 'process', 'end']
iterator = iter(dataset)

while True:
    try:
        item = next(iterator)
        print(item)
    except StopIteration:
        break

Implementing a Custom Iterator

To create a custom iterator, a class must define __next__ to manage state and retrieval, and __iter__ to satisfy the protocol. Consider a scenario where we need to process a list of numbers, skipping negatives and squaring the positive values.

class NumberProcessor:
    def __init__(self, values):
        self.values = values
        self.cursor = 0

    def __iter__(self):
        return self

    def __next__(self):
        while self.cursor < len(self.values):
            current = self.values[self.cursor]
            self.cursor += 1
            
            if current < 0:
                continue
            
            return current ** 2
        
        raise StopIteration

data = [1, -5, 3, -2, 4]
processor = NumberProcessor(data)

for result in processor:
    print(result)

In this implementation, __iter__ returns self, indicating that the object is its own iterator. This adherence to the iterator protocol ensures the object can be used directly in for loops. Without __iter__, the object would raise a TypeError when used in a loop context, even if __next__ is defined.

The Significance of Iterators

Iterators decouple the iteration logic from the data structure. Container types like lists focus on storage, while iterators manage traversal state. This separation allows for powerful patterns such as data pipelines.

Since iterators produce items on demand (lazy evaluation), they are memory efficient. They do not need to load the entire dataset into memory at once. This is particularly useful when dealing with large files or infinite sequences.

import random

class InfiniteStream:
    def __iter__(self):
        return self

    def __next__(self):
        return random.randint(1, 100)

# This loop runs indefinitely until manually stopped
# stream = InfiniteStream()
# for value in stream:
#     print(value)

In this example, the iterator generates values dynamically without storing them. This concept is foundational to generators, which are a concise way to create iterators using functions and the yield keyword.

Legacy Iteration Support

Before the iterator protocol was standardized, Python relied on the __getitem__ method for iteration. If an object lacks __iter__ but implements __getitem__ with sequential integer keys starting from 0, Python will attempt to iterate using index access. However, defining __iter__ is the modern standard for creating iterable objects.