This article explains the concepts of iterables, iterators, and generators in Python, their relationships, and how to differentiate them. The following diagram illustrates they hierarchy:

Iterables

An iterable is a broader concept then an iterator. As shown above, iterables include iterators, and generators are a special type of iterator. Broadly speaking, any object that can be traversed using a for loop is an iterable. More precisely, an object implementing the __iter__() method is an iterable. There are two ways to check if an object is iterable:

1. Use dir() to list attributes and methods; if __iter__() is present, the object is iterable. For example:

dir([1, 2, 3])
# Output includes '...', '__iter__', '__le__', '__len__', ...

2. Use isinstance() with Iterable from collections:

from collections import Iterable
print(isinstance([1, 2, 3], Iterable))
# Output: True

However, not all iterables can be used in a for loop. For example:

class MyIter:
    def __iter__(self):
        pass

my_iter = MyIter()
print(isinstance(my_iter, Iterable))  # True
for i in my_iter:
    pass
# TypeError: iter() returned non-iterator of type 'NoneType'

This shows that an iterable must correctly implement __iter__() to return an iterator for for loops to work.

Iterators

An iterator is an object that represents a data stream. It must implement both __iter__() (returning itself) and __next__() (returning the next element or raising StopIteration).

Under the hood, a for loop does the following:

1. Calls __iter__() on the iterable to get an iterator.
2. Repeatedly calls __next__() on the iterator to get elements.
3. Catches StopIteration to end the loop.

Is a list an iterator?

A list can be looped over, but it is not an iterator:

my_list = [1, 2, 3]
next(my_list)
# TypeError: 'list' object is not an iterator

Examining its methods:

dir(my_list)
# No '__next__' present

When looping, Python internally calls iter(my_list) to get a list iterator:

print(iter(my_list))
# <list_iterator object at 0x...>
print(my_list.__iter__())
# <list_iterator object at 0x...>

The list iterator has __next__:

my_iter = my_list.__iter__()
print(my_iter.__next__())  # 1
print(my_iter.__next__())  # 2
print(my_iter.__next__())  # 3
print(my_iter.__next__())  # StopIteration

Key points:

• A for loop works on the iterator, not directly on the iterbale.
• An iterator becomes exhausted after one traversal and cannot be reused.
• Multiple traversals of an iterable create new iterators each time.

Advantages of iterators:

• Memory efficiency: Iterators compute elements lazily, only when needed. For example, a file object is an iterator:

f = open("test.txt")
from collections import Iterator
isinstance(f, Iterator)  # True

Instead of reading the entire file into memory:

with open("test.txt") as f:
    data = f.readlines()
    for line in data:
        pass

Use:

with open("test.txt") as f:
    for line in f:
        pass

The latter reads one line at a time, saving memory.

Generators

A generator is a special type of iterator that is simpler to create. Generators are defined using functions with yield or generator expressions (e.g., (x*2 for x in range(10))). They maintain their execution state between calls.

Comparison: Fibonacci sequence with different approaches

Traditional (list-based):

def fibonacci_list(n):
    result = [0, 1]
    for i in range(n-1):
        result.append(result[-2] + result[-1])
    return result[1:]

res = fibonacci_list(100000)
for i in res:
    print(i)

Iterator:

class FibonacciIterator:
    def __init__(self, count):
        self.a, self.b = 0, 1
        self.count = count
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index >= self.count:
            raise StopIteration
        self.index += 1
        self.a, self.b = self.b, self.a + self.b
        return self.a

fib_iter = FibonacciIterator(100000)
for i in fib_iter:
    print(i)

Generator:

def fibonacci_generator(n):
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
        yield a

fib_gen = fibonacci_generator(100000)
for i in fib_gen:
    print(i)

Summary

• Iterable: Any object with __iter__() that returns an iterator.
• Iterator: An object with both __iter__() (returning self) and __next__(). Can be traversed once.
• Generator: A concise way to create iterators using yield or generator expressions.

Understanding these concepts helps write memory-efficient Python code by leveraging lazy evaluation.