How Python Iterators Work Behind the Scenes

Ever written for item in my_list and wondered what Python is actually doing under the hood? It feels like magic—until you realize that beneath every loop, container, and even generator, there's a simple, elegant protocol at work. Let's pop the hood and see how iterators really tick.

The Iterator Protocol: Two Methods, Infinite Power

At its core, the iterator protocol requires exactly two methods:

1. __iter__() – returns the iterator object itself
2. __next__() – returns the next value or raises StopIteration

That's it. No fancy syntax, no hidden complexity. When Python sees for x in obj, it first calls iter(obj) which invokes obj.__iter__(), then repeatedly calls next() on the result until StopIteration is raised.

Here's a stripped-down version of what for loop actually does:

# What Python's for loop translates to
iterator = iter(your_object)
while True:
    try:
        item = next(iterator)
        # do something with item
    except StopIteration:
        break

Understanding this unlocks a lot of Python's design philosophy: everything that can be iterated should be iterable, and everything that can be a context manager should be one.

The Surprising Truth: Lists Aren't Iterators

Here's where beginners get tripped up. Lists, tuples, strings, dicts—these are iterable, not iterators. An iterable is anything that can produce an iterator. An iterator is the thing that actually does the iterating.

my_list = [1, 2, 3]
iterator = iter(my_list)  # produces a list_iterator object
print(next(iterator))    # 1
print(next(iterator))    # 2

The list itself can't be used with next() directly. But the list_iterator returned by iter() can. That separation means you can have multiple independent iterations over the same list simultaneously—each gets its own iterator.

Building Your Own Iterator

Let's make this concrete. Here's a simple iterator that produces Fibonacci numbers forever (or until you stop asking):

class FibonacciIterator:
    def __init__(self):
        self.a, self.b = 0, 1

    def __iter__(self):
        return self  # because itself is an iterator

    def __next__(self):
        result = self.a
        self.a, self.b = self.b, self.a + self.b
        return result

fib = FibonacciIterator()
for _ in range(10):
    print(next(fib))  # 0, 1, 1, 2, 3, 5, 8, 13, 21, 34

Note that __iter__ returns self—this is standard for dedicated iterator classes. Each call to iter(my_iterator) gives you back the same object, which is why you can't have two independent loops over the same iterator (they'd share state).

The Magic of StopIteration: How Loops End Gracefully

Raising StopIteration is the official way to signal "no more data." But here's a subtle point: in generator functions, return implicitly raises StopIteration. In custom iterators, you must do it explicitly.

Why not just return None? Because None could be a legitimate value. StopIteration is a signal baked into the protocol, separate from valid data.

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current < 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for num in Countdown(3):
    print(num)  # 3, 2, 1, 0

No explicit break, no bounds checking in your loop—the iterator handles termination cleanly.

Generators: Iterators Without the Boilerplate

Python's yield keyword exists precisely because writing full iterator classes is tedious for simple use cases. A generator function automatically implements the iterator protocol:

def fibonacci_generator():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

When this function runs, it returns a generator object—which has __iter__ and __next__ already built. The yield statement pauses execution, preserves local state, and resumes when next() is called. It's syntactic sugar for the very same protocol.

Under the hood, there's actually a StopIteration being raised when the generator function ends (either by hitting a return or falling off the end). Python handles this automatically.

Infinite Iterators Are Fine (With Care)

Since StopIteration is only raised when you decide, you can create iterators that never stop. itertools.count() does this—it counts integers forever. Using one in a for loop without a break will hang your program.

import itertools

# This would run forever:
# for x in itertools.count():
#     print(x)

# Use islice to limit it:
for x in itertools.islice(itertools.count(), 5):
    print(x)  # 0, 1, 2, 3, 4

This is why Python's for loops are more than just sugar—they give you control over termination. The iterator doesn't know or care about boundaries.

Performance: Why Iterators Save Memory

The biggest practical benefit of iterators is lazy evaluation. A list holds all its elements in memory simultaneously. An iterator produces values on demand, one at a time.

Consider reading a file:

# Reads entire file into memory
lines = open('huge_file.txt').readlines()

# Reads one line at a time
for line in open('huge_file.txt'):
    pass

The second version works because file objects are iterators. Each call to next() reads the next line from disk. If the file is 10 GB, the first version crashes—the second barely uses memory.

Common Pitfalls and Gotchas

Exhausting an iterator: Once an iterator raises StopIteration, it's done. for loops handle this seamlessly, but calling next() manually won't restart it.

it = iter([1, 2, 3])
list(it)  # [1, 2, 3]
list(it)  # []  → exhausted!

Modifying collections during iteration: Iterators typically work over snapshots or rely on the underlying structure. Changing a dict's size while iterating raises RuntimeError. For lists, behavior is undefined.

Using next() with a default: This is a little-known trick that avoids the StopIteration trap:

it = iter([1, 2, 3])
print(next(it, 'done'))  # 1
print(next(it, 'done'))  # 2
print(next(it, 'done'))  # 3
print(next(it, 'done'))  # 'done'

next() accepts an optional default—when supplied, it returns that default instead of raising an exception.

The Bigger Picture

Iterators are one of Python's fundamental design patterns. They underpin comprehensions, map/filter/reduce, generators, async iterators, and even context managers (through __enter__/__exit__, which is a different but analogous protocol). Once you internalize the two-method protocol, you'll see it everywhere—and you'll start writing more Pythonic, memory-efficient code.

Next time you write a for loop, you'll know exactly what's happening. And when you need custom iteration, you'll reach for a generator—knowing it's just a convenient wrapper around the same simple machinery that's been there all along.