Iteration Protocol¶
Iteration dunder methods enable objects to work with for loops, comprehensions, and other iteration contexts. Like containers, callables, and context managers, iteration is a protocol --- Python doesn't care about your class's type, only whether it implements __iter__ (or falls back to __getitem__).
Mental Model
An iterable is anything that can produce an iterator (__iter__), and an iterator is anything that can produce the next value (__next__) and signal when it is done (StopIteration). Think of an iterable as a book and an iterator as a bookmark -- you can have multiple bookmarks in the same book, each tracking its own position independently.
Containers vs Iterables
The container protocol and the iteration protocol serve different purposes. Containers provide access — retrieving items by index or key (__getitem__, __len__). Iterables provide traversal — producing items one at a time in sequence (__iter__, __next__). Many objects implement both (e.g., list), but the protocols are independent: a generator is iterable but not a container; a mapping is a container but may define its own iteration order.
The Iteration Protocol¶
for item in obj:
↓
obj.__iter__() → returns iterator
↓
iterator.__next__() → returns next value
iterator.__next__() → returns next value
...
iterator.__next__() → raises StopIteration
iter: Making Objects Iterable¶
Basic Iterator¶
```python class Countdown: def init(self, start): self.start = start
def __iter__(self):
return CountdownIterator(self.start)
class CountdownIterator: def init(self, start): self.current = start
def __iter__(self):
return self
def __next__(self):
if self.current < 0:
raise StopIteration
value = self.current
self.current -= 1
return value
Usage¶
for n in Countdown(5): print(n, end=' ') # 5 4 3 2 1 0 ```
Self-Iterating Class¶
For simple cases, the object can be its own iterator:
```python class Counter: def init(self, low, high): self.low = low self.high = high
def __iter__(self):
self.current = self.low
return self
def __next__(self):
if self.current > self.high:
raise StopIteration
value = self.current
self.current += 1
return value
Warning: Can only iterate once!¶
c = Counter(1, 3) print(list(c)) # [1, 2, 3] print(list(c)) # [] - exhausted! ```
Separate Iterator (Reusable)¶
```python class Range: def init(self, start, stop, step=1): self.start = start self.stop = stop self.step = step
def __iter__(self):
# Return NEW iterator each time
return RangeIterator(self.start, self.stop, self.step)
class RangeIterator: def init(self, start, stop, step): self.current = start self.stop = stop self.step = step
def __iter__(self):
return self
def __next__(self):
if (self.step > 0 and self.current >= self.stop) or \
(self.step < 0 and self.current <= self.stop):
raise StopIteration
value = self.current
self.current += self.step
return value
Can iterate multiple times¶
r = Range(1, 5) print(list(r)) # [1, 2, 3, 4] print(list(r)) # [1, 2, 3, 4] - works again! ```
Generator-Based iter¶
The simplest way to implement iteration:
```python class Fibonacci: def init(self, limit): self.limit = limit
def __iter__(self):
a, b = 0, 1
while a < self.limit:
yield a
a, b = b, a + b
for n in Fibonacci(100): print(n, end=' ') # 0 1 1 2 3 5 8 13 21 34 55 89 ```
Benefits of Generator iter¶
```python class Lines: def init(self, filename): self.filename = filename
def __iter__(self):
with open(self.filename) as f:
for line in f:
yield line.strip()
Memory efficient - processes one line at a time¶
File automatically closed when iteration completes¶
Can iterate multiple times¶
```
next: Iterator Protocol¶
```python class InfiniteCounter: """Infinite iterator that never raises StopIteration."""
def __init__(self, start=0):
self.value = start
def __iter__(self):
return self
def __next__(self):
current = self.value
self.value += 1
return current
counter = InfiniteCounter() print(next(counter)) # 0 print(next(counter)) # 1 print(next(counter)) # 2
Use with islice to limit¶
from itertools import islice print(list(islice(InfiniteCounter(10), 5))) # [10, 11, 12, 13, 14] ```
reversed: Reverse Iteration¶
```python class Playlist: def init(self, songs): self._songs = list(songs)
def __iter__(self):
return iter(self._songs)
def __reversed__(self):
return iter(self._songs[::-1])
def __len__(self):
return len(self._songs)
songs = Playlist(['A', 'B', 'C', 'D'])
print("Forward:", list(songs))
Forward: ['A', 'B', 'C', 'D']¶
print("Reversed:", list(reversed(songs)))
Reversed: ['D', 'C', 'B', 'A']¶
```
Generator-Based reversed¶
```python class LinkedList: class Node: def init(self, value, next_node=None): self.value = value self.next = next_node
def __init__(self):
self.head = None
def prepend(self, value):
self.head = self.Node(value, self.head)
def __iter__(self):
node = self.head
while node:
yield node.value
node = node.next
def __reversed__(self):
# Collect and reverse (O(n) space)
values = list(self)
for value in reversed(values):
yield value
ll = LinkedList() for v in [1, 2, 3, 4]: ll.prepend(v)
print(list(ll)) # [4, 3, 2, 1] print(list(reversed(ll))) # [1, 2, 3, 4] ```
How for Triggers Iteration
When Python encounters for item in obj, it calls iter(obj), which is equivalent to type(obj).__iter__(obj) — looking up __iter__ on the class, not the instance. This is the same class-based lookup used by all protocols (__call__, __contains__, __enter__, etc.). The unified model: Python syntax triggers a method lookup on the type, calls it if found, tries a fallback if not, or raises an error.
getitem Fallback¶
If __iter__ isn't defined, Python tries __getitem__:
```python class OldStyleSequence: """Works with for loops via getitem."""
def __init__(self, data):
self._data = data
def __getitem__(self, index):
return self._data[index]
old = OldStyleSequence([1, 2, 3]) for item in old: print(item) # Works! Calls getitem(0), getitem(1), etc. ```
Iteration in Different Contexts¶
```python class Numbers: def init(self, data): self._data = data
def __iter__(self):
return iter(self._data)
nums = Numbers([1, 2, 3, 4, 5])
for loop¶
for n in nums: print(n)
List comprehension¶
squares = [n**2 for n in nums]
Generator expression¶
evens = (n for n in nums if n % 2 == 0)
Unpacking¶
a, b, c, d, e = nums
Built-in functions¶
print(sum(nums)) # 15 print(max(nums)) # 5 print(list(nums)) # [1, 2, 3, 4, 5] print(tuple(nums)) # (1, 2, 3, 4, 5) print(set(nums)) # {1, 2, 3, 4, 5} print(sorted(nums, reverse=True)) # [5, 4, 3, 2, 1]
in operator (uses iter if no contains)¶
print(3 in nums) # True
any/all¶
print(any(n > 4 for n in nums)) # True print(all(n > 0 for n in nums)) # True ```
Practical Example: File-Like Iteration¶
```python class CSVReader: """Iterate over CSV file as dictionaries."""
def __init__(self, filename):
self.filename = filename
def __iter__(self):
with open(self.filename) as f:
headers = None
for line in f:
values = line.strip().split(',')
if headers is None:
headers = values
else:
yield dict(zip(headers, values))
Usage¶
for row in CSVReader('data.csv'):¶
print(row['name'], row['age'])¶
```
Practical Example: Database-Like Iteration¶
```python class QueryResult: """Iterate over query results with lazy loading."""
def __init__(self, data, batch_size=100):
self._data = data
self._batch_size = batch_size
def __iter__(self):
for i in range(0, len(self._data), self._batch_size):
batch = self._data[i:i + self._batch_size]
for item in batch:
yield item
def __len__(self):
return len(self._data)
Simulated usage¶
results = QueryResult(list(range(1000)), batch_size=100) for row in results: if row > 10: break print(row) ```
Practical Example: Tree Traversal¶
```python class TreeNode: def init(self, value, children=None): self.value = value self.children = children or []
def __iter__(self):
"""Pre-order traversal."""
yield self.value
for child in self.children:
yield from child # Recursive iteration
def __reversed__(self):
"""Post-order traversal."""
for child in reversed(self.children):
yield from reversed(child)
yield self.value
Build tree: 1¶
# / | \
2 3 4¶
# / \
5 6¶
tree = TreeNode(1, [ TreeNode(2, [TreeNode(5), TreeNode(6)]), TreeNode(3), TreeNode(4) ])
print("Pre-order:", list(tree))
Pre-order: [1, 2, 5, 6, 3, 4]¶
print("Post-order:", list(reversed(tree)))
Post-order: [4, 3, 6, 5, 2, 1]¶
```
Async Iteration (aiter, anext)¶
For async contexts (Python 3.5+):
```python class AsyncRange: def init(self, start, stop): self.start = start self.stop = stop
def __aiter__(self):
self.current = self.start
return self
async def __anext__(self):
if self.current >= self.stop:
raise StopAsyncIteration
await asyncio.sleep(0.1) # Simulate async work
value = self.current
self.current += 1
return value
Usage¶
async for n in AsyncRange(0, 5):¶
print(n)¶
```
Iterator vs Iterable¶
```python
Iterable: has iter, can be iterated multiple times¶
class Iterable: def init(self, data): self._data = data
def __iter__(self):
return iter(self._data) # Returns NEW iterator
Iterator: has iter AND next, typically single-use¶
class Iterator: def init(self, data): self._data = data self._index = 0
def __iter__(self):
return self # Returns SELF
def __next__(self):
if self._index >= len(self._data):
raise StopIteration
value = self._data[self._index]
self._index += 1
return value
Testing¶
iterable = Iterable([1, 2, 3]) print(list(iterable)) # [1, 2, 3] print(list(iterable)) # [1, 2, 3] - works again!
iterator = Iterator([1, 2, 3]) print(list(iterator)) # [1, 2, 3] print(list(iterator)) # [] - exhausted! ```
Using collections.abc¶
```python from collections.abc import Iterator, Iterable
class MyIterator(Iterator): """Only need to implement next."""
def __init__(self, limit):
self.limit = limit
self.current = 0
def __next__(self):
if self.current >= self.limit:
raise StopIteration
self.current += 1
return self.current
iter is provided by Iterator ABC¶
it = MyIterator(3) print(list(it)) # [1, 2, 3] ```
The Single-Use Iterator Trap¶
This is one of the most common bugs in real Python code. An iterator is exhausted after one pass --- iterating again silently yields nothing:
python
numbers = iter([1, 2, 3])
print(sum(numbers)) # 6
print(sum(numbers)) # 0 — silently empty!
To allow multiple iterations, your class should be an iterable (returns a fresh iterator from __iter__ each time), not an iterator itself. If your __iter__ returns self and you also define __next__, the object is single-use.
Lazy vs Eager Evaluation¶
Generators and iterators are lazy --- they produce values one at a time, using constant memory regardless of dataset size. Lists are eager --- they compute and store all values upfront. Use lazy iteration when the dataset is large or potentially infinite; use eager evaluation when you need random access or multiple passes.
Key Takeaways¶
__iter__returns an iterator object__next__returns the next value or raisesStopIteration- Watch for single-use iterators --- if
__iter__returnsself, the object can only be iterated once - Use generators in
__iter__for simple cases (automatically creates fresh iterators) - Separate iterator classes allow multiple simultaneous iterations
- Lazy iteration (generators) saves memory; eager evaluation (lists) allows random access
__reversed__enables customreversed()behavior__getitem__provides fallback iteration if__iter__is missing- Use
collections.abcbase classes for compliance yield fromenables clean recursive iteration
When NOT to Implement iter
Don't implement __iter__ when your object is not conceptually a collection or sequence. If iterating over an object has no obvious meaning — what would for item in user: produce? — the iteration protocol is the wrong fit. Use explicit methods like user.get_permissions() instead. Protocols should reveal intent, not obscure it.
Exercises¶
Exercise 1.
Create a Range class that mimics Python's range() for integers. Implement __iter__ (returns a new iterator each time) and __len__. Demonstrate that the same Range object can be iterated multiple times. Use a separate iterator class or a generator in __iter__.
Solution to Exercise 1
class Range:
def __init__(self, start, stop=None, step=1):
if stop is None:
self.start, self.stop = 0, start
else:
self.start, self.stop = start, stop
self.step = step
def __iter__(self):
current = self.start
while current < self.stop:
yield current
current += self.step
def __len__(self):
return max(0, (self.stop - self.start + self.step - 1) // self.step)
r = Range(1, 6)
print(list(r)) # [1, 2, 3, 4, 5]
print(list(r)) # [1, 2, 3, 4, 5] — can iterate again
print(len(r)) # 5
Exercise 2.
Write a FileLines iterator class that takes a filename and yields lines one at a time (simulate with a list of strings). Implement __iter__ (returns self) and __next__ (returns next line, raises StopIteration when done). Show it works in a for loop.
Solution to Exercise 2
class FileLines:
def __init__(self, lines):
self.lines = lines
self.index = 0
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.lines):
raise StopIteration
line = self.lines[self.index]
self.index += 1
return line
lines = ["First line", "Second line", "Third line"]
reader = FileLines(lines)
for line in reader:
print(line)
# First line
# Second line
# Third line
Exercise 3.
Build a FibonacciSequence class where __iter__ returns a generator that yields Fibonacci numbers up to a maximum value. For example, FibonacciSequence(100) yields 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89. Show it works with list() and for loops, and can be iterated multiple times.
Solution to Exercise 3
class FibonacciSequence:
def __init__(self, max_value):
self.max_value = max_value
def __iter__(self):
a, b = 0, 1
while b <= self.max_value:
yield b
a, b = b, a + b
fib = FibonacciSequence(100)
print(list(fib)) # [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
print(list(fib)) # [1, 1, 2, ...] — can iterate again
for n in FibonacciSequence(20):
print(n, end=" ") # 1 1 2 3 5 8 13
Exercise 4.
Demonstrate the single-use iterator trap. Create a class Squares that is an iterator (returns self from __iter__, defines __next__). Show that iterating twice over the same instance gives results the first time but an empty sequence the second time. Then fix it by making Squares an iterable (returns a new iterator from __iter__ each time). Explain the design principle.
Solution to Exercise 4
# Bug: single-use iterator
class SquaresIterator:
def __init__(self, n):
self.n = n
self.i = 0
def __iter__(self):
return self # Returns self — single-use!
def __next__(self):
if self.i >= self.n:
raise StopIteration
val = self.i ** 2
self.i += 1
return val
sq = SquaresIterator(4)
print(list(sq)) # [0, 1, 4, 9]
print(list(sq)) # [] — exhausted!
# Fix: reusable iterable
class Squares:
def __init__(self, n):
self.n = n
def __iter__(self):
for i in range(self.n):
yield i ** 2 # Fresh generator each time
sq = Squares(4)
print(list(sq)) # [0, 1, 4, 9]
print(list(sq)) # [0, 1, 4, 9] — works again!
Design principle: an iterable creates a new iterator on each call to __iter__, so it can be iterated multiple times. An iterator returns self from __iter__ and tracks position internally, so it can only be consumed once. Use generators in __iter__ to get reusable iteration with minimal code.
Exercise 5.
Build a Zipper class that takes two iterables and yields pairs, like the built-in zip(). Implement __iter__ using a generator. Handle the case where the iterables have different lengths (stop at the shorter one). Then add a longest parameter that, when True, pads the shorter iterable with None (like itertools.zip_longest).
Solution to Exercise 5
class Zipper:
def __init__(self, iter_a, iter_b, longest=False):
self._a = iter_a
self._b = iter_b
self._longest = longest
def __iter__(self):
it_a = iter(self._a)
it_b = iter(self._b)
while True:
a_done = False
b_done = False
try:
val_a = next(it_a)
except StopIteration:
a_done = True
val_a = None
try:
val_b = next(it_b)
except StopIteration:
b_done = True
val_b = None
if a_done and b_done:
return
if (a_done or b_done) and not self._longest:
return
yield (val_a, val_b)
# Shortest (default)
print(list(Zipper([1, 2, 3], ["a", "b"])))
# [(1, 'a'), (2, 'b')]
# Longest
print(list(Zipper([1, 2, 3], ["a", "b"], longest=True)))
# [(1, 'a'), (2, 'b'), (3, None)]
# Reusable
z = Zipper("AB", [1, 2])
print(list(z)) # [('A', 1), ('B', 2)]
print(list(z)) # [('A', 1), ('B', 2)] — works again