Python Iterators & Generators
In Python, iteration is one of the most powerful and fundamental language features. Under the hood, iteration is governed by the formal Iterator Protocol consisting of two distinct roles:
1. What is an Iterable?
An Iterable is any Python object capable of returning its members one at a time. Examples include lists, strings, tuples, dictionaries, sets, and open file objects. An object qualifies as an iterable if it implements the __iter__() method (or __getitem__() with sequential integer indices).
2. What is an Iterator?
An Iterator is the stateful stream object that actually produces values during traversal. An iterator maintains an internal cursor in memory and must implement two methods:
__iter__(): Returns the iterator object itself.__next__(): Returns the next item from the container. If no further items remain, it must raise theStopIterationexception.
# Demonstrating the Iterator Protocol manually step-by-step:
fruits = ["Apple", "Mango", "Banana"]
# Step 1: Obtain an iterator from the iterable list:
fruit_stream = iter(fruits)
print("Iterator Type:", type(fruit_stream))
# Step 2: Fetch elements one by one via next():
print("1st Fetch:", next(fruit_stream)) # 'Apple'
print("2nd Fetch:", next(fruit_stream)) # 'Mango'
print("3rd Fetch:", next(fruit_stream)) # 'Banana'
# Step 3: Stream is now exhausted; subsequent call raises StopIteration:
try:
next(fruit_stream)
except StopIteration:
print("🛑 StopIteration Exception Caught: Stream fully exhausted!")When you write for x in fruits:, Python executes three exact C-level steps: (1) calls iter(fruits), (2) enters an infinite loop repeatedly calling next(), and (3) catches StopIteration to terminate the loop cleanly without crashing!
A Generator is a special function that produces a sequence of values lazily on demand. Unlike standard functions that compute everything up-front and return a complete list in RAM, generators calculate each item only when requested.
The Crucial Difference Between return and yield:
return: Computes the final value, destroys the function's local execution stack frame and local variables, and returns control to the caller.yield: Pauses execution, freezes the entire execution stack frame in RAM (all local variable values, loop counters, and instruction pointers), sends the yielded value to the caller, and waits. Whennext()is called again, execution resumes immediately at the exact line afteryield!
# Generator function demonstrating state suspension:
def countdown_timer(start_seconds):
"""Yields countdown numbers lazily."""
current = start_seconds
while current > 0:
print(f" [Generator Internals] Yielding {current} and freezing state...")
yield current
current -= 1 # Resumes here on next call!
print(" [Generator Internals] Countdown complete!")
print("--- Calling Generator Function ---")
# Calling the generator function returns a generator object instantly without executing code:
timer_gen = countdown_timer(3)
print("Generator Object Created:", timer_gen)
print("\n--- Iterating Through Values Lazily ---")
for sec in timer_gen:
print(f"👉 Received: {sec}s remaining")If start_seconds = 100,000,000, creating a list list(range(100_000_000)) would require ~800 Megabytes of RAM. The generator requires only ~120 Bytes because it produces numbers one at a time on the fly.
A Generator Expression uses the exact same syntax as a list comprehension, but replaces square brackets [...] with parentheses (...).
| Feature | List Comprehension [...] | Generator Expression (...) |
|---|---|---|
| Evaluation | Eager (Calculates all elements immediately) | Lazy (Calculates elements one-by-one on demand) |
| Memory | $O(N)$ memory (Scales linearly with dataset size) | $O(1)$ constant memory (~100 bytes regardless of size) |
| Access | Indexed random access (lst[5]) and slicing | Sequential streaming access only (via next() or loop) |
| Reusability | Can be iterated multiple times | Exhausted after a single iteration pass |
import sys
# 1. Eager List Comprehension (Allocates memory for 1,000,000 items in RAM):
list_squares = [x ** 2 for x in range(1_000_000)]
# 2. Lazy Generator Expression (Allocates only lightweight generator state object):
gen_squares = (x ** 2 for x in range(1_000_000))
print(f"📦 List Comprehension Memory: {sys.getsizeof(list_squares):,} bytes (~8.4 MB)")
print(f"⚡ Generator Expression Memory: {sys.getsizeof(gen_squares):,} bytes (Constantly tiny!)")
# Passing generator expressions directly into aggregation functions:
total_sum = sum(x ** 2 for x in range(1000)) # Parentheses can be omitted inside functions!
print(f"\nSum of squares up to 1000: {total_sum:,}")When passing data into aggregation functions like sum(), max(), min(), or any(), always use a generator expression without square brackets to avoid creating unnecessary intermediate lists in RAM.
Generators are one-way data streams. Once an iterator reaches the end and raises StopIteration, subsequent for-loops over that instance will execute 0 times. To re-iterate, you must instantiate a fresh generator.
Write a generator fibonacci(limit) that yields Fibonacci numbers up to limit. Print all generated numbers in a single line.
def fibonacci(limit):
a, b = 0, 1
while a <= limit:
yield a
a, b = b, a + b
print("Fibonacci numbers up to 100:")
for num in fibonacci(100):
print(num, end=" ")
print()Q What is the difference between yield and yield from in Python?
yield from iterable is used to delegate part of a generator's operations to another sub-generator or iterable, cleanly transparently forwarding values.
Q Can generators receive data from the caller while running?
Yes! You can pass values into a running generator using generator.send(value). Inside the generator, the yield expression evaluates to the received value.
Q Are generators thread-safe in Python?
Calling next() on the same generator instance simultaneously from multiple threads is not thread-safe without explicit synchronization (threading.Lock).