Iterators and Generators in Python
Overview
In data engineering and high-throughput systems, loading an entire multi-gigabyte dataset into memory at once causes Out-Of-Memory (OOM) crashes. Python solves this through the Iterator Protocol and Generators. Iterators and generators evaluate elements lazily—one at a time on demand—allowing you to stream, filter, and process datasets of arbitrary or infinite size with a constant, minimal memory footprint.
What You Will Learn
In this lesson, you will master:
- The Iterator Protocol: Understanding the relationship between
Iterable,Iterator,__iter__(), and__next__(). - Generator Functions &
yield: How theyieldstatement freezes execution state between calls. - Generator Expressions: The syntax and memory savings of
(x for x in data)vs[x for x in data]. - Pipeline Chaining: Composing streaming transformations using Python's standard
itertoolslibrary. - Two-Way Communication: Using generator
.send()and.close()methods for coroutine architectures.
Core Concepts & Mechanics
1. The Iterator Protocol
Python distinguishes between an iterable (an object that can return an iterator) and an iterator (the stream state object producing values):
- Iterable: Implements
__iter__(), which returns an iterator object (e.g.,list,dict,set,str). - Iterator: Implements
__iter__()(returningself) and__next__(). Callingnext(it)returns the subsequent item. When no items remain, it raisesStopIteration.
numbers = [1, 2, 3]
# 1. Obtain an iterator from the iterable
it = iter(numbers) # calls numbers.__iter__()
# 2. Advance the stream manually
print(next(it)) # 1 (calls it.__next__())
print(next(it)) # 2
print(next(it)) # 3
# 3. Exhaustion
# next(it) -> Raises StopIteration!
Every Python for loop is simply syntactic sugar over this exact protocol:
# What Python actually does when executing: for x in numbers:
iterator = iter(numbers)
while True:
try:
x = next(iterator)
# loop body executes here
except StopIteration:
break
Generator Functions & The yield Keyword
A generator function is a function that contains one or more yield statements. Unlike a standard function that returns a value and destroys its local stack frame, a generator pauses execution at the yield point, preserves all local variables in memory, and resumes immediately after the yield upon the next next() call.
def stream_log_lines(filepath: str):
"""Streams a massive log file line by line without loading the full file into RAM."""
with open(filepath, "r", encoding="utf-8") as f:
for line in f:
if "ERROR" in line:
yield line.strip()
# Calling the function does NOT run the code; it returns a generator object
log_gen = stream_log_lines("/var/log/app.log")
print(type(log_gen)) # <class 'generator'>
# Data is pulled lazily when iterated
for error_line in log_gen:
print(f"Alert: {error_line}")
Memory Footprint Comparison: List vs Generator
Consider generating a sequence of 10,000,000 integers:
import sys
# 1. List Comprehension (Eager Materialization)
# Allocates all 10 million integers in RAM simultaneously
list_comp = [x * 2 for x in range(10_000_000)]
print(f"List Memory: {sys.getsizeof(list_comp) / (1024 * 1024):.2f} MB") # ~79.2 MB
# 2. Generator Expression (Lazy Stream)
# Allocates only the generator state machine
gen_exp = (x * 2 for x in range(10_000_000))
print(f"Gen Memory: {sys.getsizeof(gen_exp)} bytes") # ~104 bytes!
The generator expression consumes 104 bytes whether processing 10 items, 10 million items, or an infinite stream.
Building Composable Streaming Data Pipelines
You can chain generators together to form clean, declarative, Unix-style streaming pipelines:
# Step 1: Source generator (reads raw sensor data)
def raw_readings():
readings = ["23.4", "INVALID", "24.1", "N/A", "25.0", "22.8"]
for r in readings:
yield r
# Step 2: Cleaning filter generator
def valid_floats(stream):
for item in stream:
try:
yield float(item)
except ValueError:
continue
# Step 3: Transformation generator (Celsius to Fahrenheit)
def celsius_to_fahrenheit(stream):
for c in stream:
yield (c * 9/5) + 32
# Assemble the streaming pipeline
pipeline = celsius_to_fahrenheit(valid_floats(raw_readings()))
for temp_f in pipeline:
print(f"{temp_f:.1f}°F")
# Outputs: 74.1°F, 75.4°F, 77.0°F, 73.0°F
Common Mistakes & Gotchas
- Generator Exhaustion: Generators are one-time streams. Once consumed, a generator remains empty. Attempting to iterate over it a second time produces nothing without any warning or error:
python
g = (x for x in [1, 2, 3]) list(g) # [1, 2, 3] list(g) # [] -> Empty! Must re-instantiate generator - Calling
len()on a Generator: You cannot calllen(generator)because the length of a lazy stream is unknown until it is exhausted. Doing so raisesTypeError: object of type 'generator' has no len(). - Indexing a Generator: Generators do not support indexing (
gen[0]raisesTypeError). Usenext(gen)oritertools.islice()to access items.
Performance & Best Practices
- Use
itertoolsfor Fast Stream Processing: Python’s standarditertoolsmodule (islice,chain,groupby,tee) provides C-optimized building blocks for generators:pythonimport itertools # Take only the first 5 elements of a generator first_five = list(itertools.islice(pipeline, 5)) - Leverage
yield fromfor Sub-generators: Instead of writingfor item in sub_generator: yield item, useyield from sub_generator, which delegates directly to the sub-generator at C speed and handles two-way exception passing.
Interview Perspective
Python Interview Question: What is the difference between an Iterable and an Iterator, and how does yield work under the hood?
Key Talking Points for Candidates:
- Iterable vs Iterator: An iterable implements
__iter__()to produce an iterator. An iterator implements__next__()to produce values sequentially and__iter__()returning itself. - Stack Frame Preservation: When a function contains
yield, CPython flags the function withCO_GENERATOR. Calling it allocates aPyGenObjectwrapping the function frame. - Execution Freezing: When execution hits
yield, Python saves the current instruction pointer (f_lasti) and local variables in the frame heap, yielding control back to the caller. Upon the next invocation, Python resumes execution directly at the saved instruction pointer.
Interactive Practice Challenge
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.