advanced

Iterators and Generators in Python

By DataPlayArena Technical Review BoardUpdated 2026-07-209 min readRuntime Verified

Overview

In data engineering and high-throughput systems, loading an entire multi-gigabyte dataset into memory at once causes Out-Of-Memory (OOM) crashes. Python solves this through the Iterator Protocol and Generators. Iterators and generators evaluate elements lazily—one at a time on demand—allowing you to stream, filter, and process datasets of arbitrary or infinite size with a constant, minimal memory footprint.

What You Will Learn

In this lesson, you will master:
  • The Iterator Protocol: Understanding the relationship between Iterable, Iterator, __iter__(), and __next__().
  • Generator Functions & yield: How the yield statement freezes execution state between calls.
  • Generator Expressions: The syntax and memory savings of (x for x in data) vs [x for x in data].
  • Pipeline Chaining: Composing streaming transformations using Python's standard itertools library.
  • Two-Way Communication: Using generator .send() and .close() methods for coroutine architectures.

Core Concepts & Mechanics

1. The Iterator Protocol

Python distinguishes between an iterable (an object that can return an iterator) and an iterator (the stream state object producing values):

  1. Iterable: Implements __iter__(), which returns an iterator object (e.g., list, dict, set, str).
  2. Iterator: Implements __iter__() (returning self) and __next__(). Calling next(it) returns the subsequent item. When no items remain, it raises StopIteration.
python
numbers = [1, 2, 3]

# 1. Obtain an iterator from the iterable
it = iter(numbers)  # calls numbers.__iter__()

# 2. Advance the stream manually
print(next(it))  # 1  (calls it.__next__())
print(next(it))  # 2
print(next(it))  # 3

# 3. Exhaustion
# next(it) -> Raises StopIteration!

Every Python for loop is simply syntactic sugar over this exact protocol:

python
# What Python actually does when executing: for x in numbers:
iterator = iter(numbers)
while True:
    try:
        x = next(iterator)
        # loop body executes here
    except StopIteration:
        break

Generator Functions & The yield Keyword

A generator function is a function that contains one or more yield statements. Unlike a standard function that returns a value and destroys its local stack frame, a generator pauses execution at the yield point, preserves all local variables in memory, and resumes immediately after the yield upon the next next() call.

python
def stream_log_lines(filepath: str):
    """Streams a massive log file line by line without loading the full file into RAM."""
    with open(filepath, "r", encoding="utf-8") as f:
        for line in f:
            if "ERROR" in line:
                yield line.strip()

# Calling the function does NOT run the code; it returns a generator object
log_gen = stream_log_lines("/var/log/app.log")
print(type(log_gen))  # <class 'generator'>

# Data is pulled lazily when iterated
for error_line in log_gen:
    print(f"Alert: {error_line}")

Memory Footprint Comparison: List vs Generator

Consider generating a sequence of 10,000,000 integers:

python
import sys

# 1. List Comprehension (Eager Materialization)
# Allocates all 10 million integers in RAM simultaneously
list_comp = [x * 2 for x in range(10_000_000)]
print(f"List Memory: {sys.getsizeof(list_comp) / (1024 * 1024):.2f} MB")  # ~79.2 MB

# 2. Generator Expression (Lazy Stream)
# Allocates only the generator state machine
gen_exp = (x * 2 for x in range(10_000_000))
print(f"Gen Memory:  {sys.getsizeof(gen_exp)} bytes")  # ~104 bytes!

The generator expression consumes 104 bytes whether processing 10 items, 10 million items, or an infinite stream.


Building Composable Streaming Data Pipelines

You can chain generators together to form clean, declarative, Unix-style streaming pipelines:

python
# Step 1: Source generator (reads raw sensor data)
def raw_readings():
    readings = ["23.4", "INVALID", "24.1", "N/A", "25.0", "22.8"]
    for r in readings:
        yield r

# Step 2: Cleaning filter generator
def valid_floats(stream):
    for item in stream:
        try:
            yield float(item)
        except ValueError:
            continue

# Step 3: Transformation generator (Celsius to Fahrenheit)
def celsius_to_fahrenheit(stream):
    for c in stream:
        yield (c * 9/5) + 32

# Assemble the streaming pipeline
pipeline = celsius_to_fahrenheit(valid_floats(raw_readings()))

for temp_f in pipeline:
    print(f"{temp_f:.1f}°F")
# Outputs: 74.1°F, 75.4°F, 77.0°F, 73.0°F

Common Mistakes & Gotchas

  • Generator Exhaustion: Generators are one-time streams. Once consumed, a generator remains empty. Attempting to iterate over it a second time produces nothing without any warning or error:
    python
    g = (x for x in [1, 2, 3])
    list(g)  # [1, 2, 3]
    list(g)  # [] -> Empty! Must re-instantiate generator
    
  • Calling len() on a Generator: You cannot call len(generator) because the length of a lazy stream is unknown until it is exhausted. Doing so raises TypeError: object of type 'generator' has no len().
  • Indexing a Generator: Generators do not support indexing (gen[0] raises TypeError). Use next(gen) or itertools.islice() to access items.

Performance & Best Practices

  • Use itertools for Fast Stream Processing: Python’s standard itertools module (islice, chain, groupby, tee) provides C-optimized building blocks for generators:
    python
    import itertools
    
    # Take only the first 5 elements of a generator
    first_five = list(itertools.islice(pipeline, 5))
    
  • Leverage yield from for Sub-generators: Instead of writing for item in sub_generator: yield item, use yield from sub_generator, which delegates directly to the sub-generator at C speed and handles two-way exception passing.

Interview Perspective

Python Interview Question: What is the difference between an Iterable and an Iterator, and how does yield work under the hood?

Key Talking Points for Candidates:

  1. Iterable vs Iterator: An iterable implements __iter__() to produce an iterator. An iterator implements __next__() to produce values sequentially and __iter__() returning itself.
  2. Stack Frame Preservation: When a function contains yield, CPython flags the function with CO_GENERATOR. Calling it allocates a PyGenObject wrapping the function frame.
  3. Execution Freezing: When execution hits yield, Python saves the current instruction pointer (f_lasti) and local variables in the frame heap, yielding control back to the caller. Upon the next invocation, Python resumes execution directly at the saved instruction pointer.

Interactive Practice Challenge

Challenge: Chunked Stream Generator

Write a generator function called chunk_stream(iterable, chunk_size) that accepts any iterable and yields successive lists of size chunk_size. The final chunk may contain fewer elements if the total count is not evenly divisible.

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.