Generators & Iterators

Overview

Generators yield values lazily, conserving memory when processing large datasets.

Learning Objectives

• Write generator functions using the yield keyword.
• Create lazy generator expressions.
• Understand iterator protocols.

Concept Explanation

Generators are special functions that return lazy iterables using the yield keyword. Unlike standard functions that construct and return an entire list at once, generators yield values one-at-a-time on demand, consuming O(1) memory space regardless of the dataset size.

Code Examples

Example 1 — Basics

This example introduces the fundamental syntax and concepts.

def count_three():
    yield 1
    yield 2
    yield 3

Example 2 — Everyday Usage

This example demonstrates a realistic scenario handling business parameters.

# Stream file lines lazily to save memory
def read_logs(filepath):
    with open(filepath, 'r') as f:
        for line in f:
            yield line.strip()

# Lines are loaded one-at-a-time, conserving memory

Example 3 — Advanced Example

This example shows clean, production-grade code structure following senior development standards.

# Generate Fibonacci sequence numbers indefinitely without crashing
def fibonacci_generator(limit):
    a, b = 0, 1
    count = 0
    while count < limit:
        yield a
        a, b = b, a + b
        count += 1

for num in fibonacci_generator(10):
    print(num, end=' ')

Visual Flow

The following execution flow represents the step-by-step evaluation inside the interpreter:

Call generator → Return iterator object → Request next value → Run code to yield → Pause state → Return value

Common Mistakes

Review these common pitfalls when working with this topic:

• Converting generators immediately to lists (e.g. list(gen)), which wastes memory.
• Forgetting that generators are single-use; once consumed, they cannot be reset.
• Confusing yield (pauses execution) with return (stops execution).
• Attempting index queries on generator objects.

Best Practices

Enforce these Pythonic best practices in your codebase:

squares = [x*x for x in range(1000000)] # Consumes significant memory

squares = (x*x for x in range(1000000)) # Generator consumes minimal memory

Performance Notes

Keep these optimization guidelines in mind for performance-sensitive hotpaths:

• Generators have O(1) memory complexity, making them essential for processing massive datasets or parsing large files.

Quick Revision

Use these key summaries for last-minute revision:

• yield returns values and pauses function execution.
• Generators generate values lazily on demand.
• They consume O(1) memory space.
• Generators cannot be reset or reused.
• Exhausted generators raise StopIteration.
• Generator expressions use parentheses.