Dict & Set Comprehensions in Python
Overview
Just as list comprehensions provide an expressive syntax for constructing lists, Python provides dictionary comprehensions ({k: v for ...}) and set comprehensions ({v for ...}). These constructs allow data engineers to build lookups, invert index mappings, deduplicate data streams, and group tabular metrics into key-value stores with optimal speed and concise, readable code.
What You Will Learn
In this lesson, you will master:
- Dictionary Comprehension Syntax: Building
{key_expr: val_expr for item in iterable if condition}. - Set Comprehension Syntax: Constructing unique collections
{val_expr for item in iterable}. - Key Inversion: Reversing mapping relationships (values becoming keys).
- Data Aggregation & Grouping: Creating grouped lookup structures directly from tabular datasets.
- Duplicate Key Overwriting: Understanding how Python handles duplicate keys generated during comprehension execution.
Dictionary Comprehensions
1. Basic Syntax & Lookup Construction
A dictionary comprehension constructs a key-value mapping in a single pass:
users = [
{"id": 101, "name": "Alice", "role": "admin"},
{"id": 102, "name": "Bob", "role": "viewer"},
{"id": 103, "name": "Charlie", "role": "editor"},
]
# Fast O(1) user ID to name lookup table
user_lookup = {user["id"]: user["name"] for user in users}
print(user_lookup)
# Output: {101: 'Alice', 102: 'Bob', 103: 'Charlie'}
# Filtering: Map only privileged users
admin_lookup = {user["id"]: user["name"] for user in users if user["role"] in ("admin", "editor")}
print(admin_lookup)
# Output: {101: 'Alice', 103: 'Charlie'}
2. Inverting Dictionaries (Bidirectional Lookups)
When converting a code-to-description mapping into a description-to-code reverse lookup:
status_codes = {
200: "OK",
400: "BAD_REQUEST",
404: "NOT_FOUND",
500: "INTERNAL_SERVER_ERROR"
}
# Invert keys and values
code_by_status = {status: code for code, status in status_codes.items()}
print(code_by_status)
# Output: {'OK': 200, 'BAD_REQUEST': 400, 'NOT_FOUND': 404, 'INTERNAL_SERVER_ERROR': 500}
Collision Warning on Key Inversion
If the original dictionary contains duplicate values (e.g., {1: "A", 2: "A"}), inverting it will overwrite previously evaluated keys! The resulting dictionary will retain only the last encountered key for that value.
Set Comprehensions
Set comprehensions use the same {} syntax as dictionary comprehensions, but specify a single expression instead of a key: value pair:
log_entries = [
"INFO: Server started",
"WARN: Disk usage high",
"ERROR: DB connection timeout",
"INFO: User login",
"ERROR: Out of memory"
]
# Extract unique log levels
log_levels = {entry.split(":")[0] for entry in log_entries}
print(log_levels)
# Output: {'INFO', 'WARN', 'ERROR'} (order not guaranteed)
Set comprehensions automatically discard duplicate values during construction, producing a deduplicated hash set without intermediate list allocation.
Practical Applications in Data Engineering
1. Grouping and Aggregating by Category
Combining comprehensions with Python collections:
transactions = [
("engineering", 450.0),
("marketing", 200.0),
("engineering", 300.0),
("sales", 150.0),
("marketing", 120.0),
]
# Extract unique departments using a set comprehension
departments = {dept for dept, _ in transactions}
# Build department totals using dictionary comprehension
dept_totals = {
dept: sum(amount for d, amount in transactions if d == dept)
for dept in departments
}
print(dept_totals)
# Output: {'engineering': 750.0, 'marketing': 320.0, 'sales': 150.0}
2. Normalizing and Cleansing Raw Headers
raw_payload = {
" First_Name ": "John",
"LAST_NAME": "Doe",
" EMAIL_ADDRESS ": "john.doe@example.com",
"AGE": 30
}
# Clean all keys to lowercase snake_case
clean_payload = {
key.strip().lower().replace(" ", "_"): val
for key, val in raw_payload.items()
}
print(clean_payload)
# Output: {'first_name': 'John', 'last_name': 'Doe', 'email_address': 'john.doe@example.com', 'age': 30}
Common Mistakes & Gotchas
- Empty
{}Ambiguity: Remember that{}always denotes an empty dictionary, never an empty set. To create an empty set, useset(). - Unhashable Keys in Dictionary Comprehension: The key expression must evaluate to a hashable type. Attempting
{list(val): key for key, val in data.items()}triggersTypeError: unhashable type: 'list'. - Hidden O(N^2) Nested Iteration: In the aggregation example above, nesting
sum(... for ... in transactions)inside a dictionary comprehension iterates through the entire list for every department. For large datasets ($> 100,000$ rows), usecollections.defaultdict(float)to achieve $O(N)$ linear time.
Performance & Best Practices
- Use
.items()for Iteration: When transforming an existing dictionary, always iterate overdict.items()to extract both key and value simultaneously, avoiding redundantdict[k]lookups. - Prefer Dict Comprehensions over
dict(zip(...)): Whiledict(zip(keys, values))is common, a dictionary comprehension{k: v for k, v in zip(keys, values)}allows inline filtering and transformation before inserting into the dictionary.
Interview Perspective
Python Interview Question: What happens during a dictionary comprehension if multiple items produce identical keys?
Key Talking Points for Candidates:
- Last-Write-Wins Rule: If two iterations evaluate to the same key, Python replaces the existing value with the newly computed value for that key.
- Deterministic Outcome: The resulting dictionary will contain the value from the last item encountered in the iteration order.
- Mitigation: If all values must be preserved for duplicated keys, you must accumulate them into a list or set using
collections.defaultdict(list)instead of a standard dictionary comprehension.
Interactive Practice Challenge
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.