DataFrame Transformations
Overview
Learn about Spark transformations, including the difference between narrow and wide transformations and how lineage graphs track operations.
What You Will Learn
In this lesson, you will learn:
- Transformations API: The basic logic of operations that return new DataFrames.
- Lazy Evaluation: How Spark postpones physical execution until an action runs.
- Narrow vs Wide: Distinguishing map-like filters from shuffle-heavy operations.
Detailed Concept Explanation
In Spark, operations are divided into two main categories: Transformations and Actions.
A Transformation takes an existing DataFrame and returns a new one. It does not modify the source data because DataFrames are immutable.
Spark divides transformations into two types based on partition operations:
- Narrow Transformations: Operations where each input partition contributes to exactly one output partition (e.g.
filter()orselect()). No network shuffle is required. - Wide Transformations: Operations where data must be partitioned across the network to compute results (e.g.
groupBy()ororderBy()). This triggers a network shuffle.
All transformations are evaluated lazily. When you write transformation code, Spark adds the operation to a Directed Acyclic Graph (DAG) lineage instead of running it immediately.
Code Examples
Input Dataset Preview
Below is the dataset we will apply transformations to:
| name | age |
|---|---|
| Alice | 22 |
| Bob | 28 |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("Transformations").getOrCreate()
data = [("Alice", 22), ("Bob", 28)]
df = spark.createDataFrame(data, ["name", "age"])
# Chain narrow transformations (filter and select)
transformed_df = df.filter(col("age") > 25).select("name")
# Trigger action to output result
transformed_df.show()
Expected Output
+----+
|name|
+----+
| Bob|
+----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("TransformScala").getOrCreate()
import spark.implicits._
val df = Seq(("Alice", 22), ("Bob", 28)).toDF("name", "age")
val transformed = df.filter($"age" > 25).select("name")
transformed.show()
Expected Output
+----+
|name|
+----+
| Bob|
+----+
SQL Implementation
SELECT name
FROM users
WHERE age > 25;
Expected Output
Executing this SQL query returns:
| name |
|---|
| Bob |
Common Mistakes
- Expecting Immediate Execution: Writing multiple lines of transformation code and expecting Spark to run them right away. Spark will only execute the code when you call an action like
show()orcount().
Best Practices
- Minimize Wide Transformations: Minimize wide transformations (like joins or orderBys) in your pipelines, as they require shuffling data over the network, which is expensive.
Interview Perspective
What is the difference between a narrow and a wide transformation?
In a narrow transformation (e.g. filter(), select()), all data required to process a partition resides on a single node. In a wide transformation (e.g. groupBy(), orderBy()), data must be gathered from multiple nodes across the network, triggering a shuffle.