RDD Transformations
Overview
Learn how to transform RDD datasets using mapping, filtering, and flattening operations.
What You Will Learn
In this lesson, you will learn:
- Map RDD: Applying one-to-one record conversions (
map()). - Filter RDD: Dropping rows based on conditions (
filter()). - FlatMap RDD: Splitting and flattening records (
flatMap()).
Detailed Concept Explanation
RDD transformations are lazy operations that compile execution paths without executing them. Common transformations:
map(func): Passes each record through a function, returning a new RDD with one output record for each input record (1-to-1).filter(func): Evaluates a boolean condition for each record, keeping only those that returnTrue.flatMap(func): Similar tomap(), but each input record can map to 0 or more output records. It flattens the output array.
Code Examples
Input Dataset Preview
Below is the text sentences list:
| sentence |
|---|
| Hello world |
| Spark rules |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("TransformRDD").getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize(["Hello world", "Spark rules"])
# flatMap splits sentences into words, map capitalizes them
words_rdd = rdd.flatMap(lambda line: line.split(" ")) \
.map(lambda word: word.upper())
print(words_rdd.collect())
Expected Output
['HELLO', 'WORLD', 'SPARK', 'RULES']
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("TransformRDDScala").getOrCreate()
val sc = spark.sparkContext
val rdd = sc.parallelize(Seq("Hello world", "Spark rules"))
val words = rdd.flatMap(line => line.split(" ")).map(word => word.toUpperCase)
println(words.collect().mkString(", "))
Expected Output
HELLO, WORLD, SPARK, RULES
SQL Perspective
SQL Query Support:
To run SQL queries, convert the transformed RDD into a DataFrame:
df = words_rdd.map(lambda x: (x,)).toDF(["word"])
df.createOrReplaceTempView("words_table")
spark.sql("SELECT * FROM words_table").show()
Common Mistakes
- Confusing map and flatMap: Using
map()to split sentences into arrays. This will return an RDD of lists (e.g.[['Hello', 'world'], ['Spark', 'rules']]) rather than a flattened RDD of strings. UseflatMap()to flatten lists automatically.
Best Practices
- Avoid Heavy Closures: Avoid passing heavy local variables inside map functions; use broadcast variables instead.
Interview Perspective
What is the difference between map() and flatMap() in the RDD API?
map() applies a function to each record, returning exactly one output record for each input. flatMap() applies a function that returns an array/collection of items and flattens that collection, mapping each input record to 0 or more output records.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.