intermediate

RDD Transformations

By DataPlayArena Technical Review BoardUpdated 2026-07-0810 min readRuntime Verified

Overview

Learn how to transform RDD datasets using mapping, filtering, and flattening operations.

What You Will Learn

In this lesson, you will learn:
  • Map RDD: Applying one-to-one record conversions (map()).
  • Filter RDD: Dropping rows based on conditions (filter()).
  • FlatMap RDD: Splitting and flattening records (flatMap()).

Detailed Concept Explanation

RDD transformations are lazy operations that compile execution paths without executing them. Common transformations:

  • map(func): Passes each record through a function, returning a new RDD with one output record for each input record (1-to-1).
  • filter(func): Evaluates a boolean condition for each record, keeping only those that return True.
  • flatMap(func): Similar to map(), but each input record can map to 0 or more output records. It flattens the output array.

Code Examples

Input Dataset Preview

Below is the text sentences list:

sentence
Hello world
Spark rules

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TransformRDD").getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize(["Hello world", "Spark rules"])

# flatMap splits sentences into words, map capitalizes them
words_rdd = rdd.flatMap(lambda line: line.split(" ")) \
               .map(lambda word: word.upper())
print(words_rdd.collect())

Expected Output

text
['HELLO', 'WORLD', 'SPARK', 'RULES']

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkContext.parallelize
flatMap(split)
map(upper)
collect()
print()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("TransformRDDScala").getOrCreate()
val sc = spark.sparkContext

val rdd = sc.parallelize(Seq("Hello world", "Spark rules"))
val words = rdd.flatMap(line => line.split(" ")).map(word => word.toUpperCase)
println(words.collect().mkString(", "))

Expected Output

text
HELLO, WORLD, SPARK, RULES

SQL Perspective

SQL Query Support:

To run SQL queries, convert the transformed RDD into a DataFrame:

python
df = words_rdd.map(lambda x: (x,)).toDF(["word"])
df.createOrReplaceTempView("words_table")
spark.sql("SELECT * FROM words_table").show()

Common Mistakes

  • Confusing map and flatMap: Using map() to split sentences into arrays. This will return an RDD of lists (e.g. [['Hello', 'world'], ['Spark', 'rules']]) rather than a flattened RDD of strings. Use flatMap() to flatten lists automatically.

Best Practices

  • Avoid Heavy Closures: Avoid passing heavy local variables inside map functions; use broadcast variables instead.

Interview Perspective

What is the difference between map() and flatMap() in the RDD API?

map() applies a function to each record, returning exactly one output record for each input. flatMap() applies a function that returns an array/collection of items and flattens that collection, mapping each input record to 0 or more output records.


Interactive Challenges

Challenge 1: Filter even values (Beginner)

Write a PySpark RDD filter expression to keep only even numbers from an integer RDD.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.