RDD Transformations

Overview

Learn how to transform RDD datasets using mapping, filtering, and flattening operations.

What You Will Learn

Detailed Concept Explanation

RDD transformations are lazy operations that compile execution paths without executing them. Common transformations:

• map(func): Passes each record through a function, returning a new RDD with one output record for each input record (1-to-1).
• filter(func): Evaluates a boolean condition for each record, keeping only those that return True.
• flatMap(func): Similar to map(), but each input record can map to 0 or more output records. It flattens the output array.

Code Examples

Input Dataset Preview

Below is the text sentences list:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TransformRDD").getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize(["Hello world", "Spark rules"])

# flatMap splits sentences into words, map capitalizes them
words_rdd = rdd.flatMap(lambda line: line.split(" ")) \
               .map(lambda word: word.upper())
print(words_rdd.collect())

Expected Output

['HELLO', 'WORLD', 'SPARK', 'RULES']

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("TransformRDDScala").getOrCreate()
val sc = spark.sparkContext

val rdd = sc.parallelize(Seq("Hello world", "Spark rules"))
val words = rdd.flatMap(line => line.split(" ")).map(word => word.toUpperCase)
println(words.collect().mkString(", "))

Expected Output

HELLO, WORLD, SPARK, RULES

SQL Perspective

df = words_rdd.map(lambda x: (x,)).toDF(["word"])
df.createOrReplaceTempView("words_table")
spark.sql("SELECT * FROM words_table").show()

Common Mistakes

• Confusing map and flatMap: Using map() to split sentences into arrays. This will return an RDD of lists (e.g. [['Hello', 'world'], ['Spark', 'rules']]) rather than a flattened RDD of strings. Use flatMap() to flatten lists automatically.

Best Practices

• Avoid Heavy Closures: Avoid passing heavy local variables inside map functions; use broadcast variables instead.

Interview Perspective

Interactive Challenges

Related Topics

• RDD Actions
• Pair RDDs
• Introduction to RDD