Introduction to RDD
Overview
Resilient Distributed Datasets (RDDs) are the foundational data abstraction of Apache Spark. Learn the core properties of RDDs and why they provide fault tolerance across clusters.
What You Will Learn
In this lesson, you will learn:
- RDD Definition: Resilient (fault-tolerant), Distributed (partitioned), Dataset (records).
- Immutability: Why RDDs cannot be changed once created.
- Lineage Graphs: How RDDs recover from worker node failures without replicating data.
Detailed Concept Explanation
An RDD represents an immutable, partitioned collection of records that can be operated on in parallel.
- Resilient: Fault-tolerant. If a worker node crashes and a partition is lost, Spark automatically rebuilds that partition using the RDD's Lineage Graph (the history of transformations used to build it).
- Distributed: The data is split into partitions and distributed across different nodes in the cluster.
- Dataset: A collection of objects.
Code Examples
Input Dataset Preview
Below is the list of integers we will represent as an RDD:
| value |
|---|
| 10 |
| 20 |
| 30 |
| 40 |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("RDDBasics").getOrCreate()
sc = spark.sparkContext
# Create an RDD from a local list
rdd = sc.parallelize([10, 20, 30, 40])
print("Total Records:", rdd.count())
Expected Output
Total Records: 4
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("RDDScala").getOrCreate()
val sc = spark.sparkContext
val rdd = sc.parallelize(Seq(10, 20, 30, 40))
println(s"Total Records: ${rdd.count()}")
Expected Output
Total Records: 4
SQL Perspective
SQL Query Support:
RDDs are low-level, unstructured collections of objects and do not support Spark SQL directly. To run SQL queries, you must first convert the RDD to a DataFrame and register it as a view:
# Convert RDD to DataFrame
df = rdd.map(lambda x: (x,)).toDF(["value"])
df.createOrReplaceTempView("numbers")
spark.sql("SELECT SUM(value) FROM numbers").show()
Common Mistakes
- Mutating RDDs: Expecting an RDD to update in-place. Because RDDs are immutable, any transformation returns a new RDD.
Best Practices
- Use DataFrames: Only fall back to the RDD API when you need low-level, type-safe custom object controls. Otherwise, use DataFrames for Catalyst optimizations.
Interview Perspective
How does Spark guarantee fault tolerance using RDDs?
Spark does not replicate data across nodes to guarantee fault tolerance. Instead, it tracks the history of transformations (the Lineage Graph) used to create each RDD. If a partition is lost due to a node crash, Spark uses the lineage to recompute only that lost partition on another worker.