beginner
RDD vs DataFrame
10 min readLast updated: 2026-07-08
Overview
Learn the differences between low-level RDDs and structured DataFrames, and understand when to use each API.
What You Will Learn
In this lesson, you will learn:
- Core Differences: Unstructured objects vs structured relational tables.
- Optimization: Why DataFrames run faster due to Catalyst optimizations.
- API Conversion: Converting RDDs to DataFrames and vice versa.
Detailed Concept Explanation
Spark provides two primary developer APIs: the low-level RDD API and the structured DataFrame API.
| Feature | RDD API | DataFrame API |
|---|---|---|
| Structure | Unstructured (raw Java/Python objects) | Structured (rows organized into columns) |
| Optimization | None (user writes execution logic) | Automatic (Catalyst Optimizer & Tungsten) |
| SQL Support | No | Yes |
| Type Safety | Type-safe (Scala/Java compile-time check) | Less type-safe (runtime string resolution) |
Code Examples
Input Dataset Preview
Below is the employee dataset:
| name | dept |
|---|---|
| Alice | HR |
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("RDDvsDF").getOrCreate()
sc = spark.sparkContext
# RDD representation
rdd = sc.parallelize([("Alice", "HR")])
# Convert RDD to DataFrame
df = rdd.toDF(["name", "dept"])
df.show()
# Convert DataFrame back to RDD
back_to_rdd = df.rdd
print("RDD Row count:", back_to_rdd.count())
Expected Output
text
+-----+----+
| name|dept|
+-----+----+
|Alice| HR|
+-----+----+
RDD Row count: 1
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkContext.parallelize
toDF([name
dept])
show()
df.rdd
rdd.count()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("RDDvsDFScala").getOrCreate()
val sc = spark.sparkContext
import spark.implicits._
val rdd = sc.parallelize(Seq(("Alice", "HR")))
// Convert RDD to DataFrame
val df = rdd.toDF("name", "dept")
df.show()
// Convert DataFrame back to RDD
val backToRDD = df.rdd
println(s"RDD Row count: ${backToRDD.count()}")
Expected Output
text
+-----+----+
| name|dept|
+-----+----+
|Alice| HR|
+-----+----+
RDD Row count: 1
Common Mistakes
- Unnecessary Conversions: Frequently converting data back and forth between RDDs and DataFrames. This breaks execution plans, invalidates Catalyst optimizations, and adds serialization overhead. Keep data inside DataFrames.
Best Practices
- Prefer DataFrames: Always prefer DataFrames for standard data tasks. Only use RDDs when you need to write low-level, type-safe custom object code.
Interview Perspective
When would you choose to use the RDD API instead of the DataFrame API?
Choose the RDD API only when:
- You need to write custom low-level object code that cannot be represented in relational rows.
- You are maintaining legacy Spark codebases built before Spark 2.0.
- You need to manage hardware memory layouts explicitly. For all other cases, use DataFrames to take advantage of Catalyst optimizations.