Introduction to DataFrames
Overview
A Spark DataFrame is a distributed collection of data organized into named columns. Conceptually equivalent to a table in a relational database or a data frame in R/Python, Spark DataFrames are highly optimized and designed to scale to petabytes of data.
What You Will Learn
In this lesson, you will learn:
- DataFrame Definition: What makes a Spark DataFrame distributed and immutable.
- Structured Schema: How columns, rows, and schema datatypes organize data.
- Memory Layout: The optimization advantages of using Spark DataFrames over RDDs.
Detailed Concept Explanation
Spark DataFrames are built on top of the low-level RDD API. However, unlike RDDs, DataFrames have a structured schema. This means Spark knows the column names and data types of your dataset at compile time.
This structure allows the Spark engine to perform advanced optimizations:
- Catalyst Optimizer: Builds logical plans and pushes down filters.
- Tungsten Engine: Optimizes physical execution memory layout and CPU code generation.
DataFrames are immutable (cannot be changed once created) and evaluated lazily (transformations are only executed when an action is triggered).
Code Examples
Input Dataset Preview
Below is the customer dataset we will represent as a Spark DataFrame:
| customerId | name | country |
|---|---|---|
| 101 | Alice | USA |
| 102 | Bob | Canada |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("IntroDF").getOrCreate()
data = [(101, "Alice", "USA"), (102, "Bob", "Canada")]
df = spark.createDataFrame(data, ["customerId", "name", "country"])
df.show()
Expected Output
+----------+-----+-------+
|customerId| name|country|
+----------+-----+-------+
| 101|Alice| USA|
| 102| Bob| Canada|
+----------+-----+-------+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("IntroDFScala").getOrCreate()
import spark.implicits._
val df = Seq(
(101, "Alice", "USA"),
(102, "Bob", "Canada")
).toDF("customerId", "name", "country")
df.show()
Expected Output
+----------+-----+-------+
|customerId| name|country|
+----------+-----+-------+
| 101|Alice| USA|
| 102| Bob| Canada|
+----------+-----+-------+
SQL Implementation
SELECT customerId, name, country
FROM customers;
Expected Output
Executing this SQL query returns:
| customerId | name | country |
|---|---|---|
| 101 | Alice | USA |
| 102 | Bob | Canada |
Common Mistakes
- Mutating DataFrames In-Place: Trying to change values inside an existing DataFrame. Since they are immutable, any transformation returns a new DataFrame.
Performance Notes
- Schema Optimization: Explaining schema details explicitly bypasses Spark scanning physical files, saving network read costs.
Best Practices
- Never Use Loops: Avoid looping through DataFrame rows using standard Python loops. Use built-in column functions to let Spark parallelize execution.
Interview Perspective
What is a Spark DataFrame, and how does it relate to RDDs?
A DataFrame is a distributed collection of rows organized into columns. Under the hood, DataFrames are built on RDDs but carry an associated schema. This schema allows Spark to use the Catalyst Optimizer to optimize execution plans, which is not possible with low-level RDDs.