Reading Data
Overview
Learn how to read data from different file formats into Spark DataFrames using the DataFrameReader API.
What You Will Learn
In this lesson, you will learn:
- DataFrameReader: The interface structure (
spark.read). - Format Options: Loading CSV, JSON, Parquet, and ORC.
- Configuration Options: Specifying headers, separators, and paths.
Detailed Concept Explanation
Spark provides a unified interface for loading datasets called DataFrameReader, accessed via spark.read. The basic syntax is:
spark.read.format("format").option("key", "value").load("path")
Common formats include:
csv: Text-based comma separated format.json: Structured text format.parquet&orc: Column-oriented, highly optimized binary storage.
Code Examples
Input Dataset Preview
Let's assume there is a local CSV file at users.csv with these records:
id,name,role
1,Alice,Admin
2,Bob,User
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ReadData").getOrCreate()
# Load CSV with headers
df = spark.read.format("csv") \
.option("header", "true") \
.option("inferSchema", "true") \
.load("users.csv")
df.show()
Expected Output
+---+-----+----+
| id| name|role|
+---+-----+----+
| 1|Alice|Admin|
| 2| Bob|User|
+---+-----+----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("ReadDFScala").getOrCreate()
val df = spark.read.format("csv")
.option("header", "true")
.option("inferSchema", "true")
.load("users.csv")
df.show()
Expected Output
+---+-----+----+
| id| name|role|
+---+-----+----+
| 1|Alice|Admin|
| 2| Bob|User|
+---+-----+----+
SQL Implementation
-- Read directly from a CSV file using SQL path syntax
SELECT id, name, role
FROM csv.`users.csv`;
Expected Output
Executing this SQL query returns:
| id | name | role |
|---|---|---|
| 1 | Alice | Admin |
| 2 | Bob | User |
Common Mistakes
- Forgetting header option in CSV: If omitted, Spark treats the first row (the column header names) as actual data, corrupting the schema mapping.
Best Practices
- Prefer Parquet: When processing huge files, prefer Parquet over CSV or JSON. Parquet is compressed, binary, column-oriented, and substantially faster.
Interview Perspective
What is the general syntax pattern for loading files in Spark?
The standard pattern is spark.read.format("format_name").option("key", "value").load("path"). For common formats, Spark also provides shortcut methods like spark.read.csv("path") or spark.read.parquet("path").
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.