beginner

Reading Data

By DataPlayArena Technical Review BoardUpdated 2026-07-088 min readRuntime Verified

Overview

Learn how to read data from different file formats into Spark DataFrames using the DataFrameReader API.

What You Will Learn

In this lesson, you will learn:
  • DataFrameReader: The interface structure (spark.read).
  • Format Options: Loading CSV, JSON, Parquet, and ORC.
  • Configuration Options: Specifying headers, separators, and paths.

Detailed Concept Explanation

Spark provides a unified interface for loading datasets called DataFrameReader, accessed via spark.read. The basic syntax is: spark.read.format("format").option("key", "value").load("path")

Common formats include:

  • csv: Text-based comma separated format.
  • json: Structured text format.
  • parquet & orc: Column-oriented, highly optimized binary storage.

Code Examples

Input Dataset Preview

Let's assume there is a local CSV file at users.csv with these records:

text
id,name,role
1,Alice,Admin
2,Bob,User

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ReadData").getOrCreate()
# Load CSV with headers
df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("users.csv")
df.show()

Expected Output

text
+---+-----+----+
| id| name|role|
+---+-----+----+
|  1|Alice|Admin|
|  2|  Bob|User|
+---+-----+----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
read.format(csv)
option(header
true)
load(path)
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ReadDFScala").getOrCreate()
val df = spark.read.format("csv")
  .option("header", "true")
  .option("inferSchema", "true")
  .load("users.csv")
df.show()

Expected Output

text
+---+-----+----+
| id| name|role|
+---+-----+----+
|  1|Alice|Admin|
|  2|  Bob|User|
+---+-----+----+

SQL Implementation

sql
-- Read directly from a CSV file using SQL path syntax
SELECT id, name, role 
FROM csv.`users.csv`;

Expected Output

Executing this SQL query returns:

idnamerole
1AliceAdmin
2BobUser

Common Mistakes

  • Forgetting header option in CSV: If omitted, Spark treats the first row (the column header names) as actual data, corrupting the schema mapping.

Best Practices

  • Prefer Parquet: When processing huge files, prefer Parquet over CSV or JSON. Parquet is compressed, binary, column-oriented, and substantially faster.

Interview Perspective

What is the general syntax pattern for loading files in Spark?

The standard pattern is spark.read.format("format_name").option("key", "value").load("path"). For common formats, Spark also provides shortcut methods like spark.read.csv("path") or spark.read.parquet("path").


Interactive Challenges

Challenge 1: Loading JSON Files (Beginner)

Write a PySpark snippet to read a JSON file located at 'data.json' using the format shortcut method.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.