Reading Data

Overview

Learn how to read data from different file formats into Spark DataFrames using the DataFrameReader API.

What You Will Learn

Detailed Concept Explanation

Spark provides a unified interface for loading datasets called DataFrameReader, accessed via spark.read. The basic syntax is: spark.read.format("format").option("key", "value").load("path")

Common formats include:

• csv: Text-based comma separated format.
• json: Structured text format.
• parquet & orc: Column-oriented, highly optimized binary storage.

Code Examples

Input Dataset Preview

Let's assume there is a local CSV file at users.csv with these records:

id,name,role
1,Alice,Admin
2,Bob,User

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ReadData").getOrCreate()
# Load CSV with headers
df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("users.csv")
df.show()

Expected Output

+---+-----+----+
| id| name|role|
+---+-----+----+
|  1|Alice|Admin|
|  2|  Bob|User|
+---+-----+----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ReadDFScala").getOrCreate()
val df = spark.read.format("csv")
  .option("header", "true")
  .option("inferSchema", "true")
  .load("users.csv")
df.show()

Expected Output

+---+-----+----+
| id| name|role|
+---+-----+----+
|  1|Alice|Admin|
|  2|  Bob|User|
+---+-----+----+

SQL Implementation

-- Read directly from a CSV file using SQL path syntax
SELECT id, name, role 
FROM csv.`users.csv`;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Forgetting header option in CSV: If omitted, Spark treats the first row (the column header names) as actual data, corrupting the schema mapping.

Best Practices

• Prefer Parquet: When processing huge files, prefer Parquet over CSV or JSON. Parquet is compressed, binary, column-oriented, and substantially faster.

Interview Perspective

Interactive Challenges

Related Topics

• Creating DataFrames
• Writing Data
• Schema Inference