CSV
Overview
CSV (Comma-Separated Values) is a common text-based file format. Learn how to read and write CSV files in Spark, configure separators, and handle headers.
What You Will Learn
In this lesson, you will learn:
- CSV Reader: Loading CSV files with specific configurations.
- Separator Options: Setting custom delimiters (e.g. tabs or pipes).
- Schema Options: Toggling headers and escaping quotes.
Detailed Concept Explanation
CSV is a plain-text format where each line represents a row and columns are separated by a delimiter (default is a comma ,).
Because CSV is a flat text format, it does not store schema metadata. When reading CSVs, Spark must either guess the column types (inferSchema) or consume an explicit StructType schema.
Key reader options:
header: Settrueif the first row contains column names.sepordelimiter: Configures column separators (e.g.\tfor tab,|for pipe).multiLine: Settrueif rows span multiple lines.
Code Examples
Input Dataset Preview
Below is the users file users.csv:
id,name,city
1,Alice,London
2,Bob,New York
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CSVRead").getOrCreate()
# Read CSV with headers and commas
df = spark.read.format("csv") \
.option("header", "true") \
.option("sep", ",") \
.load("users.csv")
df.show()
Expected Output
+---+-----+--------+
| id| name| city|
+---+-----+--------+
| 1|Alice| London|
| 2| Bob|New York|
+---+-----+--------+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("CSVScala").getOrCreate()
val df = spark.read.format("csv")
.option("header", "true")
.option("sep", ",")
.load("users.csv")
df.show()
Expected Output
+---+-----+--------+
| id| name| city|
+---+-----+--------+
| 1|Alice| London|
| 2| Bob|New York|
+---+-----+--------+
SQL Implementation
SELECT id, name, city
FROM csv.`users.csv`;
Expected Output
Executing this SQL query returns:
| id | name | city |
|---|---|---|
| 1 | Alice | London |
| 2 | Bob | New York |
Common Mistakes
- Forgetting multiLine for Line Breaks: Attempting to read CSV files where columns contain raw line breaks without setting
multiLinetotrue. This causes Spark to break rows incorrectly, corrupting the dataset.
Best Practices
- Explicit Delimiters: Always specify the
sepoption explicitly, even for commas, to ensure compatibility.
Interview Perspective
What are the performance limitations of using CSV files in Spark?
CSV is a row-oriented, text-based format. Spark cannot skip rows or columns when reading CSVs; it must read and parse the entire file line-by-line. Additionally, because CSV contains no metadata, Spark must execute expensive schema inference or parse custom schemas.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.