CSV

Overview

CSV (Comma-Separated Values) is a common text-based file format. Learn how to read and write CSV files in Spark, configure separators, and handle headers.

What You Will Learn

Detailed Concept Explanation

CSV is a plain-text format where each line represents a row and columns are separated by a delimiter (default is a comma ,).

Because CSV is a flat text format, it does not store schema metadata. When reading CSVs, Spark must either guess the column types (inferSchema) or consume an explicit StructType schema.

Key reader options:

• header: Set true if the first row contains column names.
• sep or delimiter: Configures column separators (e.g. \t for tab, | for pipe).
• multiLine: Set true if rows span multiple lines.

Code Examples

Input Dataset Preview

Below is the users file users.csv:

id,name,city
1,Alice,London
2,Bob,New York

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CSVRead").getOrCreate()

# Read CSV with headers and commas
df = spark.read.format("csv") \
    .option("header", "true") \
    .option("sep", ",") \
    .load("users.csv")
df.show()

Expected Output

+---+-----+--------+
| id| name|    city|
+---+-----+--------+
|  1|Alice|  London|
|  2|  Bob|New York|
+---+-----+--------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("CSVScala").getOrCreate()
val df = spark.read.format("csv")
  .option("header", "true")
  .option("sep", ",")
  .load("users.csv")
df.show()

Expected Output

+---+-----+--------+
| id| name|    city|
+---+-----+--------+
|  1|Alice|  London|
|  2|  Bob|New York|
+---+-----+--------+

SQL Implementation

SELECT id, name, city 
FROM csv.`users.csv`;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Forgetting multiLine for Line Breaks: Attempting to read CSV files where columns contain raw line breaks without setting multiLine to true. This causes Spark to break rows incorrectly, corrupting the dataset.

Best Practices

• Explicit Delimiters: Always specify the sep option explicitly, even for commas, to ensure compatibility.

Interview Perspective

Interactive Challenges

Related Topics

• JSON
• Reading Data
• Parquet