Writing Data
Overview
Learn how to save your Spark DataFrames to files or databases using the DataFrameWriter API and its supported save modes.
What You Will Learn
In this lesson, you will learn:
- DataFrameWriter: The saving interface structure (
df.write). - Save Modes: Append, Overwrite, Ignore, and ErrorIfExists.
- Target Formats: Saving datasets as CSV, JSON, and Parquet.
Detailed Concept Explanation
Similar to reading, Spark uses a unified interface for writing data accessed via df.write. The basic syntax is:
df.write.format("format").mode("save_mode").save("path")
Save Modes
overwrite: Overwrites any existing data in the target directory.append: Appends the DataFrame rows to the existing files in the directory.ignore: If files exist, Spark quietly skips the write operation without errors.errorifexists(default): Throws a runtime exception if the target path is not empty.
Code Examples
Input Dataset Preview
Below is the output DataFrame we want to save to disk:
| name | score |
|---|---|
| Alice | 95 |
| Bob | 88 |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("WriteData").getOrCreate()
data = [("Alice", 95), ("Bob", 88)]
df = spark.createDataFrame(data, ["name", "score"])
# Write to Parquet with overwrite mode
df.write.format("parquet") \
.mode("overwrite") \
.save("output_scores")
Expected Output
This creates a directory named output_scores containing compressed Parquet partition files (part-00000-*.parquet).
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("WriteScala").getOrCreate()
val df = Seq(("Alice", 95), ("Bob", 88)).toDF("name", "score")
df.write.format("parquet")
.mode("overwrite")
.save("output_scores")
Expected Output
Saves the sequence as Parquet files under the target directory.
SQL Implementation
-- Creating tables and writing directly in SQL
CREATE TABLE scores_parquet
USING parquet
OPTIONS (path 'output_scores_sql')
AS SELECT 'Alice' AS name, 95 AS score;
Expected Output
Creates a Spark SQL table backed by Parquet files on disk.
Common Mistakes
- Writing to a File, Not a Directory: Spark always writes outputs as a directory containing multiple partitioned files (one per partition), rather than a single flat file.
Best Practices
- Use PartitionBy: When saving large datasets, use
.partitionBy("column")to organize output directories by keys, speeding up downstream filters.
Interview Perspective
What are the main SaveModes supported by Spark's DataFrameWriter?
Spark supports four main save modes: overwrite (replaces files), append (adds files), ignore (silently skips if target exists), and errorifexists (default behavior, throws exception if directory exists).