Parquet

Overview

Parquet is the standard binary columnar storage format for Big Data. Learn how it compresses datasets, stores metadata, and optimizes reading speeds in Spark.

What You Will Learn

Detailed Concept Explanation

Unlike CSV or JSON, which store data in rows, Apache Parquet stores data in columns.

Why Columnar Storage?

• Query speed: If a table has 100 columns but your query only selects 2 (SELECT age, country), Spark only reads the bytes for those 2 columns, skipping the other 98. This is called Column Pruning.
• Excellent compression: Because a column contains values of the same data type, algorithms can compress it far more efficiently than mixed row-based text. By default, Spark compresses Parquet using Snappy.
• Self-describing: Parquet files contain schema details and summary statistics (like min/max values) in their footer, allowing Spark to skip reading entire row groups if they do not match filter conditions (called Predicate Pushdown).

Code Examples

Input Dataset Preview

Below is the log records dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ParquetTest").getOrCreate()
data = [(1, "INFO"), (2, "WARN")]
df = spark.createDataFrame(data, ["id", "level"])

# Write to Parquet (Snappy compression is default)
df.write.format("parquet").mode("overwrite").save("logs.parquet")

# Read Parquet
read_df = spark.read.format("parquet").load("logs.parquet")
read_df.show()

Expected Output

+---+-----+
| id|level|
+---+-----+
|  1| INFO|
|  2| WARN|
+---+-----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ParquetScala").getOrCreate()
import spark.implicits._

val df = Seq((1, "INFO"), (2, "WARN")).toDF("id", "level")
df.write.format("parquet").mode("overwrite").save("logs.parquet")

val readDF = spark.read.format("parquet").load("logs.parquet")
readDF.show()

Expected Output

+---+-----+
| id|level|
+---+-----+
|  1| INFO|
|  2| WARN|
+---+-----+

SQL Implementation

-- Query a local Parquet directory directly in SQL
SELECT * 
FROM parquet.`logs.parquet`
WHERE level = 'WARN';

Expected Output

Executing this SQL query returns:

Common Mistakes

• Checking Schema via Pre-scan: Assuming reading Parquet requires setting inferSchema=True. Parquet files store their schema natively in their metadata, making pre-scans unnecessary.

Best Practices

• Dynamic Partition Pruning: Partition your Parquet data on disk by directories (e.g. year=2026/month=07) using .partitionBy() to let Spark skip reading folders that do not match filters.

Interview Perspective

Interactive Challenges

Related Topics

• ORC
• Avro
• Reading Data