Partitions

Overview

Partitions are logical chunks of your distributed data. Learn how they map to tasks and why partition count is critical for performance tuning.

What You Will Learn

Detailed Concept Explanation

A Partition is a logical slice of a distributed dataset stored across the worker nodes. Spark does not load the entire dataset into a single JVM; instead, it splits the rows into partitions to allow parallel processing.

Why Partition Size Matters

• Too many small partitions: Creates excessive metadata management overhead on the driver node and schedules too many tiny, short-lived tasks, wasting CPU time.
• Too few large partitions: Reduces parallelism (leaving executor cores idle) and risks Out Of Memory (OOM) errors if a partition exceeds executor RAM.
• Target Size: A good rule of thumb is to aim for 100MB to 200MB per partition when uncompressed in memory.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PartitionsConcept").getOrCreate()

# Create DataFrame
df = spark.createDataFrame([("A", 1), ("B", 2)], ["key", "val"])

# Check default partition count
print("Default partitions:", df.rdd.getNumPartitions())

# Re-partition to 8 slices
repartitioned_df = df.repartition(8)
print("Updated partitions:", repartitioned_df.rdd.getNumPartitions())

Expected Output

Default partitions: 1
Updated partitions: 8

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("PartitionsScala").getOrCreate()
import spark.implicits._

val df = Seq(("A", 1), ("B", 2)).toDF("key", "val")
println(s"Default partitions: ${df.rdd.getNumPartitions}")

val repartitioned = df.repartition(8)
println(s"Updated partitions: ${repartitioned.rdd.getNumPartitions}")

Interview Perspective

Related Topics

• Repartition vs Coalesce
• Tasks
• Small File Problem