Block Manager

Overview

Learn how Spark's Block Manager functions as a local key-value store to manage data blocks (partitions, broadcasts, and cache layers) on every node.

What You Will Learn

Detailed Concept Explanation

The Block Manager is a key-value store that runs on the Driver and every Executor. It manages the storage and retrieval of data blocks:

• Cached Partitions: Saves partitions in memory or on disk.
• Shuffle Blocks: Manages shuffle write files and read fetches.
• Broadcast Variables: Caches broadcast data locally.

Driver Master Coordinator

The Block Manager on the Driver acts as the Block Manager Master. Executor block managers report their block metadata locations to the master, allowing executors to look up and fetch blocks from other nodes over the network.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("BlockManager").getOrCreate()

# Cache DataFrame (Block Manager saves partitions in memory)
df = spark.range(1, 1000).cache()

# Trigger Action to store blocks
print("Count:", df.count())

# Clean up blocks from memory
df.unpersist()

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("BlockManagerScala").getOrCreate()

val df = spark.range(1, 1000).cache()
println(s"Count: ${df.count()}")

df.unpersist()

Related Topics

• Memory Management
• Cache vs Persist
• Broadcast Variables