Broadcast Variables

Overview

Learn how to use Broadcast Variables to share large, read-only lookup datasets with all executor worker nodes, saving network bandwidth.

What You Will Learn

Detailed Concept Explanation

By default, when you reference a local variable inside a map function, Spark ships a copy of that variable to every task over the network. If a job has 1,000 tasks, the variable is serialized and sent 1,000 times.

A Broadcast Variable solves this by sending the variable exactly once per executor machine (node) rather than once per task. Tasks on that executor then access the shared value locally from the executor's cache.

Code Examples

Input Dataset Preview

Below is the zip codes lookup dictionary:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Broadcast").getOrCreate()
sc = spark.sparkContext

# Define lookup dictionary
zip_map = {"10001": "New York", "90001": "Los Angeles"}

# Create Broadcast Variable
broadcast_zips = sc.broadcast(zip_map)

# Map zip codes to city names
rdd = sc.parallelize(["10001", "90001", "10001"])
cities_rdd = rdd.map(lambda zip_code: broadcast_zips.value.get(zip_code, "Unknown"))
print(cities_rdd.collect())

Expected Output

['New York', 'Los Angeles', 'New York']

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("BroadcastScala").getOrCreate()
val sc = spark.sparkContext

val zipMap = Map("10001" -> "New York", "90001" -> "Los Angeles")
val broadcastZips = sc.broadcast(zipMap)

val rdd = sc.parallelize(Seq("10001", "90001", "10001"))
val cities = rdd.map(zipCode => broadcastZips.value.getOrElse(zipCode, "Unknown"))
println(cities.collect().mkString(", "))

Expected Output

New York, Los Angeles, New York

Common Mistakes

• Modifying Broadcast Data: Attempting to update or write to a broadcast variable inside worker tasks. Broadcast variables are read-only; any updates will cause inconsistent data states across the cluster.

Best Practices

• Keep it Read-Only: Use broadcast variables only for read-only lookups (like lookup tables or configurations).
• Use value property: Access the broadcasted data inside your tasks using the .value property.

Interview Perspective

Interactive Challenges

Related Topics

• Accumulators
• Broadcast Join
• RDD Actions