intermediate

Narrow vs Wide Transformations

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn to distinguish between Narrow Transformations (which run locally on partitions) and Wide Transformations (which force network shuffles).

What You Will Learn

In this lesson, you will learn:
  • Narrow Transformations: Map-side transformations with no network dependencies.
  • Wide Transformations: Operations requiring network shuffles.
  • Performance Impacts: Why minimizing shuffles improves query speed.

Detailed Concept Explanation

Spark transformations are categorized into narrow or wide based on their partition dependencies.

Narrow Transformations

  • Behavior: Each input partition maps to at most one output partition. The data does not leave the executor node.
  • Examples: map(), filter(), flatMap(), union(), select().
  • Performance: Very fast, as operations run in parallel across partitions with zero network I/O.

Wide Transformations

  • Behavior: Multiple input partitions map to multiple output partitions. Records with matching keys are routed across nodes. This network reorganization is called a Shuffle.
  • Examples: groupBy(), join(), repartition(), distinct().
  • Performance: Expensive due to network latency, disk serialization, and synchronization barriers.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("NarrowWide").getOrCreate()

df = spark.createDataFrame([("A", 1), ("B", 2)], ["key", "val"])

# Narrow: filter keeps data on same partition
narrow_df = df.filter(df.val > 1)

# Wide: groupByKey shuffles data across network
wide_df = df.groupBy("key").count()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDataFrame
filter(val > 1) [Narrow]
groupBy(key).count() [Wide - Shuffle]

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("NarrowWideScala").getOrCreate()
import spark.implicits._

val df = Seq(("A", 1), ("B", 2)).toDF("key", "val")

val narrow = df.filter($"val" > 1)
val wide = df.groupBy("key").count()

Interview Perspective

What is the difference between a narrow and a wide dependency in Spark?

A narrow dependency means each partition of the child RDD/DataFrame depends on at most one partition of the parent, requiring no network shuffles (e.g. map, filter). A wide dependency means child partitions depend on multiple parent partitions, forcing data to be shuffled across nodes over the network (e.g. groupBy, join).


Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.