beginner

DAG

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how Spark compiles your transformations into a Directed Acyclic Graph (DAG) to model execution pipelines and recover from node failures.

What You Will Learn

In this lesson, you will learn:
  • DAG Definition: Directed (ordered steps), Acyclic (no loops), Graph (vertices/edges).
  • Lineage: How DAG tracks dependencies.
  • Visualizing Plans: Inspecting DAGs using the .explain() method.

Detailed Concept Explanation

A DAG is the logical representation of your Spark query execution plan.

  • Directed: Each step in the pipeline points to the next step.
  • Acyclic: There are no loops or circular loops in the graph.
  • Graph: Made of vertices (transformations) and edges (dependencies/shuffles).

When an action is called, the DAG Scheduler divides the graph into execution stages based on shuffle boundaries, which are then split into parallel tasks sent to executors.


Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DAGConcept").getOrCreate()

data = [("IT", 5000), ("HR", 4000), ("IT", 6000)]
df = spark.createDataFrame(data, ["dept", "salary"])

# DAG contains: Filter -> GroupBy -> Sum
result = df.filter(df.salary > 4000).groupBy("dept").sum("salary")

# Print the physical execution plan compiled from the DAG
result.explain()

Expected Output

text
== Physical Plan ==
AdaptiveSparkPlan isAdaptive=true
+- HashAggregate(keys=[dept#0], functions=[sum(salary#1L)])
   +- Exchange hashpartitioning(dept#0, 200)
      +- HashAggregate(keys=[dept#0], functions=[partial_sum(salary#1L)])
         +- Filter (isnotnull(salary#1L) AND (salary#1L > 4000))
            +- Scan ExistingRDD[dept#0,salary#1L]

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDataFrame
filter(salary > 4000)
groupBy(dept).sum()
explain()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("DAGScala").getOrCreate()
import spark.implicits._

val df = Seq(("IT", 5000), ("HR", 4000), ("IT", 6000)).toDF("dept", "salary")
val result = df.filter($"salary" > 4000).groupBy("dept").sum("salary")

result.explain()

Interview Perspective

What is a DAG in Apache Spark?

A Directed Acyclic Graph (DAG) is the logical execution plan constructed by Spark's DAG Scheduler when an action is triggered. It maps out the sequence of transformations (vertices) and the dependencies between them (edges), allowing Spark to optimize stages and recompute lost data partitions if worker nodes fail.


Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.