Actions
Overview
Learn about Spark Actions—the operations that trigger the execution of the lazy transformations stored in the DAG.
What You Will Learn
In this lesson, you will learn:
- Action Concept: The execution trigger mechanism.
- Common Actions: Using
show(),count(),collect(), andwrite(). - Memory Limits: Avoiding Driver OOM errors during data collection.
Detailed Concept Explanation
Because Spark uses lazy evaluation, transformations do not execute immediately. Instead, Spark records them in a Directed Acyclic Graph (DAG).
An Action tells Spark to compile and run the DAG. Actions pull data out of the Spark executor system and either:
- Return results to the Driver process (e.g.
.collect(),.count(),.take()). - Write results to external storage (e.g.
.write.save()).
Code Examples
Input Dataset Preview
Below is the dataset we will count and inspect:
| log_id | level |
|---|---|
| 1001 | ERROR |
| 1002 | INFO |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Actions").getOrCreate()
data = [(1001, "ERROR"), (1002, "INFO")]
df = spark.createDataFrame(data, ["log_id", "level"])
# Trigger Actions
record_count = df.count() # Action 1: returns integer
print(f"Total Rows: {record_count}")
df.show() # Action 2: prints table to stdout
Expected Output
Total Rows: 2
+------+-----+
|log_id|level|
+------+-----+
| 1001|ERROR|
| 1002| INFO|
+------+-----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("ActionsScala").getOrCreate()
import spark.implicits._
val df = Seq((1001, "ERROR"), (1002, "INFO")).toDF("log_id", "level")
val cnt = df.count()
println(s"Total Rows: $cnt")
df.show()
Expected Output
Total Rows: 2
+------+-----+
|log_id|level|
+------+-----+
| 1001|ERROR|
| 1002| INFO|
+------+-----+
SQL Implementation
SELECT COUNT(*)
FROM logs;
Expected Output
Executing this SQL query returns:
| count(1) |
|---|
| 2 |
Common Mistakes
- OOM from collect(): Using
.collect()to pull a massive dataset to the driver node. This will crash the driver with an Out Of Memory (OOM) error. Use.take(n)instead.
Best Practices
- Prefer Writing to Disk: In production pipelines, write results directly to disk using
.write.save()rather than bringing data back to the driver.
Interview Perspective
What is the difference between a Transformation and an Action in Spark?
A transformation (e.g. filter(), select()) is lazy and returns a new DataFrame, adding to the logical execution plan without executing it. An action (e.g. count(), collect()) triggers execution, compiling the plan and running tasks across the cluster.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.