Actions

Overview

Learn about Spark Actions—the operations that trigger the execution of the lazy transformations stored in the DAG.

What You Will Learn

Detailed Concept Explanation

Because Spark uses lazy evaluation, transformations do not execute immediately. Instead, Spark records them in a Directed Acyclic Graph (DAG).

An Action tells Spark to compile and run the DAG. Actions pull data out of the Spark executor system and either:

1. Return results to the Driver process (e.g. .collect(), .count(), .take()).
2. Write results to external storage (e.g. .write.save()).

Code Examples

Input Dataset Preview

Below is the dataset we will count and inspect:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Actions").getOrCreate()
data = [(1001, "ERROR"), (1002, "INFO")]
df = spark.createDataFrame(data, ["log_id", "level"])

# Trigger Actions
record_count = df.count() # Action 1: returns integer
print(f"Total Rows: {record_count}")

df.show() # Action 2: prints table to stdout

Expected Output

Total Rows: 2
+------+-----+
|log_id|level|
+------+-----+
|  1001|ERROR|
|  1002| INFO|
+------+-----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ActionsScala").getOrCreate()
import spark.implicits._

val df = Seq((1001, "ERROR"), (1002, "INFO")).toDF("log_id", "level")
val cnt = df.count()
println(s"Total Rows: $cnt")
df.show()

Expected Output

Total Rows: 2
+------+-----+
|log_id|level|
+------+-----+
|  1001|ERROR|
|  1002| INFO|
+------+-----+

SQL Implementation

SELECT COUNT(*) 
FROM logs;

Expected Output

Executing this SQL query returns:

Common Mistakes

• OOM from collect(): Using .collect() to pull a massive dataset to the driver node. This will crash the driver with an Out Of Memory (OOM) error. Use .take(n) instead.

Best Practices

• Prefer Writing to Disk: In production pipelines, write results directly to disk using .write.save() rather than bringing data back to the driver.

Interview Perspective

Interactive Challenges

Related Topics

• DataFrame Transformations
• Performance Tuning Checklist
• Caching & Persistence