RDD Actions

Overview

Learn about RDD actions—the execution commands that compile and run tasks to return data to the driver node or save it to storage.

What You Will Learn

Detailed Concept Explanation

RDD actions compile the lineage DAG and trigger task execution across the cluster. Common actions:

• collect(): Retrieves all records from the executors and returns them to the driver as a list.
• count(): Returns the total number of records.
• take(n): Returns the first n records.
• reduce(func): Aggregates RDD elements using an associative and commutative function (e.g. summing numbers).

Code Examples

Input Dataset Preview

Below is the numbers dataset we will aggregate:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("RDDActions").getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize([1, 2, 3, 4])

# Sum elements using reduce
total_sum = rdd.reduce(lambda a, b: a + b)
print("Total Sum:", total_sum)

Expected Output

Total Sum: 10

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("RDDActionsScala").getOrCreate()
val sc = spark.sparkContext

val rdd = sc.parallelize(Seq(1, 2, 3, 4))
val totalSum = rdd.reduce((a, b) => a + b)
println(s"Total Sum: $totalSum")

Expected Output

Total Sum: 10

Common Mistakes

• OOM from collect(): Using .collect() on massive RDDs. This pulls all data into the driver's memory, which will trigger an Out Of Memory (OOM) crash. Use .take(n) or save results to disk.

Best Practices

• Save to storage: Use .saveAsTextFile("path") to write large RDD outputs directly to distributed storage, avoiding the driver node completely.

Related Topics

• RDD Transformations
• Pair RDDs
• Persistence