advanced

AggregateByKey

By DataPlayArena Technical Review BoardUpdated 2026-07-0810 min readRuntime Verified

Overview

Learn how to use aggregateByKey to aggregate Pair RDD values using different functions for local combining and cross-partition merging.

What You Will Learn

In this lesson, you will learn:
  • AggregateByKey API: Custom key-value aggregations.
  • Zero Value: Setting the starting value for the aggregate.
  • Two-phase logic: Defining separate seqOp (local) and combOp (global) functions.

Detailed Concept Explanation

aggregateByKey is a powerful wide transformation on Pair RDDs. It is used when the output value of your aggregation has a different type than the input values (for example, taking a list of integers and calculating the average—which requires tracking both sum and count as a tuple).

Parameters

It takes three parameters:

  1. zeroValue: The starting value for the aggregation (e.g. (0, 0)).
  2. seqOp: The function used to aggregate values locally on each partition (map-side combining).
  3. combOp: The function used to merge the aggregates from different partitions over the network.

Code Examples

Input Dataset Preview

Below is the sales scores dataset:

itemscore
A10
B15
A20

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("AggregateByKey").getOrCreate()
sc = spark.sparkContext
pair_rdd = sc.parallelize([("A", 10), ("B", 15), ("A", 20)])

# Calculate sum and count for each key
zero_val = (0, 0)
seq_op = lambda acc, value: (acc[0] + value, acc[1] + 1)
comb_op = lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])

aggregated = pair_rdd.aggregateByKey(zero_val, seq_op, comb_op)
print(aggregated.collect())

Expected Output

text
[('A', (30, 2)), ('B', (15, 1))]

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkContext.parallelize
aggregateByKey(zeroVal seqOp combOp)
collect()
print()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("AggregateByKeyScala").getOrCreate()
val sc = spark.sparkContext

val pairRDD = sc.parallelize(Seq(("A", 10), ("B", 15), ("A", 20)))

val zeroVal = (0, 0)
val seqOp = (acc: (Int, Int), value: Int) => (acc._1 + value, acc._2 + 1)
val combOp = (acc1: (Int, Int), acc2: (Int, Int)) => (acc1._1 + acc2._1, acc1._2 + acc2._2)

val aggregated = pairRDD.aggregateByKey(zeroVal)(seqOp, combOp)
println(aggregated.collect().mkString(", "))

Expected Output

text
(A,(30,2)), (B,(15,1))

Common Mistakes

  • Zero Value Mutation: Designing zero values that mutate inside the seqOp. Always return a new tuple/object to prevent partition memory corruption.

Interview Perspective

What are the three parameters of aggregateByKey and what do they do?
  1. Zero Value: The starting value for the aggregation in each partition.
  2. seqOp (Sequence Operator): Aggregates values locally on each partition, taking the accumulator and a row value.
  3. combOp (Combine Operator): Merges the local accumulators from different partitions over the network.

Interactive Challenges

Challenge 1: Calculate Averages (Advanced)

How do you compute the final average for each key from the output of the aggregateByKey example?

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.