advanced

CombineByKey

By DataPlayArena Technical Review BoardUpdated 2026-07-0810 min readRuntime Verified

Overview

Learn how to use combineByKey, the low-level aggregation primitive that underlies Spark's key-value operations.

What You Will Learn

In this lesson, you will learn:
  • CombineByKey API: The low-level key-value aggregation mechanism.
  • Create Combiner: Defining the initial value mapper.
  • Merge Value & Merge Combiners: Merging values locally and globally.

Detailed Concept Explanation

combineByKey is the lowest-level aggregation function in Spark. Both reduceByKey and aggregateByKey are implemented on top of it.

It takes three functions as parameters:

  1. createCombiner: Converts the first value found for a key in a partition into an accumulator (e.g. lambda x: (x, 1)).
  2. mergeValue: Merges subsequent values found in the same partition into the accumulator.
  3. mergeCombiners: Merges accumulators from different partitions over the network.

Code Examples

Input Dataset Preview

Below is the department scores dataset:

deptscore
HR80
IT90
HR85

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CombineByKey").getOrCreate()
sc = spark.sparkContext
pair_rdd = sc.parallelize([("HR", 80), ("IT", 90), ("HR", 85)])

# Define combine functions
create_combiner = lambda val: (val, 1)
merge_value = lambda acc, val: (acc[0] + val, acc[1] + 1)
merge_combiners = lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])

combined = pair_rdd.combineByKey(create_combiner, merge_value, merge_combiners)
print(combined.collect())

Expected Output

text
[('HR', (165, 2)), ('IT', (90, 1))]

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkContext.parallelize
combineByKey(create mergeValue mergeCombiners)
collect()
print()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("CombineByKeyScala").getOrCreate()
val sc = spark.sparkContext

val pairRDD = sc.parallelize(Seq(("HR", 80), ("IT", 90), ("HR", 85)))

val createCombiner = (val: Int) => (val, 1)
val mergeValue = (acc: (Int, Int), val: Int) => (acc._1 + val, acc._2 + 1)
val mergeCombiners = (acc1: (Int, Int), acc2: (Int, Int)) => (acc1._1 + acc2._1, acc1._2 + acc2._2)

val combined = pairRDD.combineByKey(createCombiner, mergeValue, mergeCombiners)
println(combined.collect().mkString(", "))

Expected Output

text
(HR,(165,2)), (IT,(90,1))

Interview Perspective

What is combineByKey and what are its three parameter functions?

combineByKey is Spark's lowest-level key-value aggregation method.

  1. createCombiner: Converts the first value found for a key in a partition into an accumulator.
  2. mergeValue: Merges subsequent values in the same partition into the accumulator.
  3. mergeCombiners: Merges accumulators from different partitions over the network.

Interactive Challenges

Challenge 1: Map to Averages (Advanced)

How do you calculate the final average scores per department from the combineByKey output?

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.