advanced
CombineByKey
10 min readLast updated: 2026-07-08
Overview
Learn how to use combineByKey, the low-level aggregation primitive that underlies Spark's key-value operations.
What You Will Learn
In this lesson, you will learn:
- CombineByKey API: The low-level key-value aggregation mechanism.
- Create Combiner: Defining the initial value mapper.
- Merge Value & Merge Combiners: Merging values locally and globally.
Detailed Concept Explanation
combineByKey is the lowest-level aggregation function in Spark. Both reduceByKey and aggregateByKey are implemented on top of it.
It takes three functions as parameters:
createCombiner: Converts the first value found for a key in a partition into an accumulator (e.g.lambda x: (x, 1)).mergeValue: Merges subsequent values found in the same partition into the accumulator.mergeCombiners: Merges accumulators from different partitions over the network.
Code Examples
Input Dataset Preview
Below is the department scores dataset:
| dept | score |
|---|---|
| HR | 80 |
| IT | 90 |
| HR | 85 |
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CombineByKey").getOrCreate()
sc = spark.sparkContext
pair_rdd = sc.parallelize([("HR", 80), ("IT", 90), ("HR", 85)])
# Define combine functions
create_combiner = lambda val: (val, 1)
merge_value = lambda acc, val: (acc[0] + val, acc[1] + 1)
merge_combiners = lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])
combined = pair_rdd.combineByKey(create_combiner, merge_value, merge_combiners)
print(combined.collect())
Expected Output
text
[('HR', (165, 2)), ('IT', (90, 1))]
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkContext.parallelize
combineByKey(create mergeValue mergeCombiners)
collect()
print()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("CombineByKeyScala").getOrCreate()
val sc = spark.sparkContext
val pairRDD = sc.parallelize(Seq(("HR", 80), ("IT", 90), ("HR", 85)))
val createCombiner = (val: Int) => (val, 1)
val mergeValue = (acc: (Int, Int), val: Int) => (acc._1 + val, acc._2 + 1)
val mergeCombiners = (acc1: (Int, Int), acc2: (Int, Int)) => (acc1._1 + acc2._1, acc1._2 + acc2._2)
val combined = pairRDD.combineByKey(createCombiner, mergeValue, mergeCombiners)
println(combined.collect().mkString(", "))
Expected Output
text
(HR,(165,2)), (IT,(90,1))
Interview Perspective
What is combineByKey and what are its three parameter functions?
combineByKey is Spark's lowest-level key-value aggregation method.
createCombiner: Converts the first value found for a key in a partition into an accumulator.mergeValue: Merges subsequent values in the same partition into the accumulator.mergeCombiners: Merges accumulators from different partitions over the network.