SPARK Reference Guide

Revision Time: 10 mins

RDD API Reference

Low-level Resilient Distributed Dataset (RDD) transformations, actions, partition controls, and memory persistence.

map

Return: RDD

Applies a transformation function to each individual element of the RDD and returns a new RDD.

Used In: Element Parsing, Data Formatting

Syntax signature:rdd.map(func)

Code snippet:

python

rdd = sc.parallelize([1, 2, 3, 4])
rdd2 = rdd.map(lambda x: (x, x * 10))
rdd2.collect()

Expected Output:[(1, 10), (2, 20), (3, 30), (4, 40)]

Common Mistakes:Executing database calls inside `map()`, creating 1 million DB connection handshakes instead of 1 per partition.

Remember: Operates element-by-element. If initializing external DB connections or HTTP clients, use `mapPartitions` instead.

flatMap

Return: RDD

Applies a function returning an iterable to each element, then flattens the result into a single stream.

Used In: Tokenization, Text Processing, Flattening Nested Lists

Syntax signature:rdd.flatMap(func)

Code snippet:

python

lines = sc.parallelize(['hello world', 'apache spark'])
words = lines.flatMap(lambda line: line.split(' '))
words.collect()

Expected Output:['hello', 'world', 'apache', 'spark']

Remember: Returns 0, 1, or multiple output elements per input item. Essential for word counts and tokenization.

mapPartitions

Return: RDD

Applies a transformation function to each partition block of the RDD (passes an iterator).

Used In: External DB Writes, Model Inference, Batching

Syntax signature:rdd.mapPartitions(func)

Code snippet:

python

def process_partition(iterator):
    # Initialize expensive DB connection once per partition
    db = connect_db()
    for item in iterator:
        yield db.transform(item)
rdd.mapPartitions(process_partition)

Expected Output:Transformed RDD iterator stream

Remember: Dramatic performance optimization when opening heavy resources (DB handles, socket connections, machine learning models).

reduceByKey

Return: RDD

Merges values for each key using an associative and commutative reduce function.

Used In: Word Count, Key-wise Aggregations

Syntax signature:rdd.reduceByKey(func)

Code snippet:

python

pairs = sc.parallelize([('a', 1), ('b', 1), ('a', 2)])
counts = pairs.reduceByKey(lambda x, y: x + y)
counts.collect()

Expected Output:[('a', 3), ('b', 1)]

Common Mistakes:Using `groupByKey().mapValues(sum)` instead of `reduceByKey(add)`.

Comparison:

vs groupByKey: reduceByKey combines data locally on each mapper node before sending across the network. groupByKey sends ALL raw pairs across the network.

Remember: Performs map-side combining (combiner) automatically before shuffling, making it infinitely faster than `groupByKey()`.

groupByKey

Return: RDD[(K, Iterable[V])]

Groups values for each key in the RDD into a single sequence (Iterable).

Used In: Grouping without Aggregation

Syntax signature:rdd.groupByKey()

Code snippet:

python

pairs = sc.parallelize([('dept1', 100), ('dept1', 200), ('dept2', 300)])
grouped = pairs.groupByKey().mapValues(list)
grouped.collect()

Expected Output:[('dept1', [100, 200]), ('dept2', [300])]

Common Mistakes:Using groupByKey for sums or averages. Use `reduceByKey` or `aggregateByKey` instead.

Remember: WARNING: Does NOT perform map-side combining. If a key has 10 million items, all 10M items are transferred across the network to 1 reducer, causing OutOfMemory errors.

aggregateByKey

Return: RDD[(K, U)]

Aggregates values of each key using an initial zero value, a sequence op (within partition), and a combine op (between partitions).

Used In: Custom Multi-metric Aggregations, Average Computation

Syntax signature:rdd.aggregateByKey(zeroValue, seqOp, combOp)

Code snippet:

python

# Compute average per key: zeroValue=(sum, count)
zero = (0, 0)
seqOp = lambda acc, val: (acc[0] + val, acc[1] + 1)
combOp = lambda acc1, acc2: (acc1[0] + acc2[0], acc1[1] + acc2[1])
avg_rdd = rdd.aggregateByKey(zero, seqOp, combOp).mapValues(lambda x: x[0]/x[1])

Expected Output:[('dept1', 150.0), ('dept2', 300.0)]

Remember: Allows the output aggregated value type `U` to be completely different from input value type `V`.

coalesce vs repartition

Return: RDD

Changes the number of partitions in the RDD.

Used In: Partition Tuning, Output File Consolidation

Syntax signature:rdd.coalesce(numPartitions, shuffle=False) | rdd.repartition(numPartitions)

Code snippet:

python

# Reduce 100 partitions to 10 without shuffle
rdd_small = rdd.coalesce(10)

# Increase 10 partitions to 50 with full shuffle
rdd_large = rdd.repartition(50)

Expected Output:Resized RDD with target partition count

Common Mistakes:Calling `coalesce(100)` when initial partitions are 10. `coalesce` without `shuffle=True` cannot increase partitions.

Remember: Use `coalesce` to REDUCE partitions (avoids shuffle by merging adjacent partitions). Use `repartition` to INCREASE partitions or rebalance skewed partitions (always triggers shuffle).

collect / take / count

Return: list / int

Core RDD action operations that trigger execution and retrieve data.

Used In: Driver Inspection, Quick Testing

Syntax signature:rdd.collect() | rdd.take(n) | rdd.count()

Code snippet:

python

print('Count:', rdd.count())
print('First 3:', rdd.take(3))

Expected Output:

Count: 500000
First 3: [1, 2, 3]

Common Mistakes:Calling `rdd.collect()` in production pipelines.

Remember: `collect()` pulls ALL records across all worker nodes to driver RAM. On 100GB datasets, it crashes driver with OutOfMemoryError. Use `take(n)` or `first()` for inspection.

persist / cache

Return: RDD

Persists RDD in memory or disk for fast reuse across multiple downstream actions.

Used In: Iterative Machine Learning, Multi-action Pipelines

Syntax signature:rdd.cache() | rdd.persist(StorageLevel.MEMORY_AND_DISK)

Code snippet:

python

from pyspark import StorageLevel
rdd.persist(StorageLevel.MEMORY_AND_DISK_SER)
rdd.count() # Triggers caching
rdd.take(10) # Uses cached memory

Expected Output:Cached RDD

Remember: Storage Levels: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, DISK_ONLY. Always `unpersist()` when RDD is no longer needed.

Previous TopicDataFrame API Next TopicSpark SQL