RDD API Reference

Low-level Resilient Distributed Dataset (RDD) transformations and actions.

map

Return: RDD

Applies a transformation function to each element of the dataset.

Syntax signature:rdd.map(func)

Code snippet:

python

rdd.map(lambda x: x * 2)

Remember: map operates element-by-element. If you need to open database connections, use mapPartitions instead.

mapPartitions

Return: RDD

Applies a transformation function to each partition block of the RDD.

Syntax signature:rdd.mapPartitions(func)

Code snippet:

python

rdd.mapPartitions(lambda part: [sum(part)])

Remember: Excellent for initiating expensive connections (e.g. database client hooks) once per partition rather than once per row.

reduceByKey

Return: RDD

Merges the values for each key using an associative reduce function.

Syntax signature:rdd.reduceByKey(func)

Code snippet:

python

rdd.reduceByKey(lambda a, b: a + b)

Remember: Performs map-side aggregation automatically, making it much more network-efficient than groupByKey.

collect

Return: list

Returns all elements of the RDD to the driver node as a local list.

Syntax signature:rdd.collect()

Code snippet:

python

rdd.collect()

Remember: Critical Red Flag: Calling collect() on huge datasets will overload driver RAM and raise OutOfMemory errors.