Performance Optimization Reference

Guidelines and techniques to optimize Spark applications.

cache

Return: DataFrame

Persists the DataFrame with default storage level (MEMORY_AND_DISK).

Syntax signature:df.cache()

Code snippet:

python

df.cache()

Remember: Call actions (like count) immediately after caching to force materialization of the cached memory.

coalesce

Return: DataFrame

Decreases partition numbers without triggering a full network shuffle.

Syntax signature:df.coalesce(numPartitions)

Code snippet:

python

df.coalesce(5)

Remember: Always use coalesce when shrinking partition counts before output writes to prevent network shuffles.

repartition

Return: DataFrame

Changes partition counts dynamically by forcing a full network shuffle.

Syntax signature:df.repartition(numPartitions, *cols)

Code snippet:

python

df.repartition(10, 'country')

Remember: Use repartition to increase partition size or distribute values evenly on key hashes to mitigate skew.