Spark Configurations Reference

Important Spark properties to tune memory, parallelism, and AQE.

spark.sql.shuffle.partitions

Return: None

Sets default partition numbers for shuffles.

Syntax signature:spark.conf.set('spark.sql.shuffle.partitions', '200')

Code snippet:

python

spark.conf.set('spark.sql.shuffle.partitions', '100')

Remember: Defaults to 200. Shrink for small datasets to avoid scheduling overhead on empty tasks.

spark.sql.adaptive.enabled

Return: None

Enables Adaptive Query Execution (AQE).

Syntax signature:spark.conf.set('spark.sql.adaptive.enabled', 'true')

Code snippet:

python

spark.conf.set('spark.sql.adaptive.enabled', 'true')

Remember: Always keep true in Spark 3+ to handle partition sizing, skew joins, and join plan optimizations automatically.