Spark Architecture Reference

Quick architecture rules mapping Driver, Cluster Manager, and Executors.

Driver Memory

Return: CLI Flag

Allocates RAM heap space for the Driver process.

Syntax signature:--driver-memory 4g

Code snippet:

python

spark-submit --driver-memory 4g app.py

Remember: Increase if collecting large summaries or broadcasting heavy schemas to prevent Driver out-of-memory.

Executor Cores

Return: CLI Flag

Specifies number of CPU cores allocated per Executor.

Syntax signature:--executor-cores 4

Code snippet:

python

spark-submit --executor-cores 4 app.py

Remember: Best practice is to use 4 to 5 cores per executor. Higher counts can lead to Garbage Collection stalls.