beginner
spark-submit
Overview
Learn how to use the spark-submit command-line utility to package, submit, and configure applications for execution on your cluster.
What You Will Learn
In this lesson, you will learn:
- Submit CLI: The flags and structure of spark-submit statements.
- Driver configurations: Configuring memory, cores, and drivers allocations.
- Class mappings: Targeting specific execution entries.
Detailed Concept Explanation
The spark-submit script is the standard tool used to launch applications on a Spark cluster. It configures the classpaths, loads dependencies, and sets JVM runtime flags before launching the driver process.
Common CLI Flags
--master: Specifies the cluster manager URL (e.g.spark://ip:7077,yarn,k8s://...).--deploy-mode: Chooses whether to run the driver locally (client) or on a worker node (cluster).--class: The entry point class name for Scala/Java applications (e.g.com.example.MyApp).--executor-memory: Memory allocated per executor process (e.g.4G).--executor-cores: Number of CPU cores allocated per executor (e.g.2).--num-executors: Total number of executors to launch (YARN only).
Code Examples
Command-Line submit syntax
bash
spark-submit \
--master local[*] \
--name "MyPythonJob" \
--executor-memory 2G \
--executor-cores 2 \
my_app.py
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
spark-submit utility
parse CLI flags
start JVM driver process
allocate executors
run app
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.