beginner

spark-submit

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how to use the spark-submit command-line utility to package, submit, and configure applications for execution on your cluster.

What You Will Learn

In this lesson, you will learn:
  • Submit CLI: The flags and structure of spark-submit statements.
  • Driver configurations: Configuring memory, cores, and drivers allocations.
  • Class mappings: Targeting specific execution entries.

Detailed Concept Explanation

The spark-submit script is the standard tool used to launch applications on a Spark cluster. It configures the classpaths, loads dependencies, and sets JVM runtime flags before launching the driver process.

Common CLI Flags

  • --master: Specifies the cluster manager URL (e.g. spark://ip:7077, yarn, k8s://...).
  • --deploy-mode: Chooses whether to run the driver locally (client) or on a worker node (cluster).
  • --class: The entry point class name for Scala/Java applications (e.g. com.example.MyApp).
  • --executor-memory: Memory allocated per executor process (e.g. 4G).
  • --executor-cores: Number of CPU cores allocated per executor (e.g. 2).
  • --num-executors: Total number of executors to launch (YARN only).

Code Examples

Command-Line submit syntax

bash
spark-submit \
  --master local[*] \
  --name "MyPythonJob" \
  --executor-memory 2G \
  --executor-cores 2 \
  my_app.py

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
spark-submit utility
parse CLI flags
start JVM driver process
allocate executors
run app

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.