spark-submit

Overview

Learn how to use the spark-submit command-line utility to package, submit, and configure applications for execution on your cluster.

What You Will Learn

In this lesson, you will learn:

Submit CLI: The flags and structure of spark-submit statements.
Driver configurations: Configuring memory, cores, and drivers allocations.
Class mappings: Targeting specific execution entries.

Detailed Concept Explanation

The spark-submit script is the standard tool used to launch applications on a Spark cluster. It configures the classpaths, loads dependencies, and sets JVM runtime flags before launching the driver process.

Common CLI Flags

--master: Specifies the cluster manager URL (e.g. spark://ip:7077, yarn, k8s://...).
--deploy-mode: Chooses whether to run the driver locally (client) or on a worker node (cluster).
--class: The entry point class name for Scala/Java applications (e.g. com.example.MyApp).
--executor-memory: Memory allocated per executor process (e.g. 4G).
--executor-cores: Number of CPU cores allocated per executor (e.g. 2).
--num-executors: Total number of executors to launch (YARN only).

Code Examples

Command-Line submit syntax

bash

spark-submit \
  --master local[*] \
  --name "MyPythonJob" \
  --executor-memory 2G \
  --executor-cores 2 \
  my_app.py

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

spark-submit utility

parse CLI flags

start JVM driver process

allocate executors

run app