Kubernetes

Overview

Learn how to deploy Spark natively on Kubernetes clusters, containerize applications, and manage dynamic executor Pod configurations.

What You Will Learn

In this lesson, you will learn:

K8s native scheduling: How Spark submits tasks directly to K8s API servers.
Containerization: Building Docker images with Spark binaries.
Dynamic Pods: Scaling executor pods up and down automatically.

Detailed Concept Explanation

Apache Spark supports Kubernetes (K8s) as a native resource manager. Instead of configuring standalone worker daemons, Spark interacts directly with the Kubernetes API server.

Native Execution Flow

Submission: You run spark-submit targeting the K8s API server endpoint.
Driver Pod: K8s schedules and runs a Driver Pod on a cluster node.
Executor Pods: The driver pod contacts K8s to spin up Executor Pods dynamically.
Execution: The driver sends tasks to the executor pods.
Termination: When the job finishes, the executor pods are destroyed, and the driver pod transitions to a completed state to preserve logs.

Code Examples

Kubernetes Submit Command

bash

# Submit application to Kubernetes Master Endpoint
spark-submit \
  --master k8s://https://k8s-apiserver:6443 \
  --deploy-mode cluster \
  --name spark-k8s-job \
  --conf spark.kubernetes.container.image=my-registry/spark-py:v3.2.0 \
  --conf spark.executor.instances=3 \
  local:///opt/spark/work-dir/app.py

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

spark-submit master(k8s)

API Server launches Driver Pod

Driver requests Executor Pods

Tasks executed

Executor Pods terminated