Apache Mesos

Overview

Learn about Apache Mesos deployment architecture and understand historic resource scheduling differences in legacy Spark pipelines.

What You Will Learn

In this lesson, you will learn:

Mesos Architecture: Master and Agent daemons coordination.
Fine-Grained vs. Coarse-Grained: Historic resource sharing mechanisms.
Legacy migration: Moving from Mesos to modern schedulers (YARN/K8s).

Detailed Concept Explanation

Apache Mesos is a legacy open-source cluster manager designed to manage resources across entire data centers. (Note: Spark has officially deprecated Mesos support in recent versions, but it remains a common topic in legacy migration architectures).

Scheduling Modes

Coarse-Grained (Default): Spark allocates static resources up front, launching fixed executors that remain active throughout the application run. This provides fast execution but locks up cluster resources.
Fine-Grained (Legacy): Spark dynamically releases executor resources back to Mesos when tasks are idle. This increases resource efficiency but introduces launch latency overheads.

Code Examples

Mesos Submit Command

bash

# Submit job to Mesos master url
spark-submit \
  --master mesos://mesos-master-ip:5050 \
  --deploy-mode client \
  my_app.py