advanced

Serialization

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how serialization formats affect Spark query speeds, and configure Kryo Serialization to optimize network data transfers.

What You Will Learn

In this lesson, you will learn:
  • Data Serialization: Converting objects into byte streams for storage and network transfer.
  • Java Serializer: The default Java serialization format.
  • Kryo Serializer: A compact, high-performance serialization format.

Detailed Concept Explanation

Whenever Spark shuffles data across worker nodes or spills partitions to disk, it must convert Java objects into byte streams. This process is called Serialization.

Serializer Options

  1. Java Serializer (Default): Uses standard Java Object Serialization. It is flexible and easy to use but produces large, bloated byte payloads, wasting network bandwidth.
  2. Kryo Serializer: A compact serialization framework. It is 10x faster and produces up to 4x smaller payloads than Java serialization, saving significant network I/O.

To enable Kryo, you configure it in the Spark session: spark.serializer = org.apache.spark.serializer.KryoSerializer


Code Examples

Python (PySpark) Configuration

python
from pyspark.sql import SparkSession

# Configure Kryo serialization
spark = SparkSession.builder \
    .appName("KryoTest") \
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
    .config("spark.kryoserializer.buffer.max", "64m") \
    .getOrCreate()

df = spark.range(1, 1000).repartition(10)
df.collect()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
config(spark.serializer
KryoSerializer)
range(1
1000)
repartition(10)
collect()

Scala Configuration

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("KryoScala")
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .config("spark.kryoserializer.buffer.max", "64m")
  .getOrCreate()

val df = spark.range(1, 1000).repartition(10)
df.collect()

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.