advanced

Kafka Streaming

By DataPlayArena Technical Review BoardUpdated 2026-07-0910 min readRuntime Verified

Overview

Connect Spark Structured Streaming to Apache Kafka to read and write real-time message streams.

What You Will Learn

In this lesson, you will learn:
  • Kafka Integration: Streaming data from Kafka topics.
  • Value Casting: Converting binary messages to structured columns.
  • Writing to Kafka: Sending structured query outputs back to Kafka topics.

Detailed Concept Explanation

To read streams from Apache Kafka, you use readStream with the kafka format.

The source schema contains key metadata:

  • key (binary) and value (binary)
  • topic (string)
  • partition (int)
  • offset (long)

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("KafkaStreaming") \
    .config("spark.jars.packages", "org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.0") \
    .getOrCreate()

# Stream from Kafka topic
kafka_stream = spark.readStream.format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "realtime-topic") \
    .load()

# Cast binary value payload to string
messages = kafka_stream.selectExpr("CAST(value AS STRING) AS msg")

# Print to console
query = messages.writeStream.format("console").start()
query.awaitTermination(5)
query.stop()

Expected Output

text
+---------------------+
|                  msg|
+---------------------+
|{"event": "purchase"}|
+---------------------+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
readStream.format(kafka)
load()
selectExpr(CAST value AS STRING)
writeStream.format(console)
start()
awaitTermination(5)
stop()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("KafkaStreamingScala")
  .config("spark.jars.packages", "org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.0")
  .getOrCreate()

val kafkaStream = spark.readStream.format("kafka")
  .option("kafka.bootstrap.servers", "localhost:9092")
  .option("subscribe", "realtime-topic")
  .load()

val messages = kafkaStream.selectExpr("CAST(value AS STRING) AS msg")

val query = messages.writeStream.format("console").start()
query.awaitTermination(5000)
query.stop()

Expected Output

text
+---------------------+
|                  msg|
+---------------------+
|{"event": "purchase"}|
+---------------------+

Common Mistakes

  • Missing Kafka dependencies: Running Kafka stream queries without loading the Kafka connector package dependency at startup, which will throw a class-not-found exception.

Best Practices

  • Define schemas for JSON payloads: After casting the binary value to a string, use from_json with an explicit schema to parse JSON messages into structured columns.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.