Avro

Overview

Avro is a row-based, binary serialization format. Learn why it is the standard choice for streaming message pipelines (like Apache Kafka) and schema registries.

What You Will Learn

Detailed Concept Explanation

Unlike Parquet and ORC, which are columnar, Apache Avro is a row-based binary format.

Why Row-Based for Streaming?

• Fast Writes: Row-oriented formats write rows sequentially, making them ideal for writing incoming records one-by-one in real-time streaming systems like Apache Kafka. Columnar formats are slow for streaming because they require buffering data in memory to group columns together before writing.
• Strict Schema: Avro stores its schema as a JSON header, ensuring that reader and writer systems always agree on data types.

Code Examples

Input Dataset Preview

Below is the users logs dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("AvroTest").getOrCreate()
data = [(1, "click"), (2, "scroll")]
df = spark.createDataFrame(data, ["userId", "event"])

# Write to Avro
df.write.format("avro").mode("overwrite").save("logs.avro")

# Read Avro
read_df = spark.read.format("avro").load("logs.avro")
read_df.show()

Expected Output

+------+------+
|userId| event|
+------+------+
|     1| click|
|     2|scroll|
+------+------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("AvroScala").getOrCreate()
import spark.implicits._

val df = Seq((1, "click"), (2, "scroll")).toDF("userId", "event")
df.write.format("avro").mode("overwrite").save("logs.avro")

val readDF = spark.read.format("avro").load("logs.avro")
readDF.show()

Expected Output

+------+------+
|userId| event|
+------+------+
|     1| click|
|     2|scroll|
+------+------+

SQL Implementation

-- Query Avro files directly using SQL path syntax
SELECT * 
FROM avro.`logs.avro`
WHERE event = 'click';

Expected Output

Executing this SQL query returns:

Common Mistakes

• Forgetting Spark Avro Package: In older Spark versions, Avro was not built-in. Running spark.read.format("avro") without passing the Avro coordinate package coordinates (--packages org.apache.spark:spark-avro_2.12:<version>) would throw a class-not-found exception. Always ensure the package coordinate is loaded during cluster startup.

Best Practices

• Use for Kafka Ingestion: Use Avro as the serialization format when reading or writing streams from Apache Kafka to leverage Schema Registries.

Interview Perspective

Interactive Challenges

Related Topics

• Parquet
• ORC
• Reading Data