intermediate
Azure Blob Storage
10 min readLast updated: 2026-07-08
Overview
Connect Spark to Azure Blob Storage and Azure Data Lake Storage Gen2 (ADLS Gen2) to read and write files directly from containers.
What You Will Learn
In this lesson, you will learn:
- Azure Connector: Configuring the Azure WASB and ABFS filesystem connectors.
- Credentials: Configuring storage account access keys.
- Container Paths: Reading and writing data using the
abfss://URI scheme.
Detailed Concept Explanation
Spark connects to Azure Blob Storage using the wasbs:// URI scheme and to ADLS Gen2 using the abfss:// URI scheme.
You must configure access keys or Shared Access Signature (SAS) tokens to authenticate with your Azure storage account.
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("AzureTest") \
.config("spark.jars.packages", "org.apache.hadoop:hadoop-azure:3.2.0") \
.getOrCreate()
# Configure Azure access key on the Hadoop configuration
hadoop_conf = spark._jsc.hadoopConfiguration()
hadoop_conf.set("fs.azure.account.key.mystorageaccount.dfs.core.windows.net", "MY_ACCESS_KEY")
# Read Parquet files from ADLS Gen2 container
df = spark.read.parquet("abfss://my-container@mystorageaccount.dfs.core.windows.net/data/logs.parquet")
df.show()
Expected Output
text
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
hadoopConfiguration().set(fs.azure.account.key)
read.parquet(abfss://my-container)
show()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("AzureScala").getOrCreate()
val sc = spark.sparkContext
sc.hadoopConfiguration.set("fs.azure.account.key.mystorageaccount.dfs.core.windows.net", "MY_ACCESS_KEY")
val df = spark.read.parquet("abfss://my-container@mystorageaccount.dfs.core.windows.net/data/logs.parquet")
df.show()
Expected Output
text
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+