intermediate
Amazon S3
10 min readLast updated: 2026-07-08
Overview
Connect Spark to Amazon S3 using the AWS S3A connector to read and write files directly from cloud storage buckets.
What You Will Learn
In this lesson, you will learn:
- S3A connector: Configuring the AWS S3A filesystem connector.
- Credentials: Configuring S3 access and secret keys securely.
- Bucket Paths: Reading and writing data using the
s3a://URI scheme.
Detailed Concept Explanation
Spark connects to Amazon S3 buckets using Hadoop's filesystem adapter class org.apache.hadoop.fs.s3a.S3AFileSystem.
You must specify the path using the s3a:// URI scheme. (Avoid the legacy s3n:// or s3:// schemes, as they are slower and deprecated in Hadoop).
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("S3Test") \
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0") \
.getOrCreate()
# Configure S3 credentials on the Hadoop configuration
hadoop_conf = spark._jsc.hadoopConfiguration()
hadoop_conf.set("fs.s3a.access.key", "MY_ACCESS_KEY")
hadoop_conf.set("fs.s3a.secret.key", "MY_SECRET_KEY")
hadoop_conf.set("fs.s3a.endpoint", "s3.amazonaws.com")
# Read Parquet files from an S3 bucket
df = spark.read.parquet("s3a://my-bucket/data/logs.parquet")
df.show()
Expected Output
text
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
hadoopConfiguration().set(fs.s3a)
read.parquet(s3a://my-bucket)
show()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("S3Scala").getOrCreate()
val sc = spark.sparkContext
sc.hadoopConfiguration.set("fs.s3a.access.key", "MY_ACCESS_KEY")
sc.hadoopConfiguration.set("fs.s3a.secret.key", "MY_SECRET_KEY")
val df = spark.read.parquet("s3a://my-bucket/data/logs.parquet")
df.show()
Expected Output
text
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+