Amazon S3
Overview
Connect Spark to Amazon S3 using the AWS S3A connector to read and write files directly from cloud storage buckets.
What You Will Learn
In this lesson, you will learn:
- S3A connector: Configuring the AWS S3A filesystem connector.
- Credentials: Configuring S3 access and secret keys securely.
- Bucket Paths: Reading and writing data using the
s3a://URI scheme.
Detailed Concept Explanation
Spark connects to Amazon S3 buckets using Hadoop's filesystem adapter class org.apache.hadoop.fs.s3a.S3AFileSystem.
You must specify the path using the s3a:// URI scheme. (Avoid the legacy s3n:// or s3:// schemes, as they are slower and deprecated in Hadoop).
Code Examples
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("S3Test") \
.config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0") \
.getOrCreate()
# Configure S3 credentials on the Hadoop configuration
hadoop_conf = spark._jsc.hadoopConfiguration()
hadoop_conf.set("fs.s3a.access.key", "MY_ACCESS_KEY")
hadoop_conf.set("fs.s3a.secret.key", "MY_SECRET_KEY")
hadoop_conf.set("fs.s3a.endpoint", "s3.amazonaws.com")
# Read Parquet files from an S3 bucket
df = spark.read.parquet("s3a://my-bucket/data/logs.parquet")
df.show()
Expected Output
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("S3Scala").getOrCreate()
val sc = spark.sparkContext
sc.hadoopConfiguration.set("fs.s3a.access.key", "MY_ACCESS_KEY")
sc.hadoopConfiguration.set("fs.s3a.secret.key", "MY_SECRET_KEY")
val df = spark.read.parquet("s3a://my-bucket/data/logs.parquet")
df.show()
Expected Output
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.