Selecting Columns

Overview

Learn how to retrieve specific columns, rename column headers with aliases, and drop unwanted fields from your Spark DataFrames.

What You Will Learn

Detailed Concept Explanation

Spark DataFrames provide the .select() transformation to specify which columns should be kept or calculated. To rename columns in the output, we use .alias(). To remove columns, we use .drop().

Selecting only the columns you need reduces the volume of data that Spark needs to pass between nodes, optimizing performance.

Code Examples

Input Dataset Preview

Below is the employee dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("SelectColumns").getOrCreate()
data = [("Alice", "Smith", 120000), ("Bob", "Jones", 90000)]
df = spark.createDataFrame(data, ["first_name", "last_name", "yearly_salary"])

# Select and alias a column, and drop last_name
selected_df = df.select(col("first_name"), col("yearly_salary").alias("salary"))
selected_df.show()

Expected Output

+----------+------+
|first_name|salary|
+----------+------+
|     Alice|120000|
|       Bob| 90000|
+----------+------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("SelectScala").getOrCreate()
import spark.implicits._

val df = Seq(("Alice", "Smith", 120000), ("Bob", "Jones", 90000))
  .toDF("first_name", "last_name", "yearly_salary")

val selected = df.select($"first_name", $"yearly_salary".as("salary"))
selected.show()

Expected Output

+----------+------+
|first_name|salary|
+----------+------+
|     Alice|120000|
|       Bob| 90000|
+----------+------+

SQL Implementation

SELECT first_name, yearly_salary AS salary 
FROM employees;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Passing String Literals to Col: Mixing string literals and col() objects incorrectly in complex selections can cause parsing errors.

Best Practices

• Drop Unused Columns: Drop unnecessary columns early in your execution flow to minimize RAM requirements.

Interview Perspective

Interactive Challenges

Related Topics

• Working with Columns
• Filtering Data
• Sorting Data