Introduction to Pandas
Overview
Pandas is the standard tool for data manipulation in Python. It provides Series (1D) and DataFrames (2D) to manage tabular data.
Learning Objectives
- Understand the difference between Series and DataFrames.
- Create Series and DataFrames from standard dictionaries.
- Learn why Pandas is essential for data engineering tasks.
Concept Explanation
Pandas wraps NumPy arrays to provide labeled rows and columns. A Series is a one-dimensional labeled array. A DataFrame is a two-dimensional tabular structure (like an Excel sheet or SQL table). Labeled columns make working with datasets intuitive.
Code Examples
Example 1 — Basics
This example introduces the fundamental syntax and concepts.
import pandas as pd
# Create a Series mapping values to index labels
s = pd.Series([10, 20, 30], index=['a', 'b', 'c'])
print(s)
Example 2 — Everyday Usage
This example demonstrates a realistic scenario handling business parameters.
# Create a DataFrame from dictionary list data
data = {
'Name': ['Alice', 'Bob'],
'Department': ['HR', 'IT'],
'Salary': [50000, 60000]
}
df = pd.DataFrame(data)
print(df)
Example 3 — Practical Data Engineering Example
This example shows clean, production-grade code structure following senior development standards.
# Build configuration schema DataFrame and check basic properties
configs = {
'host': ['localhost', 'dev-db'],
'port': [5432, 3306],
'ssl': [True, False]
}
df_conf = pd.DataFrame(configs, index=['pg', 'mysql'])
print('Config table shape:', df_conf.shape)
print('System indexes:', df_conf.index.tolist())
Visual Flow
The following execution flow represents the step-by-step evaluation inside the interpreter:
Source Dict Data → Allocate Labeled Series / Columns → Assemble tabular DataFrame with Row Index
Common Mistakes
Review these common pitfalls when working with this topic:
- Confusing Series (1D) with DataFrames (2D).
- Forgetting to import pandas as pd.
- Assuming index values must be unique integers.
- Confusing DataFrame columns names with data row entries.
Best Practices
Enforce these Pythonic best practices in your codebase:
import pandas
df = pandas.DataFrame(data) # Verbose module name
import pandas as pd
df = pd.DataFrame(data) # Clean standard import alias
Performance Notes
Keep these optimization guidelines in mind for performance-sensitive hotpaths:
- Pandas structures keep data aligned using indexes, ensuring fast lookups and row lookups.
Quick Revision
Use these key summaries for last-minute revision:
- Pandas is the standard tool for tabular data manipulation.
- Series represents 1D columns; DataFrame represents 2D tables.
- DataFrames are made of column Series sharing an index.
- Import pandas as pd by convention.
- DataFrames can be created from dictionaries and lists.
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.