Selecting & Filtering Data

Overview

Learn how to select specific columns, filter rows based on conditions, and slice arrays using loc and iloc.

Learning Objectives

• Select specific columns and filter rows using logical checks.
• Slice datasets using index positions (iloc).
• Slice datasets using index labels (loc).

Concept Explanation

Use loc to select rows and columns by their labels. Use iloc to select by integer index positions. You can filter rows by passing boolean conditions inside brackets.

Code Examples

Example 1 — Basics

This example introduces the fundamental syntax and concepts.

import pandas as pd
# df['name'] # Select name column
# df[df['age'] > 30] # Filter rows by age

Example 2 — Everyday Usage

This example demonstrates a realistic scenario handling business parameters.

# Select values using loc and iloc labels
data = {
    'State': ['NY', 'CA', 'TX'],
    'Sales': [500, 700, 600]
}
df = pd.DataFrame(data, index=['a', 'b', 'c'])
print('First Row using iloc:\n', df.iloc[0])
print('Row b using loc:\n', df.loc['b'])
print('Filtering Sales > 550:\n', df[df['Sales'] > 550])

Example 3 — Practical Data Engineering Example

This example shows clean, production-grade code structure following senior development standards.

# Complex multi-criteria filter and column slice
data = {
    'name': ['Alex', 'Bob', 'Charlie', 'David'],
    'dept': ['HR', 'IT', 'IT', 'HR'],
    'salary': [50000, 70000, 80000, 45000]
}
df = pd.DataFrame(data)
# Filter rows where dept is IT and salary is >= 70000, keep only name and salary columns
filtered_df = df.loc[(df['dept'] == 'IT') & (df['salary'] >= 70000), ['name', 'salary']]
print(filtered_df)

Visual Flow

The following execution flow represents the step-by-step evaluation inside the interpreter:

DataFrame → Apply conditional check → Filter rows → Select columns → Return subset DataFrame

Common Mistakes

Review these common pitfalls when working with this topic:

• Confusing loc (label-based) with iloc (integer position-based).
• Using python's logical operators (and, or) instead of pandas bitwise operators (&, |) when filtering.
• Modifying sliced DataFrames directly, raising a SettingWithCopyWarning.
• Forgetting brackets when specifying multiple filter criteria.

Best Practices

Enforce these Pythonic best practices in your codebase:

it_employees = df[df['dept'] == 'IT']
it_employees['salary'] = 90000 # Raises SettingWithCopyWarning

df.loc[df['dept'] == 'IT', 'salary'] = 90000 # Safe modification using .loc

Performance Notes

Keep these optimization guidelines in mind for performance-sensitive hotpaths:

• Slicing and filtering return views. To modify a subset safely, copy it explicitly: df.copy().

Quick Revision

Use these key summaries for last-minute revision:

• Use df[column] or df[[col1, col2]] to select columns.
• loc uses index labels; iloc uses integer positions.
• Filter rows using boolean conditions inside brackets: df[condition].
• Use & (and) and | (or) for multi-criteria filters.
• Wrap filtering conditions in parentheses: (cond1) & (cond2).