NumPy for Data Engineers

Overview

Apply NumPy to load logs files, clean numeric columns, and calculate features in data pipelines.

Learning Objectives

• Load raw tabular data from text streams into arrays.
• Filter outliers and null data using boolean index masks.
• Preprocess numeric datasets efficiently.

Concept Explanation

Data Engineers use NumPy to clean raw numeric inputs before loading them into database storage. You can parse text lines using np.genfromtxt(), filter out invalid rows using boolean indexing filters, and execute vectorized calculations.

Code Examples

Example 1 — Basics

This example introduces the fundamental syntax and concepts.

import numpy as np
# Ingest simple measurement points list
data = np.array([12, 15, 18])

Example 2 — Everyday Usage

This example demonstrates a realistic scenario handling business parameters.

# Preprocess measurement arrays to remove noise
readings = np.array([23.5, -999.0, 24.0, 1000.0, 22.8]) # Outliers present
# Filter values between 0 and 100 using bitwise logical checks
valid_mask = (readings >= 0) & (readings <= 100)
clean_readings = readings[valid_mask]
print('Cleaned readings:', clean_readings)

Example 3 — Practical Data Engineering Example

This example shows clean, production-grade code structure following senior development standards.

# Simple NumPy ETL pipeline processing raw CSV string
raw_csv = """1,4.5,100
2,-1.0,150
3,4.0,120
"""
# Load column 1 (ratings) and column 2 (reviews counts)
data_table = np.genfromtxt(raw_csv.splitlines(), delimiter=',')
ratings = data_table[:, 1]
reviews = data_table[:, 2]

# Clean missing values (-1.0 represents missing)
ratings_clean = np.where(ratings == -1.0, 3.0, ratings)
# Feature generation: weighted scores
weighted_scores = ratings_clean * (reviews / 100)
print('Cleaned ratings:', ratings_clean)
print('Weighted scores:', weighted_scores)

Visual Flow

The following execution flow represents the step-by-step evaluation inside the interpreter:

Ingest raw CSV rows → Slice rating & reviews column arrays → Clean nulls → Calculate features

Common Mistakes

Review these common pitfalls when working with this topic:

• Using slow loops to process file rows.
• Using standard python words 'and' and 'or' inside boolean indices (always use & and |).
• Forgetting to wrap comparison statements in parentheses when filtering arrays.
• Not handling NaN values (np.nan) which propagate and ruin math values.

Best Practices

Enforce these Pythonic best practices in your codebase:

filtered = arr[arr > 5 and arr < 20] # Raises ValueError

filtered = arr[(arr > 5) & (arr < 20)] # Bitwise operator with parentheses

Performance Notes

Keep these optimization guidelines in mind for performance-sensitive hotpaths:

• Vectorized boolean filtering avoids python loop overhead, allowing fast in-memory array filtering.

Quick Revision

Use these key summaries for last-minute revision:

• genfromtxt() parses text tables into arrays.
• Boolean indexing filters arrays without loops.
• Use bitwise & and | inside conditional index statements.
• Wrap individual indexing comparisons in parentheses.
• Vectorized preprocessing runs in O(N) time.