PYTHON Reference Guide

Revision Time: 5 mins

Pandas Essentials Reference

Core Pandas operations for ETL workflows — reading, exploring, cleaning, transforming, and writing data.

pd.read_csv()

Return: DataFrame

Load a CSV file into a Pandas DataFrame.

Used In: Reading raw CSV files in ETL pipelines.

Syntax signature:pd.read_csv(filepath, usecols=None, dtype=None, chunksize=None)

Code snippet:

python

import pandas as pd, io
csv = 'name,dept,salary\nAlice,DE,90000\nBob,DS,75000\nCarol,DE,85000'
df = pd.read_csv(io.StringIO(csv))
print(df.shape)

Expected Output:(3, 3)

Time Complexity: O(N)

Common Mistakes: Not specifying dtype — pandas infers types on every read, which is slow for large files.

Related Methods: pd.read_json(), pd.read_parquet()

Remember: Use usecols=['col1'] to load only needed columns — saves memory significantly on large files.

pd.read_json()

Return: DataFrame

Load a JSON file or JSON string into a Pandas DataFrame.

Used In: API response processing, loading JSON exports.

Syntax signature:pd.read_json(path_or_buf, orient=None)

Code snippet:

python

import pandas as pd
json_data = '[{"name":"Alice","dept":"DE"},{"name":"Bob","dept":"DS"}]'
df = pd.read_json(json_data)
print(df.columns.tolist())

Expected Output:['name', 'dept']

Time Complexity: O(N)

Related Methods: pd.read_csv()

Remember: Use orient='records' for list-of-dicts JSON and orient='index' for dict-of-dicts.

head() / tail() / info() / describe()

Return: DataFrame | None

Quick inspection of a DataFrame's contents, schema, and statistics.

Used In: Data exploration, schema validation, EDA.

Syntax signature:df.head(n=5) / df.tail(n=5) / df.info() / df.describe()

Code snippet:

python

import pandas as pd, io
csv = 'name,dept,salary\nAlice,DE,90000\nBob,DS,75000'
df = pd.read_csv(io.StringIO(csv))
print(df.head(1))
print(df.shape)

Expected Output:

# print(df.head(1))
   name dept  salary
0  Alice   DE   90000

# print(df.shape)
(2, 3)

Time Complexity: O(N)

Related Methods: df.dtypes, df.columns

Remember: Always run df.info() first — it reveals null counts and incorrect dtypes immediately.

loc[] / iloc[]

Return: DataFrame | Series

Select rows and columns by label (loc) or integer position (iloc).

Used In: Row sampling, column extraction, conditional slicing.

Syntax signature:df.loc[row_labels, col_labels] / df.iloc[row_idx, col_idx]

Code snippet:

python

import pandas as pd, io
csv = 'name,dept,salary\nAlice,DE,90000\nBob,DS,75000'
df = pd.read_csv(io.StringIO(csv))
print(df.loc[0, 'name'])      # label-based
print(df.iloc[1, 2])          # position-based

Expected Output:

# print(df.loc[0, 'name'])
Alice

# print(df.iloc[1, 2])
75000

Time Complexity: O(1) for single element, O(N) for slice

Common Mistakes: Confusing loc[] (label-based) and iloc[] (position-based) — they produce different results on non-integer indexes.

Related Methods: df.at[], df.iat[]

Comparison:

loc uses labels; iloc uses integer positions. With default int index they behave the same but diverge with custom indexes.

Remember: Use loc[] for column name access and iloc[] when you know the exact row/column positions.

Filtering

Return: DataFrame

Select rows matching a Boolean condition.

Used In: Row filtering in ETL, quality gates.

Syntax signature:df[df['col'] condition] or df.query('col condition')

Code snippet:

python

import pandas as pd, io
csv = 'name,dept,salary\nAlice,DE,90000\nBob,DS,75000\nCarol,DE,85000'
df = pd.read_csv(io.StringIO(csv))
result = df[(df['dept'] == 'DE') & (df['salary'] > 85000)]
print(result['name'].tolist())

Expected Output:['Alice']

Time Complexity: O(N)

Common Mistakes: Using 'and' instead of '&' between conditions raises ValueError.

Remember: Use & for AND, | for OR — not Python's and/or which don't work on Series.

sort_values()

Return: DataFrame

Sort a DataFrame by one or more columns.

Used In: Ranking records, ordering output exports.

Syntax signature:df.sort_values(by, ascending=True, na_position='last')

Code snippet:

python

import pandas as pd, io
csv = 'name,salary\nAlice,90000\nBob,75000\nCarol,85000'
df = pd.read_csv(io.StringIO(csv))
print(df.sort_values('salary', ascending=False)['name'].tolist())

Expected Output:['Alice', 'Carol', 'Bob']

Time Complexity: O(N log N)

Remember: Use by=['col1', 'col2'] to sort by multiple columns — secondary sort breaks ties.

drop_duplicates() / fillna() / dropna()

Return: DataFrame

Remove duplicates, fill missing values, or drop rows with NaN.

Used In: Data cleaning before aggregations.

Syntax signature:df.drop_duplicates(subset=None) / df.fillna(value) / df.dropna(subset=None)

Code snippet:

python

import pandas as pd
df = pd.DataFrame({'id': [1, 1, 2], 'val': [10, 10, None]})
clean = df.drop_duplicates().fillna(0)
print(clean.values.tolist())

Expected Output:[[1.0, 10.0], [2.0, 0.0]]

Time Complexity: O(N)

Related Methods: df.isna(), df.notna()

Remember: dropna() removes entire rows with any NaN by default — use subset=['col'] to target specific columns.

rename() / astype()

Return: DataFrame | Series

Rename columns or cast column data types.

Used In: Schema normalization, type enforcement.

Syntax signature:df.rename(columns={'old': 'new'}) / df['col'].astype(dtype)

Code snippet:

python

import pandas as pd, io
csv = 'emp_id,sal_amt\n1,90000\n2,75000'
df = pd.read_csv(io.StringIO(csv))
df = df.rename(columns={'emp_id': 'id', 'sal_amt': 'salary'})
df['salary'] = df['salary'].astype(float)
print(df.dtypes['salary'])

Expected Output:float64

Time Complexity: O(N)

Remember: Always rename columns early in the pipeline to use clean, consistent names throughout.

apply()

Return: Series | DataFrame

Apply a function element-wise to a column or row-wise across columns.

Used In: Custom derivations, conditional column creation.

Syntax signature:df['col'].apply(func) / df.apply(func, axis=1)

Code snippet:

python

import pandas as pd
df = pd.DataFrame({'salary': [90000, 75000, 85000]})
df['band'] = df['salary'].apply(lambda s: 'senior' if s >= 85000 else 'junior')
print(df['band'].tolist())

Expected Output:['senior', 'junior', 'senior']

Time Complexity: O(N) — slower than vectorized ops

Common Mistakes: Using apply() for simple arithmetic — use vectorized operations instead for 10-100x speedup.

Remember: apply() is slower than vectorized operations — prefer df['col'] * 2 over apply(lambda x: x*2) for math.

groupby() + agg()

Return: DataFrame

Group rows by key columns and compute aggregate statistics.

Used In: Departmental summaries, reporting aggregations.

Syntax signature:df.groupby(keys).agg({'col': 'func'})

Code snippet:

python

import pandas as pd, io
csv = 'name,dept,salary\nAlice,DE,90000\nBob,DS,75000\nCarol,DE,85000'
df = pd.read_csv(io.StringIO(csv))
result = df.groupby('dept')['salary'].agg(['mean', 'count'])
print(result.to_dict())

Expected Output:{'mean': {'DE': 87500.0, 'DS': 75000.0}, 'count': {'DE': 2, 'DS': 1}}

Time Complexity: O(N log N)

Related Methods: groupby().transform(), groupby().filter()

Remember: Pass a dict to agg() to apply different functions to different columns in one pass.

merge() / concat()

Return: DataFrame

Combine DataFrames by join key (merge) or stacking vertically/horizontally (concat).

Used In: Joining dimension tables, stacking batch files.

Syntax signature:pd.merge(df1, df2, on='key', how='inner') / pd.concat([df1, df2])

Code snippet:

python

import pandas as pd
emps = pd.DataFrame({'id': [1, 2], 'name': ['Alice', 'Bob']})
depts = pd.DataFrame({'id': [1, 2], 'dept': ['DE', 'DS']})
result = pd.merge(emps, depts, on='id')
print(result.columns.tolist())

Expected Output:['id', 'name', 'dept']

Time Complexity: O(N log N) for merge

Related Methods: df.join()

Remember: how='left' keeps all rows from the left DataFrame — equivalent to a SQL LEFT JOIN.

to_csv()

Return: None | str

Write a DataFrame to a CSV file.

Used In: Exporting cleaned data, writing ETL output files.

Syntax signature:df.to_csv(path, index=False, encoding='utf-8')

Code snippet:

python

import pandas as pd, io
df = pd.DataFrame({'name': ['Alice'], 'dept': ['DE']})
buf = io.StringIO()
df.to_csv(buf, index=False)
print(buf.getvalue())

Expected Output:

name,dept
Alice,DE

Time Complexity: O(N)

Common Mistakes: Forgetting index=False — produces an extra unnamed column in the output CSV.

Remember: Always use index=False unless the index itself carries meaning — default True adds an unwanted row number column.

Previous TopicNumPy Essentials