Review core memory models, GIL restrictions, decorators, generators, collections, gotchas, and Data Engineering best practices.
in) on a list takes O(N) linear time, while on a set or dict it takes O(1) constant time, which dramatically impacts the speed of data deduplication pipelines.# Unpacking with asterisk operator
first, *middle, last = [10, 20, 30, 40]
# Slicing: [start:stop:step] (Step -1 reverses)
rev_str = "DataPlay"[::-1]
# Zip multiple streams
keys = ["name", "role"]
vals = ["Alex", "admin"]
user = dict(zip(keys, vals))
| Collection | Lookup | Membership | Append/Add | Delete | Pop | Iteration |
|---|---|---|---|---|---|---|
| List | O(1) | O(N) | O(1) | O(N) | O(1) / O(N) | O(N) |
| Tuple | O(1) | O(N) | N/A | N/A | N/A | O(N) |
| Set | N/A | O(1) | O(1) | O(1) | O(1) | O(N) |
| Dict | O(1) | O(1) | O(1) | O(1) | O(1) | O(N) |
| Deque | O(N) | O(N) | O(1) | O(N) | O(1) | O(N) |
Key Takeaways: Always prefer sets/dicts over lists for membership checking (in) to query in O(1) average time instead of O(N) linear time.
Remember: Using list.pop(0) inside queues is a major anti-pattern. This shifts all items in memory, causing O(N) complexity; always use collections.deque.popleft() instead which runs in O(1) time.
class User:
# Pre-allocates attribute storage, saving RAM
__slots__ = ("name", "_role")
def __init__(self, name, role):
self.name = name
self._role = role
@property
def role(self):
return self._role
import functools
def logger(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
print("Calling", func.__name__)
return func(*args, **kwargs)
return wrapper
Key Takeaways: __slots__ reduces memory footprint by preventing the creation of __dict__ for each object instance. It is critical for memory optimization when instantiating millions of small records in memory.
Remember: Decorators wrap functions using closures. Always decorate the wrapper function with @functools.wraps(func) to preserve original function metadata (docstring, function name).
Key Takeaways: Multi-threading in CPython does not achieve true parallelism for CPU-heavy tasks. Use the multiprocessing module to bypass the GIL and utilize multiple cores.
Remember: Reference counting runs in O(1) time and is deterministic. Generational GC sweeps are non-deterministic and scan objects in generational cycles.
== checks value equality (calls __eq__), whereas is checks object identity (same memory address). Small integers (-5 to 256) are cached, but larger integers or identical lists will fail is.
Default args are evaluated once at definition time. def append(val, lst=[]) shares the same list across calls. Use lst=None and initialize inside the body.
copy() duplicates the outer structure but shares nested object references. Modifying a nested list in a shallow copy changes the original. Use copy.deepcopy() instead.
Multiplying a nested list copies references to the same inner list. Modifying one cell updates all rows. Use list comprehensions: [[0]*3 for _ in range(3)].
Closures bind variables by reference, not value. Lambdas created inside loops look up variables at call time, yielding duplicate values. Bind at definition: lambda x, i=i: i * x.
Adding or removing items while iterating shifts indices, causing elements to be skipped or looped twice. Iterate over a copy (for x in lst[:]) or use list comprehensions.
Key Takeaways: Avoid mutable types as function defaults or multiplier targets to prevent shared memory references.
Remember: Never mutate the dimensions of a list/collection while iterating directly over it; filter with comprehensions or iterate over a copy.
# Stream large log files line-by-line
def stream_log_lines(file_path):
with open(file_path, "r", encoding="utf-8") as f:
for line in f:
yield line.strip()
# Chunked processing in Pandas
import pandas as pd
for chunk in pd.read_csv("large.csv", chunksize=10000):
process(chunk)
Generators use yield to suspend execution and return data lazily. This keeps memory consumption constant (O(1)) regardless of the dataset size, unlike list-based structures.
Prefer native streaming solutions (csv.reader, json.JSONDecoder) or chunked buffers over loading entire document trees into memory.
NumPy vectors and Pandas DataFrames perform vector calculations via highly optimized compiled C binaries. Avoid looping through rows (.iterrows()) and use vector operations instead.
Key Takeaways: Avoid loading whole files or records using read() or to_list() in production data pipelines. Prefer generators or chunked streaming.
Remember: Pandas loops are slow. Vector operations run up to 100x faster than iteration by leveraging compiled contiguous memory operations.
Select a common Python backend developer scenario to view senior response keypoints:
Eager list loading inside functions reads the entire file into memory instead of lazy generators streaming.
Reading the entire file into a list or memory buffer at once.
Generators and lazy evaluation stream data on-demand, preventing memory bloating.
== compares values; is compares object identity (memory addresses).yield inside an O(1) memory bounds limit.copy.deepcopy() recursively duplicates nested data structures, preventing reference sharing.dict.get(key, default) safely handles missing keys instead of throwing errors.enumerate() yields index-value pairs cleanly without tracking variables.with open(...) guarantees file descriptors close automatically on exit or exception.