Python Memory, GC & Profiling
Understanding object copying is fundamental to preventing silent data corruption:
- Reference Assignment (
b = a): Zero copying. Both names point to the exact same memory address (id(a) == id(b)). - Shallow Copy (
copy.copy(a)ora.copy()): Creates a new outer container, but child elements are still references to original nested objects! - Deep Copy (
copy.deepcopy(a)): Recursively clones the outer container AND all nested child lists/dictionaries into completely independent memory objects!
import copy
# Nested data structure:
original = [[1, 2], [3, 4]]
shallow = copy.copy(original)
deep = copy.deepcopy(original)
# Modify nested element in original list:
original[0][0] = 999
print("Original list after mutation: ", original) # [[999, 2], [3, 4]]
print("Shallow copy (INNER CORRUPTED!):", shallow) # [[999, 2], [3, 4]] (Shares inner sublist!)
print("Deep copy (SAFE & ISOLATED!): ", deep) # [[1, 2], [3, 4]] (Completely untouched!)Always use copy.deepcopy() when cloning complex nested structures (like state dictionaries, game boards, or JSON configs) to prevent unintended side effects.
In standard CPython, every object is represented by a C struct called PyObject containing:
ob_refcnt: The Reference Count tracking how many variables point to this object.ob_type: Pointer to the object's data type descriptor.- Value payload.
Immediate Deallocation: As soon as an object's reference count drops to 0 (e.g. when a variable goes out of scope or del x is executed), CPython immediately reclaims the memory on the spot!
import sys
# 1. Create a fresh list object:
sample = [1, 2, 3]
# sys.getrefcount() returns current reference count (includes temporary call ref):
print("Reference count for 'sample':", sys.getrefcount(sample) - 1) # 1
# 2. Add another reference pointer:
alias_ptr = sample
print("After creating 'alias_ptr': ", sys.getrefcount(sample) - 1) # 2
# 3. Delete one reference:
del alias_ptr
print("After 'del alias_ptr': ", sys.getrefcount(sample) - 1) # 1We subtract 1 because passing sample as an argument to sys.getrefcount() temporarily increases the reference count by 1 during the function call.
Reference counting alone has a major fatal flaw: Circular References (Object A points to Object B, and Object B points to Object A). Even if you delete both external variables, their reference count never drops to 0!
To resolve this, CPython includes a Cyclic Generational Garbage Collector (GC) that runs in the background:
- Generation 0 (Youngest): Newly allocated objects. Collected very frequently.
- Generation 1: Objects surviving Gen 0 collections.
- Generation 2 (Oldest): Long-lived objects surviving Gen 1. Collected rarely.
import gc
# 1. Create a circular reference cycle:
class Node:
def __init__(self, name):
self.name = name
self.partner = None
node1 = Node("A")
node2 = Node("B")
node1.partner = node2 # node1 points to node2
node2.partner = node1 # node2 points to node1 (CYCLE!)
# Delete external variables:
del node1
del node2
# 2. Force manual garbage collection cycle to detect and destroy isolated cycles:
unreachable_objects = gc.collect()
print(f"🧹 Garbage Collector cleaned up {unreachable_objects} circular reference objects!")To prevent circular reference leaks in trees and caches, use the standard library weakref module, which creates non-owning reference pointers that do not increment ob_refcnt.
Never guess where performance bottlenecks lie — profile and measure them!
timeit: Precision micro-benchmarking tool for comparing short code snippets over thousands of runs.cProfile: Deterministic profiler that counts every function call, time per call, and cumulative bottlenecks across an entire application.
import timeit
# Microbenchmark: List Comprehension vs map() for string conversion:
time_comp = timeit.timeit('[str(x) for x in range(1000)]', number=10_000)
time_map = timeit.timeit('list(map(str, range(1000)))', number=10_000)
print(f"⚡ List Comprehension (10k runs): {time_comp:.4f} seconds")
print(f"⚡ map(str, ...) (10k runs): {time_map:.4f} seconds")
# Pythonic Design Philosophy: EAFP vs LBYL
# EAFP: "Easier to Ask for Forgiveness than Permission" (try-except) -> The Python Way!
# LBYL: "Look Before You Leap" (if-else checks)
user_profile = {"name": "Balaji"}
# The Pythonic EAFP pattern:
try:
email = user_profile["email"]
except KeyError:
email = "default@example.com"
print("User Email (EAFP):", email)EAFP avoids redundant lookups (checking if "email" in user_profile and then accessing user_profile["email"] requires TWO hash lookups; EAFP performs only ONE!).
"del obj" merely decrements the object's reference count and deletes the variable name from the local namespace. If other references exist, or if CPython's memory allocator (PyMalloc) pools the memory, the RAM is retained for future Python allocations rather than returned to the OS.
Use timeit to benchmark whether checking membership in a set ("999 in my_set") is faster than in a list ("999 in my_list") with 1,000 numbers.
import timeit
setup_code = """
my_list = list(range(1000))
my_set = set(range(1000))
"""
t_list = timeit.timeit('999 in my_list', setup=setup_code, number=100_000)
t_set = timeit.timeit('999 in my_set', setup=setup_code, number=100_000)
print(f"List 'in' check (100k runs): {t_list:.5f} sec")
print(f"Set 'in' check (100k runs): {t_set:.5f} sec")
print(f"🚀 Set is {t_list/t_set:.1f}x FASTER due to O(1) hash lookup!")Q What is the Global Interpreter Lock (GIL) in CPython?
The GIL is a mutex lock in CPython that ensures only one native CPU thread executes Python bytecode at a time, protecting CPython's reference count memory management from race conditions.
Q What is the difference between EAFP and LBYL?
LBYL (Look Before You Leap) tests preconditions with if-statements before executing. EAFP (Easier to Ask for Forgiveness than Permission) assumes valid state and catches exceptions with try-except, which is faster for the common success path in Python.
Q Why are small integers (-5 to 256) cached in CPython memory?
CPython pre-allocates an internal global array for integers between -5 and 256 because they are used constantly for loop counters and indexing, saving millions of allocation CPU cycles.