Python Closures and Late-Binding
Master Python closures and cell variables. Understand the classic loop late-binding trap with __closure__ cell inspection and default parameter fixes.
See the About page for curation and verification details.
Understanding closures and variable binding is one of the clearest dividing lines between junior and senior Python developers. In technical problem solving, questions involving nested functions, lambda expressions created inside loops, and decorators frequently trip learners up because CPython resolves free variables at function execution time, not at function definition time. This comprehensive guide walks through lexical scoping mechanics, inspects internal CPython cell objects using __closure__, breaks down the classic loop late-binding bug, and provides three common fixes.
What a Closure Actually Is in Python
A closure is a function object that retains bindings to free variables defined in its enclosing lexical scope, even after that enclosing scope has finished execution and returned. For a closure to exist in Python, two core lexical conditions must be met: 1. There must be a nested function (a function defined inside another function). 2. The nested function must reference at least one variable defined in the enclosing function (a 'free variable'). While returning the nested function is the standard pattern in factories and decorators, returning it is not an architectural requirement for closure creation: passing the nested function to an event handler, storing it in a dictionary, or registering it as a callback immediately creates and retains the closure binding.
def make_counter(start=0):
count = start
def increment(step=1):
nonlocal count
count += step
return count
return increment
counter = make_counter(10)
print('Call 1:', counter(2))
print('Call 2:', counter(3))
print('Enclosed count cell:', counter.__closure__[0].cell_contents)Verified output on Python 3.12: Call 1: 12 Call 2: 15 Enclosed count cell: 15
Notice that even though make_counter finished running and its local frame popped off the call stack, counter still retains access to count. CPython accomplishes this by creating a cell object that outlives the stack frame.
Inspecting the Internal CPython Cell Object with __closure__
When Python compiles a function containing free variables, it stores references to those variables in an attribute named __closure__. This attribute is a tuple of cell objects. Each cell object has a single attribute, cell_contents, pointing directly to the value in memory.
def outer(val):
msg = f'Value is {val}'
def inner():
return msg
return inner
f = outer(42)
print('Is closure tuple:', isinstance(f.__closure__, tuple))
print('Cell contents:', f.__closure__[0].cell_contents)Verified output: Is closure tuple: True Cell contents: Value is 42
If a nested function does NOT reference any variables from its enclosing scope, __closure__ is None. A closure only exists when free variables are captured.
The Classic Late-Binding Loop Bug Traced
A classic closure bug in Python involves generating a list of functions or lambdas inside a for loop. Consider this canonical code snippet:
funcs = [lambda: i for i in range(3)]
results = [f() for f in funcs]
print('Results:', results)Verified output: Results: [2, 2, 2]
Most junior learners expect [0, 1, 2], but Python returns [2, 2, 2]. Let us trace exactly why this happens in CPython:
| Step | Action Taken | Loop Variable i | Func 0 Closure Cell | Func 1 Closure Cell | Func 2 Closure Cell |
|---|---|---|---|---|---|
| 1 | i=0, lambda 0 created | 0 | Points to cell of i | Unbound | Unbound |
| 2 | i=1, lambda 1 created | 1 | Points to cell of i | Points to cell of i | Unbound |
| 3 | i=2, lambda 2 created | 2 | Points to cell of i | Points to cell of i | Points to cell of i |
| 4 | Loop terminates | 2 | Cell contents: 2 | Cell contents: 2 | Cell contents: 2 |
| 5 | funcs[0]() invoked | 2 | Reads cell (2) | Reads cell (2) | Reads cell (2) |
| 6 | funcs[1]() invoked | 2 | Reads cell (2) | Reads cell (2) | Reads cell (2) |
| 7 | funcs[2]() invoked | 2 | Reads cell (2) | Reads cell (2) | Reads cell (2) |
Crucial Insight: All three lambdas are created in the same scope and bind to the SAME variable name i. CPython does not copy the value of i at lambda definition time. Instead, all three lambdas point to the identical cell object. When the lambdas are later called in step 5, the loop has completed and i has final value 2. Every lambda looks up i in the shared cell and evaluates to 2.
Three Common Fixes for Late Binding
Solution 1: Default Argument Binding (The idiomatic quick fix)
Default arguments are evaluated once at function definition time. By binding i=i as a default argument, Python evaluates the current value of i immediately and binds it into the lambda's local default argument tuple:
fixed_funcs = [lambda i=i: i for i in range(3)]
print('Fixed results:', [f() for f in fixed_funcs])
print('Closure of fixed func:', fixed_funcs[0].__closure__)Verified output: Fixed results: [0, 1, 2] Closure of fixed func: None
Notice that fixed_funcs[0].__closure__ is None! Because i is now a local argument with a default value, it is no longer a free variable, so no closure cell is needed.
Solution 2: Factory Function Pattern Wrapping the lambda creation in a helper function creates a fresh local scope for each iteration:
def make_getter(val):
return lambda: val
factory_funcs = [make_getter(i) for i in range(3)]
print('Factory results:', [f() for f in factory_funcs])
print('Cell 0:', factory_funcs[0].__closure__[0].cell_contents)
print('Cell 1:', factory_funcs[1].__closure__[0].cell_contents)Verified output: Factory results: [0, 1, 2] Cell 0: 0 Cell 1: 1
Here, each call to make_getter allocates a separate local variable val in its own stack frame, producing distinct closure cells for each function.
Solution 3: functools.partial
functools.partial freezes function arguments at instantiation time, avoiding lambda closures entirely.
from functools import partial
def return_val(x):
return x
partial_funcs = [partial(return_val, i) for i in range(3)]
print('Partial results:', [f() for f in partial_funcs])Verified output: Partial results: [0, 1, 2]
Bytecode Mechanics: LOAD_DEREF vs LOAD_FAST in CPython
CPython handles local variables and closure variables using distinct bytecode instructions. Disassembling a closure using Python's dis module reveals the exact underlying mechanics:
import dis
def sample():
x = 10
def inner():
return x
return inner
dis.dis(sample)Verified bytecode output from CPython 3.12 dis.dis(sample): 0 MAKE_CELL 1 (x) 2 RESUME 0 4 LOAD_CONST 1 (10) 6 STORE_DEREF 1 (x) 8 LOAD_CLOSURE 1 (x) 10 BUILD_TUPLE 1 12 LOAD_CONST 2 (<code object inner>) 14 MAKE_FUNCTION 8 (closure) 16 STORE_FAST 0 (inner) 18 LOAD_FAST 0 (inner) 20 RETURN_VALUE
Notice the instruction STORE_DEREF instead of standard STORE_FAST. While standard local variables live in fast C array registers indexed by integers, variables captured by closures are wrapped in PyCellObject pointers, which requires LOAD_DEREF dereferencing at runtime.
Decorators as Closures: Tracing Wrappers Step by Step
Every function decorator in Python is fundamentally a closure pattern. When you apply @my_decorator to a function definition, Python passes the original function object into the decorator and binds the returned wrapper closure in its place. The wrapper function forms a closure over the original function reference.
from functools import wraps
def count_invocations(fn):
invocations = 0
@wraps(fn)
def wrapper(*args, **kwargs):
nonlocal invocations
invocations += 1
print(f'Execution #{invocations} of {fn.__name__}')
return fn(*args, **kwargs)
return wrapper
@count_invocations
def calculate_area(width, height):
return width * height
print('Area 1:', calculate_area(5, 4))
print('Area 2:', calculate_area(3, 7))Verified output: Execution #1 of calculate_area Area 1: 20 Execution #2 of calculate_area Area 2: 21
Notice that invocations is retained across calls because wrapper maintains a closure cell pointing to the mutable integer variable in count_invocations.
Four Common Closure Pitfalls to Avoid
- Forgetting the
nonlocalkeyword when modifying state: Reassigning an outer variable (count = count + 1) inside a nested function without declaringnonlocal countraisesUnboundLocalError: cannot access local variable 'count' where it is not associated with a value. - Unintended memory retention: Closures keep referenced objects alive. If a closure references a large dictionary or database connection from an enclosing scope, that object cannot be garbage collected as long as the closure function exists.
- Assuming default arguments create closures: Default argument evaluation happens once at definition time and stores values in
__defaults__, not in__closure__. - Modifying mutable closure cells concurrently: If multiple threads invoke a closure that modifies a shared cell object without locking, logical race conditions will cause lost updates and invalid application state (though CPython's GIL prevents C-level interpreter corruption).
Practice Exercises & Code Challenges
Question 1:
What is the output of this nested multiplier snippet?
def multiplier_table():
return [lambda x, factor=f: x * factor for f in [2, 3, 4]]
funcs = multiplier_table()
print([f(10) for f in funcs])Walkthrough: The default argument factor=f binds the current value of f at each iteration: 2, 3, and 4. When invoked with x = 10, the calculations are 10*2=20, 10*3=30, and 10*4=40. Output: [20, 30, 40].
Question 2:
What is the difference between global and nonlocal in Python closures?
Solution: global instructs Python to look up and rebind variables at the module/top-level namespace. nonlocal instructs Python to look up and rebind variables in the nearest enclosing non-global scope (the parent function frame).
Question 3:
How does CPython decide whether a variable should be stored in fastlocals (registers) versus a cell object?
Solution: The Python compiler performs lexical symbol table analysis (symtable) during the AST compilation pass. Normal local variables are assigned fixed array slots in co_varnames accessed via fast register instructions (LOAD_FAST and STORE_FAST, enabled when the code object has the CO_OPTIMIZED flag). However, if the compiler identifies that a local variable is referenced as a free variable inside an enclosed nested function, it places that symbol into the outer function's co_cellvars and the inner function's co_freevars. At runtime, CPython allocates a PyCellObject pointer cell on the heap via MAKE_CELL, allowing the inner function to access it via LOAD_DEREF even after the outer stack frame has deallocated.
Question 4:
Can closures be serialized using standard pickle in Python?
Solution: By default, standard pickle cannot serialize closures containing free variables because function serialization requires top-level named module references. Serializing closures in distributed task queues (such as Celery or PySpark) requires third-party picklers like dill or cloudpickle that inspect and serialize the __closure__ cell tuple.
Practice closure problem sets in our Decorators & Closures Practice track and explore our guide on The Mutable Default Arguments Trap.