PyCodeItPython trace & SQL practice
python

Python Closures and Late-Binding

Master Python closures and cell variables. Understand the classic loop late-binding trap with __closure__ cell inspection and default parameter fixes.

RD

By RichieDatalyst

Python & SQL learning guides-GitHub-PyCodeIt on LinkedIn

Published · Updated

9 min read

See the About page for curation and verification details.

Understanding closures and variable binding is one of the clearest dividing lines between junior and senior Python developers. In technical problem solving, questions involving nested functions, lambda expressions created inside loops, and decorators frequently trip learners up because CPython resolves free variables at function execution time, not at function definition time. This comprehensive guide walks through lexical scoping mechanics, inspects internal CPython cell objects using __closure__, breaks down the classic loop late-binding bug, and provides three common fixes.

What a Closure Actually Is in Python

A closure is a function object that retains bindings to free variables defined in its enclosing lexical scope, even after that enclosing scope has finished execution and returned. For a closure to exist in Python, two core lexical conditions must be met: 1. There must be a nested function (a function defined inside another function). 2. The nested function must reference at least one variable defined in the enclosing function (a 'free variable'). While returning the nested function is the standard pattern in factories and decorators, returning it is not an architectural requirement for closure creation: passing the nested function to an event handler, storing it in a dictionary, or registering it as a callback immediately creates and retains the closure binding.

def make_counter(start=0):
    count = start
    def increment(step=1):
        nonlocal count
        count += step
        return count
    return increment

counter = make_counter(10)
print('Call 1:', counter(2))
print('Call 2:', counter(3))
print('Enclosed count cell:', counter.__closure__[0].cell_contents)
Runtime Execution Output
Verified output on Python 3.12:
Call 1: 12
Call 2: 15
Enclosed count cell: 15

Notice that even though make_counter finished running and its local frame popped off the call stack, counter still retains access to count. CPython accomplishes this by creating a cell object that outlives the stack frame.

Inspecting the Internal CPython Cell Object with __closure__

When Python compiles a function containing free variables, it stores references to those variables in an attribute named __closure__. This attribute is a tuple of cell objects. Each cell object has a single attribute, cell_contents, pointing directly to the value in memory.

def outer(val):
    msg = f'Value is {val}'
    def inner():
        return msg
    return inner

f = outer(42)
print('Is closure tuple:', isinstance(f.__closure__, tuple))
print('Cell contents:', f.__closure__[0].cell_contents)
Runtime Execution Output
Verified output:
Is closure tuple: True
Cell contents: Value is 42

If a nested function does NOT reference any variables from its enclosing scope, __closure__ is None. A closure only exists when free variables are captured.

The Classic Late-Binding Loop Bug Traced

A classic closure bug in Python involves generating a list of functions or lambdas inside a for loop. Consider this canonical code snippet:

funcs = [lambda: i for i in range(3)]
results = [f() for f in funcs]
print('Results:', results)
Runtime Execution Output
Verified output:
Results: [2, 2, 2]

Most junior learners expect [0, 1, 2], but Python returns [2, 2, 2]. Let us trace exactly why this happens in CPython:

StepAction TakenLoop Variable iFunc 0 Closure CellFunc 1 Closure CellFunc 2 Closure Cell
1i=0, lambda 0 created0Points to cell of iUnboundUnbound
2i=1, lambda 1 created1Points to cell of iPoints to cell of iUnbound
3i=2, lambda 2 created2Points to cell of iPoints to cell of iPoints to cell of i
4Loop terminates2Cell contents: 2Cell contents: 2Cell contents: 2
5funcs[0]() invoked2Reads cell (2)Reads cell (2)Reads cell (2)
6funcs[1]() invoked2Reads cell (2)Reads cell (2)Reads cell (2)
7funcs[2]() invoked2Reads cell (2)Reads cell (2)Reads cell (2)

Crucial Insight: All three lambdas are created in the same scope and bind to the SAME variable name i. CPython does not copy the value of i at lambda definition time. Instead, all three lambdas point to the identical cell object. When the lambdas are later called in step 5, the loop has completed and i has final value 2. Every lambda looks up i in the shared cell and evaluates to 2.

Three Common Fixes for Late Binding

Solution 1: Default Argument Binding (The idiomatic quick fix) Default arguments are evaluated once at function definition time. By binding i=i as a default argument, Python evaluates the current value of i immediately and binds it into the lambda's local default argument tuple:

fixed_funcs = [lambda i=i: i for i in range(3)]
print('Fixed results:', [f() for f in fixed_funcs])
print('Closure of fixed func:', fixed_funcs[0].__closure__)
Runtime Execution Output
Verified output:
Fixed results: [0, 1, 2]
Closure of fixed func: None

Notice that fixed_funcs[0].__closure__ is None! Because i is now a local argument with a default value, it is no longer a free variable, so no closure cell is needed.

Solution 2: Factory Function Pattern Wrapping the lambda creation in a helper function creates a fresh local scope for each iteration:

def make_getter(val):
    return lambda: val

factory_funcs = [make_getter(i) for i in range(3)]
print('Factory results:', [f() for f in factory_funcs])
print('Cell 0:', factory_funcs[0].__closure__[0].cell_contents)
print('Cell 1:', factory_funcs[1].__closure__[0].cell_contents)
Runtime Execution Output
Verified output:
Factory results: [0, 1, 2]
Cell 0: 0
Cell 1: 1

Here, each call to make_getter allocates a separate local variable val in its own stack frame, producing distinct closure cells for each function.

Solution 3: functools.partial functools.partial freezes function arguments at instantiation time, avoiding lambda closures entirely.

from functools import partial
def return_val(x):
    return x

partial_funcs = [partial(return_val, i) for i in range(3)]
print('Partial results:', [f() for f in partial_funcs])
Runtime Execution Output
Verified output:
Partial results: [0, 1, 2]

Bytecode Mechanics: LOAD_DEREF vs LOAD_FAST in CPython

CPython handles local variables and closure variables using distinct bytecode instructions. Disassembling a closure using Python's dis module reveals the exact underlying mechanics:

import dis
def sample():
    x = 10
    def inner():
        return x
    return inner

dis.dis(sample)
Runtime Execution Output
Verified bytecode output from CPython 3.12 dis.dis(sample):
  0 MAKE_CELL                1 (x)
  2 RESUME                   0
  4 LOAD_CONST               1 (10)
  6 STORE_DEREF              1 (x)
  8 LOAD_CLOSURE             1 (x)
 10 BUILD_TUPLE              1
 12 LOAD_CONST               2 (<code object inner>)
 14 MAKE_FUNCTION            8 (closure)
 16 STORE_FAST               0 (inner)
 18 LOAD_FAST                0 (inner)
 20 RETURN_VALUE

Notice the instruction STORE_DEREF instead of standard STORE_FAST. While standard local variables live in fast C array registers indexed by integers, variables captured by closures are wrapped in PyCellObject pointers, which requires LOAD_DEREF dereferencing at runtime.

Decorators as Closures: Tracing Wrappers Step by Step

Every function decorator in Python is fundamentally a closure pattern. When you apply @my_decorator to a function definition, Python passes the original function object into the decorator and binds the returned wrapper closure in its place. The wrapper function forms a closure over the original function reference.

from functools import wraps
def count_invocations(fn):
    invocations = 0
    @wraps(fn)
    def wrapper(*args, **kwargs):
        nonlocal invocations
        invocations += 1
        print(f'Execution #{invocations} of {fn.__name__}')
        return fn(*args, **kwargs)
    return wrapper

@count_invocations
def calculate_area(width, height):
    return width * height

print('Area 1:', calculate_area(5, 4))
print('Area 2:', calculate_area(3, 7))
Runtime Execution Output
Verified output:
Execution #1 of calculate_area
Area 1: 20
Execution #2 of calculate_area
Area 2: 21

Notice that invocations is retained across calls because wrapper maintains a closure cell pointing to the mutable integer variable in count_invocations.

Four Common Closure Pitfalls to Avoid

  1. Forgetting the nonlocal keyword when modifying state: Reassigning an outer variable (count = count + 1) inside a nested function without declaring nonlocal count raises UnboundLocalError: cannot access local variable 'count' where it is not associated with a value.
  2. Unintended memory retention: Closures keep referenced objects alive. If a closure references a large dictionary or database connection from an enclosing scope, that object cannot be garbage collected as long as the closure function exists.
  3. Assuming default arguments create closures: Default argument evaluation happens once at definition time and stores values in __defaults__, not in __closure__.
  4. Modifying mutable closure cells concurrently: If multiple threads invoke a closure that modifies a shared cell object without locking, logical race conditions will cause lost updates and invalid application state (though CPython's GIL prevents C-level interpreter corruption).

Practice Exercises & Code Challenges

Question 1:

What is the output of this nested multiplier snippet?

def multiplier_table():
    return [lambda x, factor=f: x * factor for f in [2, 3, 4]]

funcs = multiplier_table()
print([f(10) for f in funcs])

Walkthrough: The default argument factor=f binds the current value of f at each iteration: 2, 3, and 4. When invoked with x = 10, the calculations are 10*2=20, 10*3=30, and 10*4=40. Output: [20, 30, 40].

Question 2:

What is the difference between global and nonlocal in Python closures? Solution: global instructs Python to look up and rebind variables at the module/top-level namespace. nonlocal instructs Python to look up and rebind variables in the nearest enclosing non-global scope (the parent function frame).

Question 3:

How does CPython decide whether a variable should be stored in fastlocals (registers) versus a cell object? Solution: The Python compiler performs lexical symbol table analysis (symtable) during the AST compilation pass. Normal local variables are assigned fixed array slots in co_varnames accessed via fast register instructions (LOAD_FAST and STORE_FAST, enabled when the code object has the CO_OPTIMIZED flag). However, if the compiler identifies that a local variable is referenced as a free variable inside an enclosed nested function, it places that symbol into the outer function's co_cellvars and the inner function's co_freevars. At runtime, CPython allocates a PyCellObject pointer cell on the heap via MAKE_CELL, allowing the inner function to access it via LOAD_DEREF even after the outer stack frame has deallocated.

Question 4:

Can closures be serialized using standard pickle in Python? Solution: By default, standard pickle cannot serialize closures containing free variables because function serialization requires top-level named module references. Serializing closures in distributed task queues (such as Celery or PySpark) requires third-party picklers like dill or cloudpickle that inspect and serialize the __closure__ cell tuple.

Practice closure problem sets in our Decorators & Closures Practice track and explore our guide on The Mutable Default Arguments Trap.

Related Python Tutorials