Debug Python Memory Leaks with tracemalloc

How to Debug Python Memory Leaks with tracemalloc

You've probably seen it happen—your Python script starts fine, but after running for a few hours, memory usage climbs and climbs. The system slows to a crawl. Eventually, your process crashes with an OutOfMemoryError.

Memory leaks in Python are real. Even with garbage collection, objects can stay referenced without you realizing it. I've wasted days tracking these down manually. The built-in tracemalloc module? That's the tool I wish I knew sooner.

What Is tracemalloc?

tracemalloc is a standard Python library (since Python 3.4) that tracks every memory allocation in your program. It doesn't just tell you how much memory you're using—it tells you exactly which line of code allocated that memory. That's gold for debugging leaks.

Here's the key insight: memory leaks in Python usually happen because objects persist in a list, dict, cache, or closure when they should have been freed. tracemalloc shows you the stack trace of where those objects were created.

Getting Started

First, you need to enable tracemalloc in your code:

import tracemalloc

tracemalloc.start()

That's it. Now your program will track every object allocation. The overhead is small—around 10-30% slowdown depending on your code—but for debugging a leak, it's worth it.

Taking a Snapshot

To figure out what's using memory, take memory snapshots at two different points in your program:

snapshot1 = tracemalloc.take_snapshot()
# ... let your program run some operations ...
snapshot2 = tracemalloc.take_snapshot()

Then compare them:

stats = snapshot2.compare_to(snapshot1, 'lineno')
for stat in stats[:10]:
    print(stat)

This prints the top 10 allocations that grew between the two snapshots, grouped by line number. You'll see something like:

/Users/pythonskillset/app/database.py:87: size=12.3 MiB (+10.1 MiB), count=2100 (+1900), average=6.0 KiB
/Users/pythonskillset/app/cache.py:42: size=5.2 MiB (+4.8 MiB), count=950 (+850), average=5.5 KiB

That "size" with the plus sign? That's the memory that leaked between snapshot1 and snapshot2. Now you know exactly which files and lines are the problem.

Finding the Actual Memory Users

Once you've identified the leaky lines, you can drill down further. Use traceback to see the full stack trace of those allocations:

for stat in stats[:3]:
    print(f"{stat.count_diff} new objects: {stat.traceback.format(limit=5)}")

This shows you the call stack that created those leaked objects. You'll see the entire chain of function calls that led to the allocation.

Real World Example

At PythonSkillset, we had a web scraper that grew from 50MB to 2GB over 6 hours. Using tracemalloc, we found the culprit: a result caching decorator that never expired keys. Each scraped page added a dictionary to an internal _cache list. The fix was adding a max size and eviction policy.

# Before (leaky)
result_cache = []
def cached_fetch(url):
    result = fetch(url)
    result_cache.append(result)
    return result

# After (fixed)
from collections import OrderedDict
result_cache = OrderedDict()
MAX_CACHE = 100
def cached_fetch(url):
    if url in result_cache:
        result_cache.move_to_end(url)
        return result_cache[url]
    result = fetch(url)
    result_cache[url] = result
    if len(result_cache) > MAX_CACHE:
        result_cache.popitem(last=False)
    return result

We reduced memory usage from 2GB to 150MB. And we found the issue in about 15 minutes with tracemalloc.

Common Leaky Patterns to Watch For

• Global lists and dicts that grow indefinitely (caches, user sessions, logs)
• Circular references with __del__ methods (garbage collector can't collect them)
• Closures that capture large temporary variables
• Thread local storage that isn't cleaned up after thread exits
• Database result sets held open without iterating fully

Using tracemalloc as a Context Manager

You don't have to run tracemalloc for your whole program. You can wrap specific sections:

with tracemalloc.Trace():
    # code you suspect is leaking
    pass

This gives you allocation stats only for that block. Much cleaner.

When to Use tracemalloc vs Other Tools

• tracemalloc: Best for finding exactly which line of code allocated leaked memory. Low overhead. Standard library.
• memory_profiler: Best for seeing memory usage over time (line by line). Useful for understanding memory spikes.
• objgraph: Best for visualizing object references and finding circular references.
• gc module: Best for checking what objects exist and if they're collectable.

Start with tracemalloc. It's the fastest path from memory problem to fix.

Final Tip

Set up a memory test in your CI pipeline. Run your code with tracemalloc, take snapshots before and after a test run, and assert that memory doesn't grow by more than some threshold. That catches leaks before they reach production.

A memory leak caught at commit time is worth a hundred late-night debugging sessions.