Track Memory Leaks With Python's tracemalloc

Tracking Down Memory Leaks With Python's tracemalloc

If you've ever had a Python program start fast and then slow to a crawl after running for a few hours, you've likely run into memory issues. Maybe your web app starts throwing MemoryError after processing a thousand requests, or your data processing script grows bigger by the minute. I've been there, and trust me, it's frustrating.

The good news is that Python comes with a built-in tool called tracemalloc that helps you figure out exactly where your memory is going. I use it regularly at PythonSkillset when debugging memory problems in production services, and it's saved me hours of head-scratching.

What is tracemalloc, really?

tracemalloc is part of Python's standard library since version 3.4. What makes it special is that it traces every memory allocation in your program, including which line of code made that allocation. It's like having a detailed receipt for every byte of memory your program uses.

Think of it as a GPS tracker for your memory. When something's taking too much space, tracemalloc can tell you exactly which function called which function called which line of code that created that big list or dictionary eating up your RAM.

Getting started with basic tracking

The first thing you need to do is enable tracemalloc at the start of your script. It's a single line:

import tracemalloc

tracemalloc.start()

After that, you can take snapshots of memory usage at different points and compare them. Here's a practical example I often use at PythonSkillset when investigating memory growth:

import tracemalloc

tracemalloc.start()

# Do some work, process some data
my_list = [i for i in range(1000000)]

# Take a snapshot
snapshot1 = tracemalloc.take_snapshot()

# Do more work that might leak memory
for _ in range(100):
    result = process_heavy_data()

# Take another snapshot
snapshot2 = tracemalloc.take_snapshot()

# Compare them
stats = snapshot2.compare_to(snapshot1, 'lineno')
for stat in stats[:10]:
    print(stat)

This comparison shows you the top 10 places where memory increased between the two snapshots. You'll see file names, line numbers, and the size of allocations.

Finding the biggest memory consumers

Sometimes you just want to know what's using all the memory right now. tracemalloc can give you a snapshot of the current state:

snapshot = tracemalloc.take_snapshot()
stats = snapshot.statistics('lineno')

# Show top 10 memory consumers
for stat in stats[:10]:
    print(f"{stat.count} blocks, {stat.size / 1024 / 1024:.1f} MB total")
    print(f"    at {stat.traceback}")

The stat.traceback shows the full call stack that led to that allocation. This is incredibly useful because many memory issues come from code deep in your call chain, not just the obvious places.

Real-world example from PythonSkillset

Let me share a situation I ran into while working on a data processing pipeline at PythonSkillset. We had a service that read CSV files, processed every row, and then stored the results. After about 50,000 rows, it would slow down significantly and eventually crash.

I added tracemalloc tracking to capture a snapshot every 10,000 rows:

import tracemalloc

tracemalloc.start()

processed_count = 0
snapshot_before = tracemalloc.take_snapshot()

for row in csv_reader:
    processed_data = heavy_processing(row)
    processed_count += 1

    if processed_count % 10000 == 0:
        snapshot_after = tracemalloc.take_snapshot()
        diff = snapshot_after.compare_to(snapshot_before, 'lineno')

        for stat in diff[:5]:
            print(f"Memory change: {stat.size_diff / 1024:.2f} KB - {stat.traceback}")

        snapshot_before = snapshot_after

The output revealed that one function in our validation module was creating a new dictionary for every row but never clearing it. It was accumulating all processed data in a global cache variable that should have been temporary. The fix was a simple del statement, but tracemalloc showed us exactly where to look.

Filtering out noise

When you're dealing with large applications, tracemalloc can produce a lot of information. Most of it will be Python's internal allocations or framework code you don't care about. You can filter by specific files or directories:

# Only look at your application's code
import os
app_dir = os.path.dirname(__file__)

stats = snapshot.statistics('lineno')
app_stats = [stat for stat in stats if app_dir in str(stat.traceback)]

Or if you're looking at specific modules:

stats = snapshot.statistics('lineno')
# Filter for traces that include 'my_module.py'
filtered = [stat for stat in stats if 'my_module.py' in str(stat.traceback)]

Performance considerations

You might be wondering: does tracemalloc slow down my program? Yes, it does add overhead. In my experience at PythonSkillset, enabling tracemalloc causes about 10-30% slowdown in CPU-intensive code, and memory usage increases by about 2-5%. For debugging purposes this is perfectly acceptable. Just don't leave it running in production unless you're actively debugging.

To minimize impact, you can enable and disable it around specific code sections:

tracemalloc.start()
# run the code you want to monitor
tracemalloc.stop()
# run the rest normally

When it doesn't help

tracemalloc won't solve every memory problem. It can't track memory allocated by C extensions (like numpy or pandas in some cases) because those go through different allocation mechanisms. It also won't help you find memory fragmentation issues directly.

For those cases, you'll want to pair tracemalloc with other tools like memory_profiler or system-level profiling. But for 90% of Python memory issues, tracemalloc alone is enough.

A practical pattern I use

Here's a pattern I've settled on at PythonSkillset for debugging memory issues in long-running services:

import tracemalloc
import logging

# Enable with a callback to check memory periodically
tracemalloc.start()

def check_memory_usage():
    snapshot = tracemalloc.take_snapshot()
    stats = snapshot.statistics('lineno')
    total_size = sum(stat.size for stat in stats)

    if total_size > 500 * 1024 * 1024:  # 500 MB
        logging.warning(f"Memory usage exceeded 500 MB: {total_size / 1024 / 1024:.1f} MB")
        for stat in stats[:5]:
            logging.warning(f"Top consumer: {stat.size / 1024:.1f} KB at {stat.traceback}")
        # Maybe trigger garbage collection or restart

This keeps an eye on memory and alerts you before things go wrong.

Final thoughts

tracemalloc is one of those tools that's incredibly powerful once you get used to it. The comparison of snapshots is what makes it special — you're not just seeing a flat list of allocations, you're seeing how memory changes over time. That's the key to finding leaks.

Next time you notice your Python program getting sluggish after running for a while, remember you have this built-in detective ready to go. Enable it, take some snapshots, and let it show you exactly where your memory went.