Profile Python Memory Usage with memory_profiler

You have probably spent hours optimizing your Python code for speed, but have you ever thought about how much memory it actually eats up? Memory leaks can slow your application to a crawl and crash your system when you least expect it. The good news? Python has a neat tool called memory_profiler that helps you track exactly what is happening with your memory, line by line.

Why memory profiling matters

When I started working on PythonSkillset's data processing pipeline, I noticed our server would crash after processing large datasets. The code was fast but memory hungry. We were loading entire files when we could have processed them in chunks. Without a memory profiler, you are essentially debugging blind. You might fix performance issues while ignoring the real memory problem.

Installing memory_profiler

Getting started is straightforward. Open your terminal and install it with pip:

pip install memory_profiler

You might also want matplotlib if you plan to generate memory usage plots:

pip install matplotlib

That is all the setup you need. No complex configuration files or dependencies that break your environment.

Using the @profile decorator

The easiest way to profile a function is adding the @profile decorator. Here is how it works:

from memory_profiler import profile

@profile
def process_data():
    data = [i for i in range(1000000)]
    processed = [x * 2 for x in data]
    return processed

if __name__ == "__main__":
    result = process_data()

Run this script with:

python -m memory_profiler your_script.py

You will see output like this:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     3     42.1 MiB     42.1 MiB           1   @profile
     4                                         def process_data():
     5     67.5 MiB     25.4 MiB           1       data = [i for i in range(1000000)]
     6     85.3 MiB     17.8 MiB           1       processed = [x * 2 for x in data]
     7     85.3 MiB      0.0 MiB           1       return processed

Look at line 5 - creating that list took 25.4 MiB. That is significant. You can see exactly which part of your code consumes memory.

Profiling memory over time

Sometimes you need to see how memory changes during execution, not just per line. Use the memory_usage function for that:

from memory_profiler import memory_usage
import time

def heavy_function():
    big_list = []
    for i in range(100000):
        big_list.append('x' * 1000)
        time.sleep(0.001)
    return big_list

mem_usage = memory_usage((heavy_function,), interval=0.1, timeout=None)
print(f"Peak memory: {max(mem_usage):.2f} MiB")
print(f"Baseline: {mem_usage[0]:.2f} MiB")

This gives you a list of memory readings at 0.1 second intervals. You can plot this data to see patterns.

Comparing memory usage between functions

A practical use case at PythonSkillset was comparing two ways to read a file. We had a function that loaded everything into memory versus one that streamed:

@profile
def load_all():
    with open('large_file.txt', 'r') as f:
        data = f.readlines()
    return data

@profile
def stream_lines():
    data = []
    with open('large_file.txt', 'r') as f:
        for line in f:
            data.append(line.strip())
    return data

The stream_lines function used 3 times less memory because it did not load the entire file at once. Without profiling, we would have stuck with the slower but not obviously problematic approach.

Plotting memory usage

Visualizing memory helps you spot leaks and spikes quickly. Here is how:

from memory_profiler import memory_usage
import matplotlib.pyplot as plt

def leaky_function():
    container = []
    for i in range(1000):
        container.append('x' * 10000)
        if i % 100 == 0:
            print(f"Step {i}: memory usage still climbing")
    return container

mem, retval = memory_usage((leaky_function,), retval=True, interval=0.05)
plt.plot(mem)
plt.ylabel('Memory (MiB)')
plt.xlabel('Time (0.05s intervals)')
plt.title('Memory Usage Over Time')
plt.show()

The plot shows you if memory keeps climbing without leveling off - a clear sign of a leak.

Practical tips from PythonSkillset

After months of profiling production code, here are the lessons we learned:

1. Always profile before optimizing - You might fix something that is not broken
2. Use @profile only on suspect functions - Decorating everything slows your script
3. Test with realistic data sizes - Small datasets hide memory issues
4. Watch for incremental growth in loops - That is where leaks hide

One common trap with memory_profiler is that it adds overhead. Your profiled code runs slower, but the memory numbers are accurate. Do not use it in production - only during development.

Wrapping up

Memory profiling with memory_profiler takes maybe five minutes to set up but saves hours of debugging. You get clear numbers showing exactly where your memory goes, plus visual plots for spotting trends. Next time your Python application feels sluggish or crashes mysteriously, run it through memory_profiler first. The problem might be a forgotten list that swallowed your RAM.

Give it a try with one of your scripts today. You might be surprised what you find hiding in those innocent looking list comprehensions.