CPU Profiling Python with perf

Getting Real CPU Profiles in Python Without the Bloat

Ever run into a situation where your Python code is slow, but you can't figure out why? You try cProfile, but it feels like running through molasses. Or maybe you're working with a production server where you can't install fancy profiling tools. The good news is, there's a simpler way using a tool that's probably already on your Linux machine: perf.

When I first started profiling Python code at PythonSkillset, I kept hitting walls with standard profilers. They'd add overhead that made measurements unreliable, or they'd miss kernel-level activity entirely. That's when I discovered using perf directly with Python, and honestly, it changed how I debug performance issues.

Let me walk you through what works, what doesn't, and how to get meaningful CPU profiles without the headache.

Why perf Instead of Built-in Profilers

Standard Python profilers like cProfile work fine for function-level timing, but they have a dirty little secret: they add significant overhead. Each function call gets instrumented, which can distort timing measurements, especially for fast functions.

perf, on the other hand, uses hardware performance counters from your CPU. It samples what the CPU is actually doing, not what Python thinks it's doing. This means:

• Almost zero overhead when profiling (1-3% typically)
• Captures kernel time, including system calls and I/O wait
• Works on production systems without code changes
• Shows you hotspots other profilers miss entirely

Getting Started with perf and Python

First, make sure you have perf installed:

sudo apt-get install linux-tools-common linux-tools-generic linux-tools-$(uname -r)

For the best results with Python, you'll want to enable frame pointer support. Without it, perf won't see the Python call stack, which makes the output useless. Add this to your Python invocation:

perf record -g -- python3 your_script.py

The -g flag enables call graph recording. This captures not just where the CPU spends time, but the full call stack at each sample point.

The Perl Maps Trick That Changed Everything

Pure Python code shows up fine with the basic approach, but once you use C extensions (NumPy, pandas, cryptography libraries), perf gets lost. It sees the C code but can't map it back to your Python lines.

Here's what PythonSkillset found works reliably: use perf-map-agent for JIT-compiled code or simply run with PYTHONPERFSUPPORT:

PYTHONPERFSUPPORT=1 perf record -g -F 99 python3 your_script.py

For Python 3.12+, there's native support using perf_jitdump. It requires compiling Python with --enable-perf-profiling flag, but most distributions now include it by default.

Reading the Output Without Going Crazy

After running your script, you'll get a perf.data file. Here's how to make sense of it:

perf report --stdio --sort=comm,dso,symbol

This shows you a nice table sorted by how much CPU time each function consumed. For Python code, look for patterns like:

• PyEval_EvalFrameDefault — means you're spending time in the interpreter loop itself, usually due to pure Python computation
• _PyObject_GenericGetAttr — indicates attribute lookups, often a sign of deep object hierarchies
• Specific C extension functions — shows where your native code hotspots are

Real Example: The Slowness Nobody Saw Coming

At PythonSkillset, we had a data processing pipeline that was crawling. cProfile showed perfectly balanced function times — no obvious bottleneck. Running perf record revealed something else entirely:

# Samples: 23K of event 'cycles:ppp'
# Event count (approx.): 1849717600
#
# Overhead  Command   Shared Object
# ........  .......  .................
    35.12%  python3  libc-2.31.so       
    12.45%  python3  [kernel.kallsyms]
     8.23%  python3  mymodule.cpython-312-x86_64-linux-gnu.so

Wait, libc taking 35% of CPU? That's a red flag. Further inspection showed the bottleneck was actually repeated realloc calls from Python's list growth pattern. A single list.extend() replacement with pre-allocating the list length cut runtime by 60%.

The best part? cProfile never showed this because realloc happens at the C library level, invisible to Python's profiler.

Tips That Actually Work

Sample rate matters. Use -F 99 (99 Hz) for most Python workloads. Too fast and you're collecting noise. Too slow and you miss short-lived functions.

Profile specific sections. Don't profile your whole script, especially imports:

perf record -g -- python3 -c "
import your_module
import time

time.sleep(2)  # Skip initialization
perf_event_begin = open('/proc/self/perf_event_open', 'w')
# ... your actual code
"

Watch out for inlining. Python's C extensions often inline functions, making them invisible to perf. If you see [unknown] entries, try perf report --demangle.

When perf Won't Work

It's not perfect. Virtual machines and containers sometimes block perf's access to hardware counters. In Docker, you need --privileged or specific seccomp profiles. AWS Lambda? Forget it — you'll need different tools entirely.

For macOS users, dtrace or Instruments are your friends, but they work differently.

The Bottom Line

If you're doing serious CPU profiling of Python code and haven't tried perf, you're probably missing half the story. The overhead is negligible, the insights are real, and it works on production systems without modifying your code.

Start simple: run perf record -g -- python3 your_script.py and see what surprises pop up. Most PythonSkillset team members were shocked the first time they saw how much time their code spent in system calls they never knew were happening.

Performance debugging shouldn't feel like guesswork. Perf gives you the actual numbers. What you do with them is up to you.