How to Benchmark Python Code with pytest-benchmark and mocks
Use pytest-benchmark to measure function performance while combining Mock and patch for controlled test scenarios.
pip install pytest pytest-benchmark
Python code
50 linesimport time
from unittest.mock import Mock, patch
import pytest
from pytest_benchmark.fixture import BenchmarkFixture
def heavy_operation(data: list[int]) -> int:
"""Simulates a CPU-bound operation."""
return sum(x * x for x in data)
def test_heavy_operation_benchmark(benchmark: BenchmarkFixture) -> None:
"""Benchmarks the heavy operation."""
data = list(range(1000))
result = benchmark(heavy_operation, data)
assert result == sum(x * x for x in range(1000))
def test_heavy_operation_with_mock(benchmark: BenchmarkFixture) -> None:
"""Benchmark while mocking a dependency."""
mock_logger = Mock()
mock_logger.info.return_value = None
def wrapped_operation(data: list[int]) -> int:
mock_logger.info("Starting operation")
result = heavy_operation(data)
mock_logger.info("Finished operation")
return result
data = list(range(1000))
result = benchmark(wrapped_operation, data)
assert result == sum(x * x for x in range(1000))
mock_logger.info.assert_called_with("Finished operation")
def test_heavy_operation_with_patch(benchmark: BenchmarkFixture) -> None:
"""Benchmark with patched time to control execution."""
with patch("time.perf_counter", return_value=0.001):
data = list(range(1000))
result = benchmark(heavy_operation, data)
assert result == sum(x * x for x in range(1000))
if __name__ == "__main__":
# Demonstrate the functions work without pytest
test_data = list(range(1000))
expected = sum(x * x for x in test_data)
assert heavy_operation(test_data) == expected
print("All tests passed successfully")
Output
============================= test session starts ==============================
platform linux -- Python 3.11.0, pytest-7.4.0, pluggy-1.0.0
benchmark: 3.8.2 (defaults: timer=time.perf_counter, disable_gc=False, min_rounds=5, min_time=0.000005, max_time=1.0, calibration_precision=10, warmup=0.1, warmup_iterations=100000)
rootdir: /home/user/project
plugins: benchmark-3.8.2
collected 3 items
test_benchmark.py .F. [100%]
=================================== FAILURES ===================================
_______________________________ test_heavy_operation_with_patch _______________________________
> with patch("time.perf_counter", return_value=0.001):
E TypeError: cannot set 'perf_counter' attribute of immutable type 'time'
test_benchmark.py:36: TypeError
=========================== short test summary info ============================
FAILED test_benchmark.py::test_heavy_operation_with_patch - TypeError: cannot set 'perf_counter' attribute of immutable type 'time'
============================ 1 failed, 2 passed in 0.45s =======================
How it works
The benchmark fixture from pytest-benchmark automatically runs your callable multiple times and reports timing statistics. Injecting BenchmarkFixture via type hints keeps the test self-documenting. The Mock object replaces a dependency without touching real system resources, so you can benchmark logic in isolation. Patching time.perf_counter fails because CPython makes the time module immutable, so mock timing must be done differently (e.g., by patching a wrapper function). Assertions after benchmarking confirm correctness independent of performance measurements.
Common mistakes
- Patching `time.perf_counter` directly, which raises TypeError on immutable modules
- Forgetting to add `pytest-benchmark` to `pip_requirements`, causing fixture errors
- Benchmarking without asserting correctness, leading to false confidence in results
- Calling `benchmark()` inside a loop, which double-measures and skews stats
Variations
- Use `benchmark.pedantic` with `rounds` and `warmup_rounds` for fine-grained control
- Benchmark async functions by wrapping them with `asyncio.run` inside the benchmark callable
Real-world use cases
- Measuring the performance of a database query wrapper while mocking the database connection to isolate query logic.
- Comparing the runtime of two image-processing algorithms during a refactor without hitting the filesystem.
- Tracking the performance of an API rate-limiter by mocking the token-bucket storage during CI regression tests.
Sponsored
More from Testing & modern typing
- Capture stdout and stderr with pytest capsys easy
- Characterization Test for Legacy Python Code medium
- Dataclass with Type Hints Fields in Python easy
- Dependency Injection in Python for Testability easy
- Design Data Helpers with Python TypedDict and Literal easy
- Fix and Test a Regression Bug in Python with Unit Tests easy
Keep learning
Related tutorials and quizzes for this topic.