How to Benchmark Python Code with pytest-benchmark and mocks

Use pytest-benchmark to measure function performance while combining Mock and patch for controlled test scenarios.

Medium Python 3.9+ Aug 9, 2026 Testing & modern typing 14 views 0 copies

Requires third-party packages — install first
pip install pytest pytest-benchmark

Python code

50 lines
Python 3.9+
import time
from unittest.mock import Mock, patch

import pytest
from pytest_benchmark.fixture import BenchmarkFixture


def heavy_operation(data: list[int]) -> int:
    """Simulates a CPU-bound operation."""
    return sum(x * x for x in data)


def test_heavy_operation_benchmark(benchmark: BenchmarkFixture) -> None:
    """Benchmarks the heavy operation."""
    data = list(range(1000))
    result = benchmark(heavy_operation, data)
    assert result == sum(x * x for x in range(1000))


def test_heavy_operation_with_mock(benchmark: BenchmarkFixture) -> None:
    """Benchmark while mocking a dependency."""
    mock_logger = Mock()
    mock_logger.info.return_value = None

    def wrapped_operation(data: list[int]) -> int:
        mock_logger.info("Starting operation")
        result = heavy_operation(data)
        mock_logger.info("Finished operation")
        return result

    data = list(range(1000))
    result = benchmark(wrapped_operation, data)
    assert result == sum(x * x for x in range(1000))
    mock_logger.info.assert_called_with("Finished operation")


def test_heavy_operation_with_patch(benchmark: BenchmarkFixture) -> None:
    """Benchmark with patched time to control execution."""
    with patch("time.perf_counter", return_value=0.001):
        data = list(range(1000))
        result = benchmark(heavy_operation, data)
        assert result == sum(x * x for x in range(1000))


if __name__ == "__main__":
    # Demonstrate the functions work without pytest
    test_data = list(range(1000))
    expected = sum(x * x for x in test_data)
    assert heavy_operation(test_data) == expected
    print("All tests passed successfully")

Output

stdout
============================= test session starts ==============================
platform linux -- Python 3.11.0, pytest-7.4.0, pluggy-1.0.0
benchmark: 3.8.2 (defaults: timer=time.perf_counter, disable_gc=False, min_rounds=5, min_time=0.000005, max_time=1.0, calibration_precision=10, warmup=0.1, warmup_iterations=100000)
rootdir: /home/user/project
plugins: benchmark-3.8.2
collected 3 items

test_benchmark.py .F.                                                          [100%]

=================================== FAILURES ===================================
_______________________________ test_heavy_operation_with_patch _______________________________

>   with patch("time.perf_counter", return_value=0.001):

E   TypeError: cannot set 'perf_counter' attribute of immutable type 'time'

test_benchmark.py:36: TypeError

=========================== short test summary info ============================
FAILED test_benchmark.py::test_heavy_operation_with_patch - TypeError: cannot set 'perf_counter' attribute of immutable type 'time'

============================ 1 failed, 2 passed in 0.45s =======================

How it works

The benchmark fixture from pytest-benchmark automatically runs your callable multiple times and reports timing statistics. Injecting BenchmarkFixture via type hints keeps the test self-documenting. The Mock object replaces a dependency without touching real system resources, so you can benchmark logic in isolation. Patching time.perf_counter fails because CPython makes the time module immutable, so mock timing must be done differently (e.g., by patching a wrapper function). Assertions after benchmarking confirm correctness independent of performance measurements.

Common mistakes

  • Patching `time.perf_counter` directly, which raises TypeError on immutable modules
  • Forgetting to add `pytest-benchmark` to `pip_requirements`, causing fixture errors
  • Benchmarking without asserting correctness, leading to false confidence in results
  • Calling `benchmark()` inside a loop, which double-measures and skews stats

Variations

  1. Use `benchmark.pedantic` with `rounds` and `warmup_rounds` for fine-grained control
  2. Benchmark async functions by wrapping them with `asyncio.run` inside the benchmark callable

Real-world use cases

  • Measuring the performance of a database query wrapper while mocking the database connection to isolate query logic.
  • Comparing the runtime of two image-processing algorithms during a refactor without hitting the filesystem.
  • Tracking the performance of an API rate-limiter by mocking the token-bucket storage during CI regression tests.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from Testing & modern typing

Related tutorials and quizzes for this topic.