Security Regression Tests with Pytest

Your app passed the security review today, but what about six months from now? The vulnerability you fixed last sprint can silently creep back in with a single refactor, a dependency update, or a team member who doesn't know the history. That is the pain this lesson solves: writing automated security regression tests with pytest that lock in fixes and catch reintroductions before they reach production. By the end of this lesson, you'll be able to build a small but powerful safety net that runs on every commit.

The problem this lesson solves

Manual security testing is a snapshot. You check for SQL injection today, find it, patch it, and move on. But code evolves — query strings change, authentication logic is refactored, and libraries get upgraded. Each change can silently undo a security fix, and the cost of discovering that in production is devastating: data breaches, outage pages, and late-night rollbacks.

Traditional unit tests usually cover functionality, not security properties. They assert that a function returns the right value, not that it rejects malicious input. Security regression tests fill that gap: they are automated tests that encode known vulnerabilities and verify the fix stays fixed. Without them, every patch is a gamble.

Consider a real-world example: a developer fixes a path traversal bug in a file download endpoint. Three weeks later, someone optimizes the path handling and accidentally reintroduces the traversal. If a regression test exists, it fails instantly. If not, the vulnerability ships to customers. This lesson gives you the practical tools to prevent that scenario.

Core concept / mental model

Think of a security regression test as a guard dog for a specific vulnerability. You know what the attack looks like, so you teach the test to bark when the attack succeeds. The mental model is simple: an attack that once worked must never work again.

Definition: A security regression test is an automated test that sends a known malicious payload to a function or endpoint and asserts that the system rejects it. It is not a test for new vulnerabilities — it's a test that old ones stay dead.

Analogy: Imagine you're a castle builder. After an intruder breaks in through a crack in the wall, you patch the crack and also station a guard there permanently. The guard's job is to sound the alarm if anyone tries to exploit that same crack. Your code is the castle, the patch is the fix, and the regression test is the guard.

Key properties:

• Specific — targets one known vulnerability, not a class of them
• Automated — runs in CI, no human intervention
• Deterministic — same input always produces the same result
• Fast — takes milliseconds, so you can run them often

A good security regression test makes the implicit security requirement explicit. It documents the vulnerability in code: anyone reading the test knows what was exploited and what the fix must prevent.

How it works step by step

Writing a security regression test follows a repeatable process. Here's the step-by-step logic, from identifying the vulnerability to running the test in CI.

Step 1: Identify the vulnerability and fix

Start with a security report, a code review, or a penetration test result. You need a concrete attack vector — for example, SQL injection via a search field — and the commit that fixed it. The fix might be parameterized queries, input validation, or output encoding.

Action: Isolate the vulnerable function or module. You'll test that exact unit, not the entire application.

Step 2: Determine the malicious input

What payload would have exploited the vulnerability? Use the same payload the attacker used or a variation that proves the attack is blocked. For SQL injection, that's something like ' OR 1=1 --. For XSS, it's <script>alert(1)</script>.

Step 3: Write the test structure

Create a test file (e.g., test_security.py) and write a function that calls the target with the malicious input. The test should assert that the result is the safe outcome: a sanitized item, an empty result, or an exception.

Step 4: Run and verify

Run the test before and after the fix to confirm it fails on vulnerable code and passes on patched code. This is called a red-green test.

Step 5: Integrate into CI

Add the test file to your regular test suite so it runs with every commit. Use pytest markers or separate tests/security/ directory to make it easy to run security tests alone.

Hands-on walkthrough

Let's implement a security regression test with pytest. We'll use a simple function that is vulnerable to SQL injection, then fix it and write the regression test.

First, the vulnerable function:

# app.py
def search_users(search_term: str, connection):
    query = f"SELECT * FROM users WHERE username = '{search_term}'"
    return connection.execute(query).fetchall()

The flaw: unsanitized input is directly interpolated into SQL. Now the fixed version:

# app_fixed.py
import sqlite3

def search_users_safe(search_term: str, connection: sqlite3.Connection):
    query = "SELECT * FROM users WHERE username = ?"
    return connection.execute(query, (search_term,)).fetchall()

Now write the security regression test:

# test_security.py
import sqlite3
from app_fixed import search_users_safe

def test_search_users_rejects_sql_injection():
    conn = sqlite3.connect(":memory:")
    conn.execute("CREATE TABLE users (username TEXT)")
    conn.execute("INSERT INTO users (username) VALUES ('alice')")

    # The attack payload
    malicious_input = "' OR 1=1 --"

    # Run the safe function
    results = search_users_safe(malicious_input, conn)

    # Assert that no rows are returned (the attack failed)
    assert results == []

Run it:

$ pytest test_security.py

Expected output:

1 passed in 0.02s

Now try running the same test against the vulnerable function (just to see it fail):

# test_security_vulnerable.py
from app import search_users
import sqlite3

def test_vulnerable_function_fails():
    conn = sqlite3.connect(":memory:")
    conn.execute("CREATE TABLE users (username TEXT)")
    conn.execute("INSERT INTO users (username) VALUES ('alice')")

    malicious_input = "' OR 1=1 --"
    results = search_users(malicious_input, conn)

    # This asserts the vulnerability still exists, so it will pass on vulnerable code
    assert len(results) > 0

This test passes on the vulnerable code and fails on the fixed code — proving the test actually detects the regression.

Pro tip: Always run your regression test against the vulnerable version at least once. If it doesn't fail there, it's not a valid regression test.

Compare options / when to choose what

When writing security regression tests, you have several options for test design and tooling. Here's a comparison table:

When to choose what:

• Use unit-level regression tests for most security fixes — they're quick and precise.
• Use E2E tests for critical endpoints (login, file upload) when you need to verify the full request/response flow.
• Use property-based testing for anything that takes untrusted input and must sanitize it robustly.
• Static analysis complements regression tests but doesn't replace them — it can't prove a fix stays fixed.

Troubleshooting & edge cases

Test passes even when the vulnerability is reintroduced

If a test doesn't fail on vulnerable code, it's not asserting the right thing. Fix: Review the assertion. Make sure it checks the result of the attack, not just that the function doesn't crash. For example, assert no rows are returned, not just that execute() didn't raise an exception.

False positives due to environment differences

Tests that depend on network, environment variables, or external services can fail randomly. Fix: Use mocks and fixtures. For database calls, use an in-memory SQLite database as shown above. For HTTP calls, use responses or pytest-mock.

Non-deterministic payloads

Random payloads (e.g., from faker) can make tests flaky. Fix: Use fixed, known payloads that specifically target the vulnerability. Save them as constants in the test file.

Slow tests

If security tests hit the disk or network, they'll slow down CI. Fix: Keep them in a separate directory (e.g., tests/security/) and mark them with @pytest.mark.security so you can run them selectively.

import pytest

@pytest.mark.security
def test_login_does_not_accept_xss():
    # test code

Then run only security tests:

$ pytest -m security

What you learned & what's next

This lesson taught you how to write security regression tests with pytest. You can now explain the core idea: automated tests that protect against regression of known vulnerabilities. You completed a practical exercise covering a SQL injection fix and verified the test catches the reintroduction. That directly meets both learning objectives.

You also saw how to compare testing approaches (unit, E2E, property-based) and handle common pitfalls like non-deterministic tests and false positives.

Next lesson: You're ready to apply this to more complex attack surfaces — like deserialization hygiene or SSRF defenses — where regression tests become even more critical. You'll learn to combine these tests with input validation patterns to build a defense-in-depth strategy.