How to Run a Permutation Test in Python

Run a Monte Carlo permutation test to compute a p-value for comparing two group means without parametric assumptions.

Medium Python 3.9+ Aug 9, 2026 A/B testing & experimentation 15 views 0 copies

permutation-test statistics ab-testing hypothesis-testing monte-carlo

Python code

27 lines

Python 3.9+

import random
import statistics

def permutation_test(group_a, group_b, n_permutations=10000, seed=42):
    random.seed(seed)
    combined = group_a + group_b
    observed_diff = abs(statistics.mean(group_a) - statistics.mean(group_b))
    
    count = 0
    n = len(group_a)
    for _ in range(n_permutations):
        random.shuffle(combined)
        perm_a = combined[:n]
        perm_b = combined[n:]
        perm_diff = abs(statistics.mean(perm_a) - statistics.mean(perm_b))
        if perm_diff >= observed_diff:
            count += 1
    
    p_value = count / n_permutations
    return observed_diff, p_value

if __name__ == "__main__":
    group_a = [5.1, 4.9, 5.6, 5.3]
    group_b = [6.2, 5.8, 6.1, 5.9]
    obs, p = permutation_test(group_a, group_b)
    print(f"Observed difference: {obs:.3f}")
    print(f"P-value: {p:.4f}")

Output

stdout

Observed difference: 0.850
P-value: 0.0202

How it works

The permutation test computes an empirical null distribution by repeatedly shuffling all values and splitting them into two groups of the original sizes. For each shuffle, the absolute mean difference is compared to the observed difference. The p-value is the fraction of permutations where the shuffled difference equals or exceeds the observed one. Seeding the random generator makes the result reproducible. This test makes no normality assumption, so it works with small or skewed samples.

Common mistakes

Shuffling the two groups separately instead of combined, breaking exchangeability
Forgetting to seed random, giving non-reproducible results on repeated runs
Using the wrong slicing index for the second permuted group
Comparing only greater-than instead of abs() when using a two-sided test

Variations

Use scipy.stats.permutation_test if SciPy is installed
Track all permuted differences to build a histogram for a visual null distribution

Real-world use cases

A/B testing a new landing page where click-through rates are heavily skewed, avoiding t-test assumptions.
Comparing model evaluation metrics between two cross-validation folds when sample sizes are tiny.
Validating a marketing experiment's uplift when the data has outliers that inflate standard deviations.

How to Run a Permutation Test in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from A/B testing & experimentation

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from A/B testing & experimentation

Keep learning

Tutorials

Quizzes