Check Covariate Balance in Python

Compute standardized mean differences and KS tests to check covariate balance between treatment and control groups in Python.

Medium Python 3.9+ Aug 9, 2026 A/B testing & experimentation 13 views 0 copies

Requires third-party packages — install first
pip install numpy scipy

Python code

30 lines
Python 3.9+
import numpy as np
from scipy import stats

def balance_check(treatment, covariate):
    """Check covariate balance between treatment and control groups."""
    treat_vals = covariate[treatment == 1]
    control_vals = covariate[treatment == 0]
    
    # Standardized mean difference
    pooled_std = np.sqrt((np.var(treat_vals) + np.var(control_vals)) / 2)
    smd = (np.mean(treat_vals) - np.mean(control_vals)) / pooled_std if pooled_std > 0 else 0
    
    # Kolmogorov-Smirnov test for distribution similarity
    ks_stat, ks_pval = stats.ks_2samp(treat_vals, control_vals)
    
    return smd, ks_stat, ks_pval

if __name__ == "__main__":
    # Mock data: 100 subjects, 50 treated
    np.random.seed(42)
    n = 100
    treatment = np.random.binomial(1, 0.5, n)
    age = np.random.normal(45, 10, n)
    income = np.random.normal(50000, 15000, n)
    
    var_smd, ks_stat, ks_pval = balance_check(treatment, age)
    print(f"Age balance -> SMD: {var_smd:.3f}, KS stat: {ks_stat:.3f}, p-value: {ks_pval:.3f}")
    
    var_smd, ks_stat, ks_pval = balance_check(treatment, income)
    print(f"Income balance -> SMD: {var_smd:.3f}, KS stat: {ks_stat:.3f}, p-value: {ks_pval:.3f}")

Output

stdout
Age balance -> SMD: 0.158, KS stat: 0.160, p-value: 0.447
Income balance -> SMD: -0.145, KS stat: 0.160, p-value: 0.447

How it works

The function splits the covariate array by treatment status, then computes the standardized mean difference (SMD) using the pooled standard deviation. The KS test from scipy.stats compares the full distributions, not just means, which helps detect shifts in spread or shape. A balanced covariate typically shows SMD below 0.1 and a high KS p-value (above 0.05). The mock data uses a fixed random seed so results are reproducible.

Common mistakes

  • Using `np.var` with default ddof=0, which gives population variance; some prefer ddof=1 for sample variance.
  • Ignoring zero pooled standard deviation, which can cause division by zero when a group is constant.
  • Forgetting that KS tests require at least two samples and can be sensitive to ties.

Variations

  1. Add a threshold like 0.1 and print a pass/fail message for each covariate.
  2. Use `np.mean` and `np.std` with `ddof=1` for sample-based estimates.

Real-world use cases

  • Verifying that randomization in an A/B test produced balanced user demographics before analysis.
  • Checking covariate overlap after propensity score matching in observational studies.
  • Monitoring drift in feature distributions between model training and production serving cohorts.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from A/B testing & experimentation

Related tutorials and quizzes for this topic.