Check Covariate Balance in Python
Compute standardized mean differences and KS tests to check covariate balance between treatment and control groups in Python.
pip install numpy scipy
Python code
30 linesimport numpy as np
from scipy import stats
def balance_check(treatment, covariate):
"""Check covariate balance between treatment and control groups."""
treat_vals = covariate[treatment == 1]
control_vals = covariate[treatment == 0]
# Standardized mean difference
pooled_std = np.sqrt((np.var(treat_vals) + np.var(control_vals)) / 2)
smd = (np.mean(treat_vals) - np.mean(control_vals)) / pooled_std if pooled_std > 0 else 0
# Kolmogorov-Smirnov test for distribution similarity
ks_stat, ks_pval = stats.ks_2samp(treat_vals, control_vals)
return smd, ks_stat, ks_pval
if __name__ == "__main__":
# Mock data: 100 subjects, 50 treated
np.random.seed(42)
n = 100
treatment = np.random.binomial(1, 0.5, n)
age = np.random.normal(45, 10, n)
income = np.random.normal(50000, 15000, n)
var_smd, ks_stat, ks_pval = balance_check(treatment, age)
print(f"Age balance -> SMD: {var_smd:.3f}, KS stat: {ks_stat:.3f}, p-value: {ks_pval:.3f}")
var_smd, ks_stat, ks_pval = balance_check(treatment, income)
print(f"Income balance -> SMD: {var_smd:.3f}, KS stat: {ks_stat:.3f}, p-value: {ks_pval:.3f}")
Output
Age balance -> SMD: 0.158, KS stat: 0.160, p-value: 0.447
Income balance -> SMD: -0.145, KS stat: 0.160, p-value: 0.447
How it works
The function splits the covariate array by treatment status, then computes the standardized mean difference (SMD) using the pooled standard deviation. The KS test from scipy.stats compares the full distributions, not just means, which helps detect shifts in spread or shape. A balanced covariate typically shows SMD below 0.1 and a high KS p-value (above 0.05). The mock data uses a fixed random seed so results are reproducible.
Common mistakes
- Using `np.var` with default ddof=0, which gives population variance; some prefer ddof=1 for sample variance.
- Ignoring zero pooled standard deviation, which can cause division by zero when a group is constant.
- Forgetting that KS tests require at least two samples and can be sensitive to ties.
Variations
- Add a threshold like 0.1 and print a pass/fail message for each covariate.
- Use `np.mean` and `np.std` with `ddof=1` for sample-based estimates.
Real-world use cases
- Verifying that randomization in an A/B test produced balanced user demographics before analysis.
- Checking covariate overlap after propensity score matching in observational studies.
- Monitoring drift in feature distributions between model training and production serving cohorts.
Sponsored
More from A/B testing & experimentation
Keep learning
Related tutorials and quizzes for this topic.