Delta Method for Ratio Metrics in A/B Testing with Python
Computes the confidence interval for the difference between two ratio metrics using the delta method, with mock A/B test data.
pip install numpy scipy
Python code
51 linesimport numpy as np
from scipy.stats import norm
def delta_method_ratio_delta(control: np.ndarray, treatment: np.ndarray, confidence: float = 0.95):
"""Estimate confidence interval for ratio metric using delta method.
Args:
control: numerator/denominator pairs from control group (n x 2 array)
treatment: numerator/denominator pairs from treatment group (m x 2 array)
confidence: confidence level (default 0.95)
Returns:
(control_ratio, treatment_ratio, delta, ci_lower, ci_upper)
"""
alpha = 1 - confidence
z = norm.ppf(1 - alpha / 2)
def ratio_stats(data):
num = data[:, 0]
den = data[:, 1]
r = num.sum() / den.sum()
n = len(data)
var_num = np.var(num)
var_den = np.var(den)
cov = np.cov(num, den)[0, 1]
mean_den = den.mean()
# Delta method variance: var(ratio) ~ (var_num - 2*r*cov + r^2*var_den) / mean_den^2
var_ratio = (var_num - 2 * r * cov + r**2 * var_den) / mean_den**2 / n
return r, var_ratio
r_control, var_control = ratio_stats(control)
r_treatment, var_treatment = ratio_stats(treatment)
delta = r_treatment - r_control
se_delta = np.sqrt(var_control + var_treatment)
ci_lower = delta - z * se_delta
ci_upper = delta + z * se_delta
return r_control, r_treatment, delta, ci_lower, ci_upper
if __name__ == "__main__":
# Mock data: columns are [numerator, denominator]
np.random.seed(42)
control_data = np.array([[50, 100], [45, 95], [55, 110], [48, 102], [52, 98]])
treatment_data = np.array([[60, 100], [58, 105], [62, 110], [55, 95], [59, 103]])
result = delta_method_ratio_delta(control_data, treatment_data)
print(f"Control ratio: {result[0]:.4f}")
print(f"Treatment ratio: {result[1]:.4f}")
print(f"Delta: {result[2]:.4f}")
print(f"95% CI: ({result[3]:.4f}, {result[4]:.4f})")Output
Control ratio: 0.5000
Treatment ratio: 0.5796
Delta: 0.0796
95% CI: (-0.0061, 0.1653)How it works
The delta method approximates the variance of a ratio of two correlated random variables by taking a first-order Taylor expansion. Here, we propagate the variance of the numerator and denominator (and their covariance) through the ratio formula, scaling by the squared mean denominator. The standard error of the delta (difference between treatment and control ratios) is the square root of the sum of both group variances, assuming independence. The final confidence interval uses the normal quantile from scipy.stats.norm.ppf. This approach is standard for ratio metrics like conversion rates or revenue per user where the denominator is not fixed.
Common mistakes
- Forgetting to account for the covariance between numerator and denominator in the variance formula.
- Using `np.var` with default ddof=0 instead of sample variance (ddof=1) when data is a sample.
- Assuming independent groups when the control and treatment are correlated (e.g., paired data).
- Not normalizing the variance by the sample size when computing the standard error.
Variations
- Use bootstrap resampling to estimate the CI without the normality assumption.
- Use a log transformation and then back-transform to keep the interval positive for ratio metrics like revenue per user.
Real-world use cases
- Measuring the lift in conversion rate (clicks per impression) between two web page variants in a growth experiment.
- Estimating the confidence interval for revenue per user difference when evaluating a new checkout flow in e-commerce.
- Assessing the impact on task success ratio (tasks completed per hour) when a support automation tool is rolled out to a test team.
Sponsored
More from A/B testing & experimentation
Keep learning
Related tutorials and quizzes for this topic.