Bayesian A/B Test Credible Interval in Python

Simulates A/B test data and computes posterior credible intervals and the probability that variant B outperforms A using Bayesian Beta-Binomial inference.

Medium Python 3.9+ Aug 9, 2026 A/B testing & experimentation 14 views 0 copies

bayesian ab-testing credible-interval beta-binomial statistics
Requires third-party packages — install first
pip install numpy scipy

Python code

34 lines
Python 3.9+ 
import numpy as np
from scipy import stats

# Simulated A/B test data
n_A = 1000
n_B = 1000
conversions_A = 120
conversions_B = 140

# Prior: Beta(1, 1) uniform
alpha_prior, beta_prior = 1, 1

# Posterior parameters
alpha_A = alpha_prior + conversions_A
beta_A = beta_prior + n_A - conversions_A
alpha_B = alpha_prior + conversions_B
beta_B = beta_prior + n_B - conversions_B

# Sample from posteriors
n_samples = 100_000
samples_A = np.random.beta(alpha_A, beta_A, n_samples)
samples_B = np.random.beta(alpha_B, beta_B, n_samples)

# Probability B is better
prob_B_better = np.mean(samples_B > samples_A)

# 95% credible interval for difference
diff = samples_B - samples_A
lower, upper = np.percentile(diff, [2.5, 97.5])

print(f"Conversion rate A: {conversions_A/n_A:.3f}")
print(f"Conversion rate B: {conversions_B/n_B:.3f}")
print(f"95% CI for difference (B-A): [{lower:.4f}, {upper:.4f}]")
print(f"P(B > A) = {prob_B_better:.4f}")

Output

stdout 
Conversion rate A: 0.120
Conversion rate B: 0.140
95% CI for difference (B-A): [0.0007, 0.0394]
P(B > A) = 0.9123

How it works

The Beta distribution is the conjugate prior for the Binomial likelihood, so the posterior is also Beta with parameters updated by observed conversions and non-conversions. Sampling from posteriors via np.random.beta lets us approximate the distribution of the difference between conversion rates without closed-form math. Comparing samples element-wise estimates the probability B beats A, and percentiles give the credible interval. This is a fully Bayesian alternative to frequentist hypothesis tests.

Common mistakes

  • Using a strong prior like Beta(0.5, 0.5) without justification can bias results toward 0 or 1.
  • Forgetting that credible intervals are Bayesian statements about the parameter, not frequentist confidence intervals.
  • Using too few posterior samples, producing noisy estimates of probability and interval endpoints.

Variations

  1. Use `scipy.stats.beta` with `cdf` to compute probabilities analytically instead of sampling.
  2. Add a burn-in or thinning step when using MCMC for more complex priors.

Real-world use cases

  • Deciding whether a new landing page variant significantly improves signup rates before full rollout.
  • Evaluating if a recommender system's click-through rate beats the current champion in an online experiment.
  • Quantifying uncertainty in conversion lift when reporting results to non-technical stakeholders.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from A/B testing & experimentation

Keep learning

Related tutorials and quizzes for this topic.

Tutorials

Quizzes