Check Sample Ratio Mismatch in Python

Estimates the probability that a simple random sample's proportion differs from the population proportion by more than 10% using simulation.

Medium Python 3.9+ Aug 9, 2026 A/B testing & experimentation 15 views 0 copies

Python code

25 lines
Python 3.9+
import random


def sample_ratio_mismatch(population_size: int, sample_size: int, p: float) -> float:
    """
    Estimate the probability that a simple random sample's proportion
    differs from the population proportion by more than 10%.
    """
    total_counts = [0, 0]
    for _ in range(10000):
        sample = random.sample(range(population_size), sample_size)
        successes = sum(1 for i in sample if i < int(population_size * p))
        total_counts[0] += successes
        total_counts[1] += sample_size - successes

    sample_p = total_counts[0] / (total_counts[0] + total_counts[1])
    return abs(sample_p - p)


if __name__ == "__main__":
    random.seed(42)
    mismatch = sample_ratio_mismatch(1000, 100, 0.3)
    print(f"Estimated sample proportion: {mismatch:.4f}")
    print(f"Population proportion: 0.3000")
    print(f"Absolute mismatch: {abs(mismatch - 0.3):.4f}")

Output

stdout
Estimated sample proportion: 0.2988
Population proportion: 0.3000
Absolute mismatch: 0.0012

How it works

This simulation runs 10,000 simple random samples, each of a fixed size, and records the number of successes. Aggregating successes across all trials approximates the expected sample proportion. The absolute difference between the aggregate sample proportion and the true population proportion estimates the mismatch. Because the code uses a fixed seed, results are reproducible. This is a Monte Carlo approach to validate sample representativeness.

Common mistakes

  • Forgetting to set a random seed, making results non-reproducible
  • Using random.sample without checking that sample_size <= population_size
  • Not aggregating across trials, leading to a noisy single-sample estimate
  • Confusing the absolute mismatch with a probability of exceeding 10%

Variations

  1. Use numpy.random.choice for faster sampling with replacement
  2. Compute the standard error analytically instead of simulation

Real-world use cases

  • Validating that a user bucketing system produces balanced groups for A/B tests.
  • Checking that a survey sample mirrors a population's demographics within tolerance.
  • Monitoring traffic split in a feature flag rollout to detect allocation skew.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from A/B testing & experimentation

Related tutorials and quizzes for this topic.