Check Sample Ratio Mismatch in Python
Estimates the probability that a simple random sample's proportion differs from the population proportion by more than 10% using simulation.
Python code
25 linesimport random
def sample_ratio_mismatch(population_size: int, sample_size: int, p: float) -> float:
"""
Estimate the probability that a simple random sample's proportion
differs from the population proportion by more than 10%.
"""
total_counts = [0, 0]
for _ in range(10000):
sample = random.sample(range(population_size), sample_size)
successes = sum(1 for i in sample if i < int(population_size * p))
total_counts[0] += successes
total_counts[1] += sample_size - successes
sample_p = total_counts[0] / (total_counts[0] + total_counts[1])
return abs(sample_p - p)
if __name__ == "__main__":
random.seed(42)
mismatch = sample_ratio_mismatch(1000, 100, 0.3)
print(f"Estimated sample proportion: {mismatch:.4f}")
print(f"Population proportion: 0.3000")
print(f"Absolute mismatch: {abs(mismatch - 0.3):.4f}")
Output
Estimated sample proportion: 0.2988
Population proportion: 0.3000
Absolute mismatch: 0.0012
How it works
This simulation runs 10,000 simple random samples, each of a fixed size, and records the number of successes. Aggregating successes across all trials approximates the expected sample proportion. The absolute difference between the aggregate sample proportion and the true population proportion estimates the mismatch. Because the code uses a fixed seed, results are reproducible. This is a Monte Carlo approach to validate sample representativeness.
Common mistakes
- Forgetting to set a random seed, making results non-reproducible
- Using random.sample without checking that sample_size <= population_size
- Not aggregating across trials, leading to a noisy single-sample estimate
- Confusing the absolute mismatch with a probability of exceeding 10%
Variations
- Use numpy.random.choice for faster sampling with replacement
- Compute the standard error analytically instead of simulation
Real-world use cases
- Validating that a user bucketing system produces balanced groups for A/B tests.
- Checking that a survey sample mirrors a population's demographics within tolerance.
- Monitoring traffic split in a feature flag rollout to detect allocation skew.
Sponsored
More from A/B testing & experimentation
Keep learning
Related tutorials and quizzes for this topic.