Chi-Square Test in Python for Conversion Mock Data
Compute the chi-square statistic and approximate p-value for a mock A/B conversion test using the standard library.
Python code
42 linesimport math
from collections import Counter
def chi_square_statistic(observed):
"""
Compute chi-square statistic for a mock conversion test.
observed: dict mapping outcomes to observed frequencies.
"""
observed = Counter(observed)
n = sum(observed.values())
expected = n / len(observed) if observed else 0
chi_sq = 0.0
for count in observed.values():
chi_sq += (count - expected) ** 2 / expected if expected else 0
return chi_sq, expected
def chi_square_to_p_value(chi_sq, df):
"""
Approximate p-value for chi-square distribution using
Wilson-Hilferty transformation.
df: degrees of freedom (k-1 for goodness of fit).
"""
if chi_sq <= 0 or df <= 0:
return 1.0
z = ((chi_sq / df) ** (1/3) - (1 - 2/(9*df))) / math.sqrt(2/(9*df))
p = 0.5 * (1 + math.erf(z / math.sqrt(2)))
return 1 - p
if __name__ == "__main__":
# Mock conversion: 100 visitors, 15 converted (success), 85 did not (failure)
observed = {"converted": 15, "not_converted": 85}
chi_sq, expected = chi_square_statistic(observed)
df = len(observed) - 1
p_value = chi_square_to_p_value(chi_sq, df)
print(f"Observed: {dict(observed)}")
print(f"Expected per category: {expected:.2f}")
print(f"Chi-square statistic: {chi_sq:.4f}")
print(f"Degrees of freedom: {df}")
print(f"P-value (approx): {p_value:.4f}")
print(f"Significant at 0.05: {p_value < 0.05}")
Output
Observed: {'converted': 15, 'not_converted': 85}
Expected per category: 50.00
Chi-square statistic: 49.0000
Degrees of freedom: 1
P-value (approx): 0.0000
Significant at 0.05: True
How it works
The chi-square statistic measures how far observed counts deviate from a uniform expected distribution. For a mock test with two outcomes (converted/not converted), expected counts are equal under the null hypothesis. The Wilson-Hilferty transformation approximates the chi-square distribution to obtain a p-value without external libraries. With only two categories, the degrees of freedom is one, and a large chi-square value like 49 indicates a highly significant deviation.
Common mistakes
- Using `chi_square` from scipy without installing it when stdlib is sufficient
- Forgetting to use integer counts; float counts can produce unexpected results
- Misinterpreting p-value: small p-value means deviation is unlikely due to chance
- Ignoring that expected counts should not be zero
Variations
- Use scipy.stats.chisquare for exact p-values if scipy is available
- Use a dictionary of observed frequencies from raw conversion data to test more than two categories
Real-world use cases
- A/B testing landing page variants: compare conversion counts against expected equal split to decide significance.
- Quality control: verify that defect counts across production shifts match an expected uniform distribution.
- Marketing analytics: check if click-through rates differ across ad creatives using observed vs. expected clicks.
Sponsored
More from A/B testing & experimentation
Keep learning
Related tutorials and quizzes for this topic.