Chi-Square Test in Python for Conversion Mock Data

Compute the chi-square statistic and approximate p-value for a mock A/B conversion test using the standard library.

Medium Python 3.9+ Aug 9, 2026 A/B testing & experimentation 12 views 0 copies

Python code

42 lines
Python 3.9+
import math
from collections import Counter

def chi_square_statistic(observed):
    """
    Compute chi-square statistic for a mock conversion test.
    observed: dict mapping outcomes to observed frequencies.
    """
    observed = Counter(observed)
    n = sum(observed.values())
    expected = n / len(observed) if observed else 0

    chi_sq = 0.0
    for count in observed.values():
        chi_sq += (count - expected) ** 2 / expected if expected else 0
    return chi_sq, expected

def chi_square_to_p_value(chi_sq, df):
    """
    Approximate p-value for chi-square distribution using
    Wilson-Hilferty transformation.
    df: degrees of freedom (k-1 for goodness of fit).
    """
    if chi_sq <= 0 or df <= 0:
        return 1.0
    z = ((chi_sq / df) ** (1/3) - (1 - 2/(9*df))) / math.sqrt(2/(9*df))
    p = 0.5 * (1 + math.erf(z / math.sqrt(2)))
    return 1 - p

if __name__ == "__main__":
    # Mock conversion: 100 visitors, 15 converted (success), 85 did not (failure)
    observed = {"converted": 15, "not_converted": 85}
    chi_sq, expected = chi_square_statistic(observed)
    df = len(observed) - 1
    p_value = chi_square_to_p_value(chi_sq, df)

    print(f"Observed: {dict(observed)}")
    print(f"Expected per category: {expected:.2f}")
    print(f"Chi-square statistic: {chi_sq:.4f}")
    print(f"Degrees of freedom: {df}")
    print(f"P-value (approx): {p_value:.4f}")
    print(f"Significant at 0.05: {p_value < 0.05}")

Output

stdout
Observed: {'converted': 15, 'not_converted': 85}
Expected per category: 50.00
Chi-square statistic: 49.0000
Degrees of freedom: 1
P-value (approx): 0.0000
Significant at 0.05: True

How it works

The chi-square statistic measures how far observed counts deviate from a uniform expected distribution. For a mock test with two outcomes (converted/not converted), expected counts are equal under the null hypothesis. The Wilson-Hilferty transformation approximates the chi-square distribution to obtain a p-value without external libraries. With only two categories, the degrees of freedom is one, and a large chi-square value like 49 indicates a highly significant deviation.

Common mistakes

  • Using `chi_square` from scipy without installing it when stdlib is sufficient
  • Forgetting to use integer counts; float counts can produce unexpected results
  • Misinterpreting p-value: small p-value means deviation is unlikely due to chance
  • Ignoring that expected counts should not be zero

Variations

  1. Use scipy.stats.chisquare for exact p-values if scipy is available
  2. Use a dictionary of observed frequencies from raw conversion data to test more than two categories

Real-world use cases

  • A/B testing landing page variants: compare conversion counts against expected equal split to decide significance.
  • Quality control: verify that defect counts across production shifts match an expected uniform distribution.
  • Marketing analytics: check if click-through rates differ across ad creatives using observed vs. expected clicks.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from A/B testing & experimentation

Related tutorials and quizzes for this topic.