t-Tests & Chi-Square

You've cleaned your data, built beautiful visualizations, and maybe even trained a model. But then comes the question that separates a report from a decision: Is this difference real, or is it just random noise? Did the new marketing campaign actually increase sign-ups, or is that just a lucky sample? Is the higher conversion rate on your redesigned page statistically significant, or could it happen by chance? This lesson cuts through the ambiguity and gives you the tools to answer these questions with confidence: run t-tests and chi-square tests in Python. You'll move from guessing to making evidence-based conclusions, a skill that data scientists use every day.

The problem this lesson solves

You have two groups: the control and the treatment. The average click-through rate for the control is 4.2%, and for the treatment it's 5.1%. The difference looks promising, but the samples are small. Could it be a fluke? Traditional data analysis often stops at descriptive statistics—calculating the mean or the proportion and calling it a day. But that's like judging a basketball player by one game. Statistical hypothesis testing is the rigorous refere, giving you a p-value that quantifies the probability of seeing a difference as extreme as yours if there were truly no difference (the null hypothesis). If that probability is low enough (typically < 0.05), you reject the null and conclude there's a significant difference. This lesson solves the problem: it gives you a structured, code-first approach to conduct these tests in Python, so your conclusions are not just numbers but statistically sound insights.

Core concept / mental model

Think of hypothesis testing like a courtroom trial. The null hypothesis H₀ is the defendant: innocent until proven guilty. In our case, H₀ says: “There is no difference between the groups.” The alternative hypothesis H₁ is the prosecution: “There is a difference.” The p-value is the weight of the evidence. If the evidence is overwhelming (p < 0.05), the jury rejects H₀ and convicts the difference of being real.

There are two key families of tests:

• t-tests: For comparing means of numeric data (e.g., revenue, age, height). They rely on the t-distribution and assume the data is approximately normally distributed, especially with small samples.
• Chi-square tests: For comparing frequencies of categorical data (e.g., gender, color, click/no-click). They check whether the observed counts differ from what you'd expect if the categories were independent.

Both tests produce a test statistic and a p-value. The p-value is your evidence: a small p-value (typically < 0.05) means your observed result is unlikely under H₀, so you reject H₀.

How it works step by step

Let's break down the process of running any hypothesis test in Python. You'll follow these steps, regardless of whether it's a t-test or chi-square:

1. State your hypotheses – Define H₀ (no effect) and H₁ (there is an effect). This is a conceptual step, but it guides your code.
2. Choose your test – Based on your data type: numeric → t-test; categorical → chi-square. Also decide between independent (two separate groups) and paired (same group measured twice) t-tests.
3. Run the test – Use scipy.stats functions like ttest_ind, ttest_rel, chi2_contingency. These functions return a test statistic and a p-value.
4. Interpret the p-value – If p < alpha (often 0.05), we reject H₀ and conclude a significant difference.
5. Note assumptions & limitations – For t-tests: homogeneity of variances (use Welch's test if in doubt). For chi-square: expected counts should be ≥ 5 in each cell.

Choosing between one-sample, independent, and paired t-tests

• One-sample t-test: Compare the mean of a single group to a known value. E.g., test if the average height of a sample matches the national average.
• Independent two-sample t-test: Compare means of two unrelated groups. E.g., treatment vs. placebo.
• Paired t-test: Compare means from the same group at two different times or under two different conditions. E.g., before vs. after a training program.

Hands-on walkthrough

Let's get our hands dirty. We'll use scipy.stats and numpy for calculations, and pandas to hold our data. First, make sure you have them installed: pip install scipy pandas numpy. Then, let's simulate a typical A/B test scenario.

Example 1: Independent t-test (A/B Testing)

Suppose you run a website and want to know if a new layout increases time spent on the page. You'll simulate two samples: one from the old layout, one from the new.

import numpy as np
from scipy.stats import ttest_ind

# Set a random seed for reproducibility
np.random.seed(42)

# Simulate time spent (in minutes) for the old and new layouts
old_layout = np.random.normal(5.0, 1.0, 50)
new_layout = np.random.normal(5.5, 1.0, 50)

# Perform the independent t-test (unequal variance assumed)
t_stat, p_value = ttest_ind(old_layout, new_layout, equal_var=False)

print("t-statistic:", t_stat)
print("p-value:", p_value)

alpha = 0.05
if p_value < alpha:
    print("Reject H₀: There is a significant difference in time spent.")
else:
    print("Fail to reject H₀: No significant difference found.")

Expected output (may vary slightly due to random seeds):

t-statistic: -2.604
p-value: 0.0107
Reject H₀: There is a significant difference in time spent.

Example 2: Paired t-test (Before/After)

Measure the same users' engagement before and after a feature update.

from scipy.stats import ttest_rel

# Simulate daily active minutes for the same users
user_id = np.arange(1, 31)
before = np.random.normal(30, 5, 30)
after = before + np.random.normal(2, 2, 30)  # add an average increase of 2 min

# Paired t-test
t_stat, p_value = ttest_rel(before, after)
print("Paired t-test p-value:", p_value)

if p_value < 0.05:
    print("Significant increase in engagement!")
else:
    print("No significant change.")

Example 3: Chi-square test for independence

Now, let's see if gender is related to product preference. We have a contingency table of observed counts.

import pandas as pd
from scipy.stats import chi2_contingency

# Contingency table: rows = gender (Male, Female), columns = product A, B, C
data = [[50, 30, 20],  # Male
        [40, 40, 20]]  # Female

chi2, p, dof, expected = chi2_contingency(data)

print("Chi-square statistic:", chi2)
print("p-value:", p)
print("Degrees of freedom:", dof)
print("Expected frequencies:")
print(expected)

if p < 0.05:
    print("There is a significant association between gender and product preference.")
else:
    print("No significant association found.")

Expected output (values will be computed from data):

Chi-square statistic: 2.63
p-value: 0.268
Degrees of freedom: 2
Expected frequencies:
[[45.0 35.0 20.0]
 [45.0 35.0 20.0]]
No significant association found.

Compare options / when to choose what

Choosing the right test is critical. Here's a quick reference table:

For independent t-tests, you have two flavors: Student's t-test (equal variances) and Welch's t-test (unequal variances). Welch's is safer by default—it doesn't assume equal variances. For chi-square, the classic is Pearson's chi-square test, but you also have Fisher's exact test when the sample sizes are small (expected counts < 5).

Use a t-test when your outcome is numeric, such as revenue, time, or score. Use a chi-square test when your outcome is categorical, such as yes/no, product choice, or demographic category.

Troubleshooting & edge cases

Here are common errors and how you'd handle them with concrete fixes.

Error: p-value is NaN

This can happen with t-tests when your data has zero variance (e.g., all values are the same). Check for constant columns and filter them out.

if np.var(data) == 0:
    print("Data has zero variance—remove it or use another test.")

Error: Chi-square may be incorrect (expected counts < 5)

The chi-square test requires expected frequencies ≥ 5 in each cell. If violated, combine categories or use fisher_exact for 2x2 tables. For larger tables, you might need to simulate the p-value with Monte Carlo: chi2_contingency(data, simulate_pvalue=True).

# Example with small counts
small_table = [[1, 2], [3, 4]]
chi2, p, dof, exp = chi2_contingency(small_table)
print("Warning: Expected counts may be too small.")
print(exp)

Choosing the wrong t-test

If you use ttest_ind on paired data, you'll lose power. Always check whether observations are paired (e.g., same subject measured twice). If you're not sure, use the independent version and note the limitation.

Multiple testing problem

If you run many t-tests, you increase the chance of a false positive (Type I error). Use a correction like Bonferroni or FDR when doing multiple comparisons.

from statsmodels.stats.multitest import multipletests
p_values = [0.01, 0.04, 0.07]
reject, p_corrected, _, _ = multipletests(p_values, method='fdr_bh')
print(p_corrected)

What you learned & what's next

You've now mastered the core of running t-tests and chi-square tests in Python. You learned to:

• Explain the core idea behind hypothesis testing—comparing observed data to what we'd expect under the null hypothesis.
• Complete practical exercises using scipy.stats to run independent, paired, and one-sample t-tests, as well as chi-square tests.
• Choose the appropriate test based on data types and research questions.
• Troubleshoot common pitfalls like small expected counts and zero variance.

These skills are essential for data-driven decision making. Next, you'll build on this foundation by learning ANOVA for comparing means across more than two groups, and regression analysis for modeling relationships between variables. You'll also explore effect sizes and confidence intervals to complement p-values. Keep practicing with real datasets, and you'll turn raw data into compelling, evidence-backed stories.