Test Hypotheses with SciPy

You've cleaned your data, engineered features, and built visualizations — but now the hard question hits: is that difference in your data real, or just random noise? Without a systematic way to answer this, every insight you present is vulnerable to being dismissed as a coincidence. This lesson arms you with the statistically rigorous framework of hypothesis testing using SciPy's stats module, the backbone of data-driven decision-making, so you can confidently validate your findings and make defensible conclusions.

The Problem This Lesson Solves

Imagine you're analyzing user engagement for two website designs. After a week, Design B shows a 5% higher average session time than Design A. Your first instinct is to celebrate, but a more skeptical voice asks: would this difference still show up if I ran the test again with a different random sample? This is the crux of the problem — sampling variability. Any metric you compute from a sample is subject to random fluctuation, and without accounting for it, you risk chasing ghosts.

In data science, the cost of ignoring this is huge: you might deploy a new feature that actually hurts performance, or reject a truly beneficial change because your data was too noisy. The solution is hypothesis testing, a formal procedure to evaluate whether observed patterns are statistically significant or likely due to chance.

By the end of this lesson, you'll be able to:

• Explain the core concepts of null and alternative hypotheses, p-values, and significance levels.
• Design and execute a hypothesis test using SciPy's ttest_ind and related functions.
• Interpret results correctly and avoid common pitfalls that lead to false conclusions.

Core Concept / Mental Model

At its heart, hypothesis testing is a decision-making framework that answers a yes/no question about a population parameter, given sample data. Think of it as a courtroom trial:

• Null Hypothesis (H₀): The defendant is innocent — no effect, no difference, nothing special is happening. In data terms, e.g., 'there is no difference in average session time between designs A and B.'
• Alternative Hypothesis (H₁): The defendant is guilty — there is an effect, difference, or relationship. e.g., 'there is a difference.'
• P-value: The probability of observing the data (or something more extreme) if the null hypothesis were true. A low p-value means that if nothing were truly different, such an outcome would be very unlikely — so you start doubting the null.
• Significance level (α): The threshold for 'low' p-value, commonly 0.05. If p < α, you reject the null hypothesis and conclude there is statistically significant evidence for the alternative.

This is a simplified, but powerful, mental model. In practice, you're never proving the alternative is true; you're showing the data is inconsistent with the null, enough to reject it.

The Role of SciPy in This Process

SciPy's stats module provides ready-made functions for various tests: t-tests, chi-square tests, ANOVA, and more. These functions handle calculation of test statistics and p-values, but the burden of correct application lies on you. You must choose the right test for your data type and question, and interpret results within the context of your domain.

How It Works Step by Step

Follow this logical sequence to test any hypothesis:

1. State your hypotheses. Define H₀ and H₁ clearly. Make them mutually exclusive and exhaustive — they cover all possibilities. Example: H₀: μ₁ = μ₂ (population means equal), H₁: μ₁ ≠ μ₂.
2. Choose the significance level (α). The risk of a false positive you're willing to accept, often 0.05.
3. Select the appropriate test. This depends on your data: are you comparing means (t-test), proportions (z-test), or distributions (chi-square)? Also, check assumptions: normality, equal variance, independence.
4. Perform the test with SciPy. This typically produces a test statistic (e.g., t-statistic) and a p-value.
5. Make a decision. If p-value < α, reject H₀. Otherwise, fail to reject H₀ (note: you never 'accept' the null).
6. Communicate results. Report the statistic, p-value, effect size, and practical significance — not just 'statistically significant'.

Key Terms

• Test Statistic: A standardized value computed from sample data, used to infer how far your sample is from the null expectation.
• p-value: The probability, under H₀, of obtaining a result at least as extreme as the one observed.
• Type I Error (false positive): Rejecting a true null hypothesis.
• Type II Error (false negative): Failing to reject a false null hypothesis.

The balance between these errors is crucial. Lowering α reduces Type I errors but increases Type II errors. In practice, you'll often trade off based on the consequences of each.

Hands-on Walkthrough

Let's put this into practice with a concrete example: comparing average session times between two designs.

Setting Up

First, ensure you have SciPy installed:

pip install scipy

Now, import the necessary modules and create sample data:

import numpy as np
from scipy import stats

# Simulate session durations (in minutes) for two designs
np.random.seed(42)
design_a = np.random.normal(loc=10, scale=2, size=30)
design_b = np.random.normal(loc=12, scale=2, size=30)

print(f"Design A mean: {design_a.mean():.2f}")
print(f"Design B mean: {design_b.mean():.2f}")

Output:

Design A mean: 9.96
Design B mean: 11.91

There's an apparent difference, but let's test if it's significant.

Performing an Independent t-test

We'll use ttest_ind for comparing the means of two independent samples. By default, this assumes equal variances, but we'll set equal_var=False to be safe (Welch's t-test), which is recommended when sample variances differ.

t_stat, p_value = stats.ttest_ind(design_a, design_b, equal_var=False)
alpha = 0.05

print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

if p_value < alpha:
    print("Reject the null hypothesis: there is a significant difference.")
else:
    print("Fail to reject the null hypothesis: no significant difference.")

Output:

t-statistic: -3.756
p-value: 0.0004
Reject the null hypothesis: there is a significant difference.

The p-value is far below 0.05, so we confidently reject H₀. The difference in session times is statistically significant.

One-Sample t-test

Sometimes you want to test whether your sample mean differs from a known population mean.

# Test if the mean session time for design A is significantly different from 10 minutes
pop_mean = 10
t_stat, p_value = stats.ttest_1samp(design_a, pop_mean)

print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

Output:

t-statistic: -0.102
p-value: 0.9193

Here, the p-value is high, so we fail to reject H₀; there's no evidence that design A's mean differs from 10 minutes.

Paired t-test

For before-and-after measurements on the same subjects, use ttest_rel.

# Simulate pre/post engagement scores for the same users
before = np.random.normal(loc=50, scale=10, size=25)
after = before + np.random.normal(loc=3, scale=5, size=25)  # true effect

t_stat, p_value = stats.ttest_rel(before, after)
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")
if p_value < 0.05:
    print("Significant improvement after the change.")
else:
    print("No significant improvement.")

Output:

t-statistic: -2.594
p-value: 0.0160
Significant improvement after the change.

Pro Tip: Always check assumptions like normality and independence before relying on a t-test. For large samples (n>30), the Central Limit Theorem often makes t-tests robust to non-normality, but for small samples, consider non-parametric alternatives like mannwhitneyu (see Variations).

Compare Options / When to Choose What

SciPy offers many tests; choosing the right one depends on your data and question. Here's a quick guide:

Choosing between parametric and non-parametric

• Parametric tests (t-tests) assume data follows a known distribution (e.g., normal) and are more powerful when assumptions hold.
• Non-parametric tests (Mann-Whitney) make fewer assumptions and are safer when data is skewed or ordinal, but they are generally less powerful.

One-tailed vs. two-tailed tests

• Two-tailed: Tests for any difference (μ₁ ≠ μ₂), more common, standard in many disciplines.
• One-tailed: Tests for direction (μ₁ > μ₂), more powerful if direction is certain, but use only when you have a strong prior reason.

Troubleshooting & Edge Cases

Even with a clear process, things go wrong. Here are common issues and how to fix them:

1. Data doesn't meet normality assumptions

Symptom: Small sample size, heavily skewed data, and t-test gives unreliable p-values. Fix: Use a non-parametric alternative like mannwhitneyu for independent groups or wilcoxon for paired data. Always visually inspect with a histogram or Q-Q plot.

# Mann-Whitney U test for non-normal data
stat, p = stats.mannwhitneyu(design_a, design_b)
print(f"Mann-Whitney U: {stat:.0f}, p-value: {p:.4f}")

2. Unequal variances between groups

Symptom: Standard t-test is sensitive to unequal variances, especially with different sample sizes. Fix: Use Welch's t-test (equal_var=False) as we did earlier, or perform Levene's test to formally check.

# Levene's test for equal variances
levene_stat, levene_p = stats.levene(design_a, design_b)
print(f"Levene test p-value: {levene_p:.4f}")
if levene_p < 0.05:
    print("Variances are significantly different - use Welch's t-test.")

3. Multiple comparisons inflating error

Symptom: Running many tests inflates the chance of false positives. Fix: Apply a correction like Bonferroni: adjust α by dividing by the number of tests, or use methods like statsmodels.stats.multitest.multipletests.

4. Misinterpreting p-value

Symptom: Thinking p-value is the probability that the null hypothesis is true. Reality: It's the probability of the data given the null. Always report effect size and confidence intervals to give practical context.

What You Learned & What's Next

You've mastered the core steps to test hypotheses with scipy stats: formulating hypotheses, selecting the right test, executing with SciPy, and interpreting results. You can now validate differences in your data with confidence, avoiding false conclusions.

Key takeaways:

• Hypothesis testing is a structured decision framework to separate signal from noise.
• SciPy's stats module provides efficient functions for t-tests, chi-square, and more.
• Correct test selection based on data type and assumptions is critical for valid results.

Your next step in the Python for data science track is to dive into regression analysis, where you'll extend these testing principles to model relationships between variables and make predictions. Build on this foundation to unlock even deeper insights from your data.