Benjamini Hochberg FDR Correction in Python
Implement the Benjamini-HHochberg false discovery rate (FDR) procedure in Python to control the expected proportion of false positives among rejected hypotheses.
Requires third-party packages — install first
pip install numpy
Python code
25 linesimport numpy as np
def benjamini_hochberg(p_values, alpha=0.05):
p_values = np.array(p_values)
n = len(p_values)
sorted_idx = np.argsort(p_values)
sorted_p = p_values[sorted_idx]
thresholds = (np.arange(1, n + 1) / n) * alpha
significant = sorted_p <= thresholds
if not significant.any():
return np.zeros(n, dtype=bool)
largest_significant = np.max(np.where(significant)[0])
selected = np.zeros(n, dtype=bool)
selected[sorted_idx[:largest_significant + 1]] = True
return selected
if __name__ == "__main__":
p_vals = [0.01, 0.04, 0.03, 0.20, 0.06, 0.012]
rejected = benjamini_hochberg(p_vals)
print("P-values:", p_vals)
print("Rejected:", rejected.tolist())
print("Significant p-values:", [p for p, r in zip(p_vals, rejected) if r])Output
P-values: [0.01, 0.04, 0.03, 0.20, 0.06, 0.012]
Rejected: [True, True, True, False, False, True]
Significant p-values: [0.01, 0.04, 0.03, 0.012]How it works
The Benjamini-Hochberg (BH) procedure sorts p-values ascending and compares each to a threshold (i/n)*alpha. The largest index where a p-value is below its threshold marks the cutoff; all p-values up to that index are declared significant. This implementation uses NumPy for efficient sorting and boolean indexing. The result is a boolean mask aligned with the original input order. The procedure controls the FDR at the chosen alpha level (default 0.05), making it less conservative than Bonferroni when many hypotheses are tested.
Common mistakes
- Forgetting that p-values must be sorted before computing thresholds
- Returning only the significant p-values instead of a boolean mask aligned with original order
- Assuming the procedure controls the family-wise error rate instead of the false discovery rate
- Not handling the case where no p-value is significant (must return all False)
Variations
- Use `statsmodels.stats.multitest.multipletests` with `method='fdr_bh'` for a tested implementation.
- Implement a step-up procedure using a for-loop that tracks the largest index satisfying the condition.
Real-world use cases
- In A/B testing, correct for multiple metrics (e.g., conversion rate, revenue) to limit false discovery when evaluating experiment success.
- In genomics research, identify differentially expressed genes across thousands of probes while controlling the FDR.
- In marketing analytics, assess the significance of multiple campaign variants to decide which features to roll out.
Sponsored
More from A/B testing & experimentation
Keep learning
Related tutorials and quizzes for this topic.