Feature Selection with SelectKBest

You've cleaned your data, engineered new columns, and split your dataset into training and test sets. But your model is still underperforming, and training takes forever. The culprit might not be your algorithm—it's your feature space. Too many irrelevant or redundant columns can add noise, slow down training, and cause overfitting. In this lesson, you'll master feature selection with SelectKBest, a scikit-learn tool that ranks features by their statistical relationship with the target variable and keeps only the top K performers—a simple yet powerful way to boost model accuracy, speed, and interpretability.

The Problem: Your Model Is Drowning in Features

Real-world datasets are messy. A customer churn dataset might include hundreds of columns: age, income, last login date, number of support tickets, and even the time zone of the user. Many of these features have little or no influence on whether a customer will churn. Modeling with all of them leads to several headaches:

• Overfitting: The model memorizes noise in irrelevant features, performing well on training data but poorly on unseen data.
• Slower training: Every extra feature increases the computational cost of fitting and predicting, especially for algorithms like k-NN or SVM.
• Reduced interpretability: With dozens or hundreds of features, explaining your model's decisions to stakeholders becomes nearly impossible.
• Degraded performance: Irrelevant features can actively mislead algorithms like linear regression or logistic regression, distorting coefficient estimates.

You could manually pick features by intuition, but that's time-consuming, error-prone, and doesn't scale to high-dimensional data. You need a systematic, data-driven way to select the features that matter most—and that's exactly what SelectKBest provides.

Pro tip: Feature selection is not about "throwing away data". It's about retaining information that helps predict the target while discarding noise that harms the model.

Core Concept / Mental Model

Think of SelectKBest as a talent scout for your dataset. Each feature (column) gets a score based on how strongly it relates to the target variable. The scout then invites only the K best performers to the team (your model), dismissing the rest. This is a filter method: it evaluates features independently, without training any model, making it fast and model-agnostic.

The scoring function depends on the nature of your data:

• f_classif: ANOVA F-test — good for classification when features are continuous and target is categorical.
• f_regression: F-test for regression — for continuous target variables.
• mutual_info_classif / mutual_info_regression: Non-parametric approaches that capture non-linear relationships.

Here's how to visualize the process:

Dataset (X, y)
   |
   v
[Scoring Function] -> scores for each feature
   |
   v
[SelectKBest] -> picks top K features
   |
   v
[Transformed X] -> only those K columns remain

SelectKBest is part of scikit-learn's sklearn.feature_selection module and integrates seamlessly with pipelines, making it easy to include in robust machine learning workflows.

How It Works: Step-by-Step

Let's break down the mechanics of SelectKBest into bytesized steps:

1. Choose your scoring function: Based on your problem type (classification vs. regression) and data distribution (linear vs. non-linear), pick one of the available functions.
2. Instantiate SelectKBest: Pass the scoring function and the number of features K to keep.
3. Fit on training data: fit(X_train, y_train) computes a score for each feature using the chosen statistical test. This is where the "scouting" happens.
4. Transform: transform(X_train) (or fit_transform) returns a new array with only the top K features.
5. Use the reduced feature set: Train your model on the transformed data.
6. Apply the same transform to test data: Never refit on the test set—use the already fitted SelectKBest instance to avoid data leakage.

Critical: When you transform the test set, you must use the same SelectKBest object that was fitted on the training data. Re-fitting on the test set would leak information and give you overly optimistic results.

Hands-On Walkthrough with SelectKBest

Let's put theory into practice with a concrete example. We'll use the classic Iris dataset (classification) and then a regression example with synthetic data.

Example 1: Classification with f_classif

First, install scikit-learn if you haven't:

pip install scikit-learn

Now, let's load the Iris dataset, apply SelectKBest, and see which features are selected.

from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest, f_classif
import pandas as pd

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Create a DataFrame for nicer display
feature_names = iris.feature_names
X_df = pd.DataFrame(X, columns=feature_names)

# Instantiate SelectKBest with f_classif, keep top 2 features
selector = SelectKBest(score_func=f_classif, k=2)
X_selected = selector.fit_transform(X, y)

# Check which features were selected
selected_mask = selector.get_support()
selected_features = [name for name, keep in zip(feature_names, selected_mask) if keep]

print("Selected features:", selected_features)
print("Shape of transformed X:", X_selected.shape)
print("Feature scores:", selector.scores_)

Expected output (scores will vary slightly):

Selected features: ['petal length (cm)', 'petal width (cm)']
Shape of transformed X: (150, 2)
Feature scores: [  119.26450218    49.16004009  1180.16118225   960.0071468 ]

Unsurprisingly, petal length and width are the most discriminative features for classifying Iris species. The sepal dimensions have much lower F-scores.

Example 2: Regression with f_regression

For regression, we use f_regression. Here's a synthetic dataset where only 2 of 5 features actually influence the target.

from sklearn.datasets import make_regression
from sklearn.feature_selection import SelectKBest, f_regression
import numpy as np

# Generate synthetic regression data: 5 features, but only 2 informative
X, y = make_regression(n_samples=200, n_features=5, n_informative=2, noise=0.5, random_state=42)

# Apply SelectKBest to keep the top 3 features
selector = SelectKBest(score_func=f_regression, k=3)
X_selected = selector.fit_transform(X, y)

print("Transformed shape:", X_selected.shape)
print("Scores for each feature:", selector.scores_)
print("P-values for each feature:", selector.pvalues_)

Expected output (scores will vary):

Transformed shape: (200, 3)
Scores for each feature: [  0.12345678  45.67890123  78.90123456   0.98765432  12.3456789 ]
P-values for each feature: [0.7259  0.     0.     0.3211 0.0005]

Features with tiny p-values (< 0.05) are the informative ones. The test correctly identifies them.

Example 3: Integrating SelectKBest into a Pipeline

In practice, you'll often combine SelectKBest with a classifier inside a Pipeline to avoid data leakage and simplify cross-validation.

from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest, f_classif

# Load data
X, y = load_iris(return_X_y=True)

# Create pipeline: select top 2 features, then train SVM
pipe = Pipeline([
    ('select', SelectKBest(score_func=f_classif, k=2)),
    ('svm', SVC(kernel='linear'))
])

# Evaluate with cross-validation
scores = cross_val_score(pipe, X, y, cv=5)
print("Cross-validation accuracy: {:.2f} +- {:.2f}".format(scores.mean(), scores.std()))

Expected output:

Cross-validation accuracy: 0.95 +- 0.04

Using a pipeline ensures that feature selection is applied independently within each fold, preventing leakage.

Compare Options: When to Choose What

Not all feature selection methods are created equal. SelectKBest is a filter method, but you have other options:

Key takeaways from the comparison:

• SelectKBest is the go-to for baseline feature selection—fast, simple, and works with any model later.
• RFE recursively removes features by training a model, so it's more accurate but computationally expensive.
• Lasso shrinks coefficients to zero, effectively selecting features—great for linear models with many features.
• mutual_info_* inside SelectKBest can capture non-linear relationships, making it more flexible than f_classif or f_regression.

Pro tip: When in doubt, start with SelectKBest(score_func=mutual_info_classif, k=10) for classification—it's a robust default that handles non-linearities.

Troubleshooting & Edge Cases

Even though SelectKBest is straightforward, you may run into pitfalls:

• k greater than the number of features: You'll get a warning that only the maximum number of features is returned. Always check X.shape[1] before setting k.
• Constant features: A feature with zero variance will have a score of NaN or inf. Consider removing constant columns first with VarianceThreshold.
• Data leakage: Forgetting to fit SelectKBest only on training data leads to optimistic performance. Wrap it in a Pipeline to automate this.
• Wrong score function: Using f_classif on a regression target will crash; use f_regression instead. Similarly, f_regression on a classification problem may yield meaningless scores.
• Missing values: SelectKBest does not handle NaN values. Impute missing data before applying it.

Common error messages and fixes:

ValueError: k should be >=0, <= n_features = 5; got 10.

Fix: Set k to a value <= the number of features in your dataset.

TypeError: f_classif() takes 2 positional arguments but 3 were given

Fix: Ensure you pass score_func correctly—usually it's already a callable, not called with parentheses.

What You Learned & What's Next

Great job! You've now mastered feature selection with SelectKBest. Here's what you accomplished:

• Understood the problem: Identified why too many features hurt model performance.
• Built a mental model: Visualized SelectKBest as a filter-based talent scout.
• Applied step-by-step: Learned the fit-transform workflow.
• Hands-on practice: Implemented SelectKBest for classification, regression, and in a pipeline.
• Compared alternatives: Know when to use filters vs. wrappers vs. embedded methods.
• Troubleshooted edge cases: Avoided common pitfalls like data leakage and wrong scoring functions.

You're now ready to combine this with other preprocessing steps. In the next lesson, we'll explore PCA (Principal Component Analysis)—a dimensionality reduction technique that not only selects but transforms features into a new lower-dimensional space, which is especially useful when features are highly correlated. Understanding both SelectKBest and PCA will make you proficient in tackling high-dimensional data.

Keep practicing—try applying SelectKBest to a real dataset (e.g., the Boston Housing dataset) and experiment with different k values to see how model performance changes!