Tune Hyperparameters with GridSearchCV

You’ve trained models that work “okay,” but you know they could perform better. The bottleneck isn’t the algorithm — it’s the unseen dials called hyperparameters that control how that algorithm learns. Manually tweaking them is slow, subjective, and rarely finds the best combination. In this lesson, you’ll master tune hyperparameters with GridSearchCV, the scikit-learn workhorse that systematically searches for the optimal hyperparameter set — turning a guess-and-check slog into a reproducible, automated pipeline.

The problem this lesson solves

Every machine learning model has two kinds of parameters: learned parameters (like weights in linear regression) that the training process adjusts automatically, and hyperparameters (like the C in logistic regression or n_estimators in a random forest) that you must set before training. Choosing the right hyperparameters can mean the difference between a model that generalizes well and one that overfits or underperforms.

Doing this by hand is painful. If you have three hyperparameters with five values each, that’s 125 combinations — and you’d have to test each one, track results, and avoid introducing bias by reusing the same test set. Without a systematic method, you’ll likely settle for a “good enough” model that’s actually far from the best.

GridSearchCV solves this by automating the search: it trains and evaluates your model on every combination in a defined grid, using cross-validation to score each one fairly, then returns the best hyperparameter set. This turns a tedious manual process into a few lines of code — and gives you confidence that you haven’t left performance on the table.

Core concept / mental model

Think of hyperparameter tuning like ordering a custom pizza. The dough, sauce, and cheese are the learned parameters — the model figures those out from the data. The toppings are the hyperparameters — you choose them upfront, and they dramatically change the final taste.

GridSearchCV is your menu: you list every possible topping combination (the grid), and the kitchen (the algorithm) bakes a pizza for each combination. Then a taste test (cross-validation) scores each one, and you pick the winner.

More formally:

• Hyperparameter: A configuration setting external to the model that isn’t learned from data. Examples: max_depth in a decision tree, learning_rate in gradient boosting.
• Grid: A dictionary mapping each hyperparameter name to a list of candidate values.
• Cross-validation (CV): Splitting the training data into multiple folds; the model trains on some folds and validates on the remaining one, rotating until every sample has been in the validation set. This gives a robust performance estimate.
• GridSearchCV: A scikit-learn class that performs an exhaustive search over the grid, evaluating each combination via CV, and exposing the best parameters and best score.

The key insight: GridSearchCV separates model training from evaluation. Each combination is trained from scratch inside a CV loop, so the results are unbiased and comparable.

How it works step by step

1. Define your model: Choose the estimator you want to tune (e.g., RandomForestClassifier, SVC).
2. Define the parameter grid: A dictionary where keys are hyperparameter names (as strings, matching the model’s parameters) and values are lists of candidate settings.
3. Instantiate GridSearchCV: Pass the estimator, the grid, the scoring metric (e.g., accuracy, f1, roc_auc), and the number of CV folds (e.g., cv=5).
4. Fit the grid search: Call .fit(X_train, y_train). This runs the full search: for each combination, it trains and evaluates the model on every fold.
5. Inspect results: Access best_params_, best_score_, cv_results_ (a dict containing detailed scores for every combination).
6. Refit and predict: By default, GridSearchCV refits the best model on the entire training set. Use the fitted grid_search object directly for predictions on new data.

The cause-and-effect chain is straightforward: a larger grid means more combinations → longer runtime, but better coverage of the hyperparameter space. Cross-validation ensures that your choice isn’t just lucky on a single train/test split.

Hands-on walkthrough

Let’s put this into practice with a classic dataset: the Iris flowers. We’ll tune a Support Vector Classifier (SVC).

First, load the data and split it:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

Now set up a parameter grid and run GridSearchCV:

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': [0.01, 0.1, 1],
    'kernel': ['rbf', 'linear']
}

grid_search = GridSearchCV(
    SVC(),
    param_grid,
    cv=5,
    scoring='accuracy'
)

grid_search.fit(X_train, y_train)

print("Best params:", grid_search.best_params_)
print("Best CV score:", grid_search.best_score_)

Expected output (numbers may vary slightly):

Best params: {'C': 1, 'gamma': 0.1, 'kernel': 'rbf'}
Best CV score: 0.9833333333333334

Now evaluate on the held-out test set:

from sklearn.metrics import accuracy_score

test_pred = grid_search.predict(X_test)
test_accuracy = accuracy_score(y_test, test_pred)
print(f"Test accuracy: {test_accuracy:.4f}")

Expected output:

Test accuracy: 1.0000

The model nails the test set. But you should always be suspicious of perfect scores on a small dataset — that’s where cross-validation saves you from overclaiming.

Let’s also inspect the full results to see how different combinations compare:

import pandas as pd

results = pd.DataFrame(grid_search.cv_results_)
print(results[['params', 'mean_test_score', 'rank_test_score']].head(10))

This prints a table showing each parameter set and its average CV score — a great way to spot trends (e.g., does C=0.1 always hurt?).

Compare options / when to choose what

GridSearchCV is the go‑to, but it’s not the only game in town. Here’s a comparison:

When to choose what:

• Use GridSearchCV when your hyperparameter space is small (≤10 combinations) and you have time to run it fully.
• Move to RandomizedSearchCV when you have 3+ hyperparameters with many values — you’ll get a good answer faster.
• For serious deep learning or huge datasets, consider Optuna for smarter sampling.

Pro tip: Always start with a coarse grid (few values) to find the promising region, then refine around that region with a finer grid. This two‑stage approach saves hours.

Troubleshooting & edge cases

• Runtime too long: You’re probably using too many combinations. Reduce the number of values per hyperparameter, or use n_jobs=-1 to parallelize across CPU cores. Also consider RandomizedSearchCV for larger spaces.
• ValueError: Invalid parameter ... for estimator: The hyperparameter name doesn’t match the estimator’s parameters. Check the model’s documentation (or model.get_params()) to get exact names. For pipelines, prefix parameters with the step name, e.g., 'svc__C'.
• Class imbalance: If your target classes are skewed, accuracy is misleading. Use scoring='roc_auc' or 'f1' instead.
• Leakage risk: Never use the test set to guide your grid. The only way to avoid overfitting your search is to rely on the CV scores, not the test score, when picking hyperparameters. Once chosen, test on the held-out set exactly once.
• Grid too coarse: If the best parameters are at the edge of your grid (e.g., C=100 is the highest value), extend the grid to see if even higher values help.
• Same score for many combos: On a small dataset, many hyperparameter sets may tie. That’s fine — pick the simplest one (Occam’s razor) or the one with the lowest complexity to avoid overfitting.

What you learned & what's next

You now know how to tune hyperparameters with GridSearchCV — you can define a grid, run a cross-validated search, interpret the results, and avoid common pitfalls. You understand the value of hyperparameter tuning and why cross-validation is essential for fair evaluation.

You’ve also built the habit of evaluating models robustly — a skill that carries forward to every model you train.

Next up in our Applied AI engineering path: Evaluating classification models — where you’ll dive deeper into metrics like precision, recall, and F1‑score, and learn to handle imbalanced datasets with confidence. You’ll reuse the tuned model from this lesson to see how these metrics tell a richer story than accuracy alone.

Keep tuning, keep evaluating, and keep improving!