Regularize with L1 and L2

Your model is memorizing the training data instead of learning the underlying patterns. It nails every training example but crumbles on new, unseen data — that's overfitting, and it's one of the most frustrating problems in applied AI engineering. The fix is regularization, and the two most common approaches are L1 and L2 penalties. In this lesson, you'll understand what these penalties do, how they differ, and how to apply them in Python to build models that generalize—not just memorize.

The problem this lesson solves

Picture this: you train a logistic regression model to classify customer churn. Your training accuracy hits 98%, but on validation data it drops to 74%. That gap is the classic symptom of overfitting — the model has learned noise and irrelevant details from the training set. The more features you have (sometimes thousands), the easier it is for the model to exploit spurious correlations. Regularize with L1 and L2 penalties directly addresses this by adding a penalty term to the loss function, discouraging the model from assigning oversized importance to any single feature. Without regularization, your models become brittle and fail in production; with it, they become robust and trustworthy.

Core concept / mental model

Think of regularization as a constraint on complexity. Imagine you're fitting a line through points scattered on a graph. A flexible model can wiggle through every point (overfit), while a rigid line captures the general trend. L1 and L2 penalties are different ways to enforce that rigidity.

• L2 penalty (Ridge) adds the squared magnitude of the coefficients as a penalty: L2 = λ * Σ(w_i²). It shrinks the coefficients towards zero but never exactly zero — it keeps all features, just reduces their impact.
• L1 penalty (Lasso) adds the absolute value of the coefficients: L1 = λ * Σ|w_i|. It can shrink some coefficients to exactly zero, effectively performing feature selection.

The hyperparameter λ (lambda) controls the strength of the penalty. A larger λ means more regularization, pushing coefficients closer to zero (or to exactly zero for L1). When λ is zero, you're back to ordinary least squares.

Key definitions:

• Overfitting: model learns training noise and fails on new data.
• Regularization: any technique that reduces overfitting by penalizing model complexity.
• Coefficient (weight): a parameter the model learns; larger absolute values mean more influence.

Here's a quick visual in words: imagine you're packing a suitcase. L2 penalty forces you to compress every item slightly to fit (all items remain). L1 penalty forces you to leave some items behind entirely (many become zero).

How it works step by step

Regularization modifies the objective function that the model optimizes during training. Here's the step-by-step mechanics:

1. Define the loss function — for linear regression, that's typically mean squared error (MSE): MSE = (1/n) * Σ(y_i - ŷ_i)².
2. Add the penalty term — the new objective becomes Loss = MSE + λ * penalty. For L2, the penalty is Σ(w_i²); for L1, it's Σ|w_i|.
3. Optimize with gradient descent — the algorithm updates weights to minimize this combined loss. The gradient of the penalty term affects the update: - L2 gradient: ∂(λ*Σ(w²))/∂w = 2λw — this pulls weights towards zero proportionally to their current magnitude. - L1 gradient: ∂(λ*Σ|w|)/∂w = λ * sign(w) — this pushes weights towards zero by a constant amount, regardless of size.
4. Choose λ — typically via cross-validation: try several values, pick the one that minimizes validation error.

This process yields a model that balances fitting the data (low MSE) with keeping coefficients small (low penalty), which reduces variance and improves generalization.

In practice, you don't implement this by hand — libraries like scikit-learn provide ready-made classes such as Ridge and Lasso.

Hands-on walkthrough

Let's apply regularization to a small dataset. We'll use the Boston housing dataset (a classic regression problem) to compare ordinary linear regression, L2 regularization (Ridge), and L1 regularization (Lasso). First, ensure you have scikit-learn installed:

pip install scikit-learn

Now, let's load the data and split it into training and test sets:

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error

# Load dataset
data = load_diabetes()
X, y = data.data, data.target

# Split into train/test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

The diabetes dataset has 10 features and 442 samples — perfect for demonstrating how L1 can zero out irrelevant features. Now we'll train three models:

# 1. Ordinary Linear Regression (no regularization)
lin = LinearRegression()
lin.fit(X_train, y_train)
y_pred_lin = lin.predict(X_test)

# 2. L2 regularization (Ridge)
ridge = Ridge(alpha=1.0)   # alpha is the lambda
ridge.fit(X_train, y_train)
y_pred_ridge = ridge.predict(X_test)

# 3. L1 regularization (Lasso)
lasso = Lasso(alpha=0.1)   # smaller alpha for Lasso often works better
lasso.fit(X_train, y_train)
y_pred_lasso = lasso.predict(X_test)

# Evaluate
mse_lin = mean_squared_error(y_test, y_pred_lin)
mse_ridge = mean_squared_error(y_test, y_pred_ridge)
mse_lasso = mean_squared_error(y_test, y_pred_lasso)

print(f"Linear Regression MSE: {mse_lin:.2f}")
print(f"Ridge (L2) MSE:        {mse_ridge:.2f}")
print(f"Lasso (L1) MSE:        {mse_lasso:.2f}")

Expected output (will vary slightly due to random splits):

Linear Regression MSE: 2900.19
Ridge (L2) MSE:        2845.11
Lasso (L1) MSE:        2784.85

In this case, Lasso gives the lowest test error, likely because it eliminated a few irrelevant features. Let's inspect the coefficients to see the feature selection effect:

# Compare coefficients
print("Linear coefficients:", lin.coef_)
print("Ridge coefficients: ", ridge.coef_)
print("Lasso coefficients: ", lasso.coef_)

You'll notice that Lasso's coefficients are either zero or reduced, while Ridge's are all nonzero but smaller than linear's.

Pro tip: Use alpha (the penalty strength) as a hyperparameter. Always tune it with cross-validation using RidgeCV and LassoCV, which automatically search for the best value.

Compare options / when to choose what

When to choose what:

• Use L2 when you suspect all features are relevant but you want to reduce variance. It's numerically stable and works well even with correlated features.
• Use L1 when you need automatic feature selection — e.g., when you have thousands of features and want to identify a small subset.
• Use Elastic Net when you have correlated features and want L1's sparsity but avoid L1's instability. It's a good default in many cases.

For neural networks, the same penalties apply: you add kernel_regularizer in Keras or weight_decay in PyTorch. The concepts transfer directly.

Troubleshooting & edge cases

• Lasso fails to converge: You might see a warning like ConvergenceWarning: Objective did not converge. Increase the max_iter parameter (e.g., Lasso(alpha=0.1, max_iter=10000)) or reduce tol.
• All coefficients become zero: If you set alpha too high, Lasso will zero out everything. Reduce alpha or use LassoCV to find an optimal value.
• Ridge doesn't improve over linear regression: If your dataset is small or features are already well-scaled, regularization may not help much. Try standardizing features first using StandardScaler.
• Feature scaling matters: L1 and L2 regularization are scale-sensitive. Always standardize your features before applying regularization; otherwise, features with larger scales get penalized more.
• Interpreting alpha: In scikit-learn, alpha is the penalty strength (our λ). A very small alpha approximates no regularization, and a large alpha over-shrinks the model.

What you learned & what's next

You now know how to regularize with L1 and L2 penalties: you understand the mathematical intuition behind Ridge and Lasso, you've seen them applied in a hands-on Python exercise, and you know how to choose between them based on your feature set and goals. Key takeaways:

• Overfitting shows up as a gap between training and validation performance.
• L2 adds squared coefficient penalty; L1 adds absolute value penalty.
• L1 performs feature selection by zeroing out coefficients.
• Tune the penalty strength with cross-validation.
• Always standardize features before applying regularization.

Your next step in the Applied AI engineering track is hyperparameter tuning — learning how to systematically find the best alpha and other hyperparameters using grid search and random search. This will complement your regularization skills and make your models even more reliable in production.

Now, try experimenting with RidgeCV and LassoCV on a larger dataset to see how cross-validation picks the best alpha. Happy modeling!