Implement Logistic Regression

You have a pile of labeled data — emails marked spam or not, transactions flagged as fraudulent or safe — and you need a model that draws a clean line between the two groups so you can classify new, unseen examples. Linear regression won't cut it because it predicts continuous numbers and doesn't respect class boundaries. This is where logistic regression for classification steps in: a simple, fast, and surprisingly powerful algorithm that outputs probabilities between 0 and 1, letting you say "there's an 85% chance this email is spam." In this lesson, you'll implement logistic regression from scratch in Python, understand the math that makes it tick, and apply it to a real dataset — all without needing a PhD in statistics.

The problem this lesson solves

Many classification tasks in applied AI start with the same frustration: your data is labeled, but you have no idea where to begin. You could reach for a deep neural network, but that's overkill for a few thousand rows and will take ages to train on a laptop. You need something that:

• Learns from historical examples (supervised learning)
• Outputs a probability or class membership, not just a number
• Handles nonlinear relationships? Not necessarily — linear boundaries are often enough
• Runs quickly, even on modest hardware

Linear regression fails on classification because it can predict values below 0 or above 1, and it treats the distance between classes as meaningful. For example, predicting "spam" as 0 and "not spam" as 1, a value of 0.5 is ambiguous, and a value of 3 has no meaning. Logistic regression solves this by squeezing the output through a sigmoid function, which maps any real number to the range (0, 1). Now you get a probability you can threshold: above 0.5 means class 1, below means class 0.

Why it matters now: In the Applied AI engineering track, you've already mastered data manipulation and visualization. Logistic regression is arguably the most interpretable classification model you'll build — it's the baseline every data scientist reaches for before trying anything fancier.

Core concept / mental model

Think of logistic regression as a weighted voting system. Each input feature (e.g., "contains the word 'lottery'", "email length") gets a weight that reflects how strongly it pushes toward class 1 (spam). The model combines all these weighted votes into a single score, then passes it through a sigmoid function to turn the score into a probability.

Here's the math in plain English:

[ z = w_0 + w_1 x_1 + w_2 x_2 + \dots + w_n x_n ]

where ( w_0 ) is the bias (intercept), and ( w_i ) are the feature weights. Then:

[ \hat{p} = \sigma(z) = \frac{1}{1 + e^{-z}} ]

If ( \hat{p} \ge 0.5 ), predict class 1; otherwise, class 0.

The sigmoid is the heart of logistic regression. It's an S-shaped curve that flattens near 0 and 1, which makes it perfect for modeling probabilities. The model learns the optimal weights by minimizing a loss function called log-loss (or binary cross-entropy). During training, it uses gradient descent to adjust the weights so the predicted probabilities get closer to the true labels.

A diagram in words

Imagine a scatter plot with two clusters: circles (class 0) on the left, triangles (class 1) on the right. Logistic regression finds a straight line (the decision boundary) that best separates them. Points far to the right have probability near 1; points far to the left near 0. The line itself corresponds to ( z = 0 ), where the probability is 0.5.

How it works step by step

The full training process boils down to these steps:

1. Initialize weights — start with zeros or small random values.
2. Compute predictions — calculate ( \hat{p} ) for every training example using the current weights.
3. Calculate loss — use log-loss to measure how wrong the predictions are.
4. Compute gradient — the direction and magnitude of change needed for each weight to reduce loss.
5. Update weights — take a step in the negative gradient direction (gradient descent).
6. Repeat — loop for a fixed number of epochs or until the loss converges.

The loss function

For a single example with true label ( y ) (0 or 1), the log-loss is:

[ L = -\left[ y \log(\hat{p}) + (1-y) \log(1-\hat{p}) \right] ]

If the true label is 1 and the model predicts 0.9, the loss is small; if it predicts 0.1, the loss is huge. This asymmetric penalization ensures the model is confident when it's right and humbled when it's wrong.

Gradient descent update

For each weight ( w_j ):

[ w_j := w_j - \alpha \cdot \frac{1}{m} \sum_{i=1}^{m} (\hat{p}^{(i)} - y^{(i)}) x_j^{(i)} ]

where ( \alpha ) is the learning rate and ( m ) is the number of training examples. Intuitively, if the model overestimates the probability for a positive example, the weight decreases; if it underestimates, the weight increases.

Key insight: The gradient formula is stunningly simple — it's just the residual (prediction minus true value) times the feature value, averaged over all examples. That's why logistic regression is so fast to train.

Hands-on walkthrough

Now let's implement logistic regression for classification from scratch in Python, using only NumPy. We'll use a synthetic dataset so you can see exactly what's happening under the hood.

Step 1: Generate and split data

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate a dataset with 2 informative features and some noise
X, y = make_classification(
    n_samples=1000,
    n_features=2,
    n_informative=2,
    n_redundant=0,
    n_clusters_per_class=1,
    random_state=42
)

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Training samples: {X_train.shape[0]}, test samples: {X_test.shape[0]}")

Expected output:

Training samples: 800, test samples: 200

Step 2: Implement logistic regression class

Here's a complete, from-scratch implementation:

class LogisticRegressionScratch:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.weights = None
        self.bias = None

    def _sigmoid(self, z):
        return 1 / (1 + np.exp(-z))

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0

        for _ in range(self.n_iterations):
            # Compute predictions
            linear_model = np.dot(X, self.weights) + self.bias
            y_pred = self._sigmoid(linear_model)

            # Compute gradients
            error = y_pred - y
            dw = (1 / n_samples) * np.dot(X.T, error)
            db = (1 / n_samples) * np.sum(error)

            # Update weights
            self.weights -= self.learning_rate * dw
            self.bias -= self.learning_rate * db

    def predict_proba(self, X):
        linear_model = np.dot(X, self.weights) + self.bias
        return self._sigmoid(linear_model)

    def predict(self, X, threshold=0.5):
        proba = self.predict_proba(X)
        return (proba >= threshold).astype(int)

Step 3: Train and evaluate

# Train the model
model = LogisticRegressionScratch(learning_rate=0.1, n_iterations=1000)
model.fit(X_train, y_train)

# Predictions and accuracy
from sklearn.metrics import accuracy_score, confusion_matrix

y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)

print(f"Accuracy on test set: {accuracy:.2f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, y_pred))

Expected output (varies slightly due to randomness):

Accuracy on test set: 0.99
Confusion matrix:
[[ 95   2]
 [  0 103]]

The model performs exceptionally well because the dataset is linearly separable (with some noise). Notice how few lines of code are needed — that's the beauty of logistic regression.

Tuning the learning rate and iterations

Try running the same training with different hyperparameters to see their impact:

# Too high learning rate (diverges)
model_fast = LogisticRegressionScratch(learning_rate=2.0, n_iterations=100)
model_fast.fit(X_train, y_train)
print("Fast LR accuracy:", accuracy_score(y_test, model_fast.predict(X_test)))

# Too few iterations (underfits)
model_slow = LogisticRegressionScratch(learning_rate=0.01, n_iterations=10)
model_slow.fit(X_train, y_train)
print("Few iterations accuracy:", accuracy_score(y_test, model_slow.predict(X_test)))

Expected output:

Fast LR accuracy: 0.5
Few iterations accuracy: 0.87

The fast learning rate makes the loss oscillate and fail to converge; the too-few iterations leaves the model underfit. A middle ground (like 0.1 and 1000) works best.

Using scikit-learn for comparison

In practice, you'll often use a battle-tested library:

from sklearn.linear_model import LogisticRegression

sk_model = LogisticRegression(max_iter=1000)
sk_model.fit(X_train, y_train)
sk_pred = sk_model.predict(X_test)
print("Scikit-learn accuracy:", accuracy_score(y_test, sk_pred))

Expected output:

Scikit-learn accuracy: 0.99

Your from-scratch model matches the optimized library version — a great sanity check.

Compare options / when to choose what

Logistic regression isn't your only option. Here's how it stacks up against common alternatives:

For the vast majority of tabular classification problems, logistic regression should be your first model — it gives you a baseline accuracy and interpretable coefficients. If you need nonlinearity, you can use polynomial features or kernels, but that adds complexity.

When to move beyond logistic regression

• Data is huge (millions of rows): Linear models still scale well, but you might prefer stochastic gradient descent variants.
• Boundary is highly nonlinear: Consider tree-based methods or deep learning.
• You need to model interactions automatically: Logistic regression requires manual feature engineering.

Troubleshooting & edge cases

Implementing logistic regression is straightforward, but you'll hit a few common pitfalls:

1. Learning rate too high → loss explodes

If your accuracy suddenly drops to ~50% (random guessing), your learning rate is too large. The gradient updates overshoot and the cost nan or cycles. Fix: reduce learning_rate to 0.01 or 0.001, and watch the loss per iteration.

2. Sigmoid overflow

Computing np.exp(-z) with a large positive z (like -1000) causes overflow warnings. Fix: clip z to a range like [-500, 500] before applying the sigmoid, or use scipy.special.expit which handles this.

3. Feature scales vary wildly

If one feature ranges from 0 to 1 and another from 0 to 10000, gradient descent becomes painfully slow. Fix: standardize features (subtract mean, divide by standard deviation) before training.

4. Imbalanced classes

The model will predict the majority class almost always if 95% are class 0. Fix: use class weights (class_weight='balanced' in scikit-learn) or adjust the decision threshold to 0.3 instead of 0.5.

5. Confusion over multiclass

Logistic regression is inherently binary. For multi-class problems, use multi_class='multinomial' in scikit-learn or the one-vs-rest strategy. Our from-scratch code works only for binary; extend it with softmax for multiclass.

What you learned & what's next

In this lesson, you implemented logistic regression for classification from scratch in Python. You now understand:

• The sigmoid function maps linear scores to probabilities
• Log-loss and gradient descent drive weight updates
• You can build a working classifier in ~30 lines of NumPy
• Hyperparameters like learning rate and iterations critically affect convergence
• When to choose logistic regression versus more complex models

You also debugged common issues like learning rate divergence and feature scaling.

Now you're ready to take your model from scratch to production. The next lesson in the Applied AI engineering track focuses on regularization techniques — helping you prevent overfitting when you add more features. Regularization adds a penalty for large weights, which is a natural extension of what you just built. You'll learn L1 (Lasso) and L2 (Ridge) penalties and how to implement them in your own logistic regression code.

But before moving on, consider this: how would you handle a dataset where the boundary is not linear? You could add polynomial features, but that leads to the curse of dimensionality — the perfect segue into the importance of regularization. Keep that thought as you dive into the next lesson.