Build a Simple Perceptron

When you first hear about neural networks, the terminology can feel like a wall: neurons, weights, biases, activation functions, backpropagation. But beneath every transformer, every vision model, and every recommendation system lies a tiny, elegant unit — the perceptron. If you've ever wondered how a machine can learn a decision boundary from data, this lesson removes the mystery. You'll implement a classic perceptron from scratch in pure Python, watch it learn the logic of an OR gate, and understand the exact math and mechanics that power modern AI. By the end, you'll not only have a working model — you'll have the confidence to see any neural network as a stack of these simple learners.

The problem this lesson solves

Machine learning libraries like scikit-learn or PyTorch make it almost too easy to train a model: a couple of lines and the algorithm just... works. But this convenience comes at a cost — it hides the fundamental mechanics of learning. If you blindly call Perceptron.fit() without understanding what's happening under the hood, you'll struggle when things go wrong — when your model won't converge, when your features need scaling, or when you need to explain why a decision was made.

This lesson addresses the "black box" problem head-on. By building a simple perceptron from scratch, you will:

• Demystify the training loop — see how data flows through the model and how weights are updated.
• Understand the core math — the weighted sum, the step function, and the delta rule.
• Build a solid foundation — before moving on to multi-layer networks, activation functions, and gradient descent in later lessons.

The perceptron is the hello world of neural networks. Mastering it now will pay off tenfold when you encounter more complex architectures.

Core concept / mental model

Think of a perceptron as a single neuron making a binary decision. It's like a tiny, decision-making fish in a pond: it watches a few inputs, and based on their weighted importance, it either swims toward food (output 1) or stays put (output 0).

The anatomy of a perceptron:

• Inputs (x₁, x₂, …, xₙ): features of a single sample — like numbers, booleans, or measurements.
• Weights (w₁, w₂, …, wₙ): how much each input matters. Positive weight = pushes toward the "yes" class; negative weight = pushes toward "no".
• Bias (b): a threshold that shifts the decision boundary — think of it as the neuron's base temperament.
• Activation function: the step function that converts the weighted sum into a binary output (0 or 1).

The perceptron computes a weighted sum: z = w₁·x₁ + w₂·x₂ + … + wₙ·xₙ + b. Then it applies the step function: if z > 0, output is 1, otherwise 0. This is a linear decision boundary — in two dimensions, it's a straight line; in higher dimensions, a hyperplane.

Learning means adjusting the weights and bias so the decision boundary correctly separates the training examples. The perceptron does this by comparing its prediction to the true label and nudging the weights in the right direction — like turning a dial until the picture comes into focus.

How it works step by step

Now let's break down the training algorithm, known as the perceptron learning rule. This is the heart of the matter.

1. Initialize weights and bias

Start with random weights (or zeros) and a bias (often 0). The initial decision boundary is arbitrary.

2. Iterate over training samples

For each sample (x₁, x₂, …, xₙ, y) where y is the true label (0 or 1):

• Compute the weighted sum z = w·x + b.
• Apply the step function to get a prediction ŷ (0 or 1).

3. Update weights (the delta rule)

The weight update rule is the essence of learning:

• Error e = y - ŷ — the difference between true and predicted. This will be -1, 0, or +1.
• Update each weight: wᵢ = wᵢ + η · e · xᵢ
• Update bias: b = b + η · e

Here, η (eta) is the learning rate — a small positive number (e.g., 0.1) that controls step size.

Intuition: If the perceptron predicts 0 but the true label is 1 (error = +1), we increase weights for inputs that are positive, making the weighted sum larger — pushing the output toward 1. Conversely, if it predicts 1 but the true label is 0 (error = -1), we decrease weights. The bias adjusts the threshold too.

4. Repeat epochs

One full pass over the entire training set is called an epoch. Repeat until the perceptron makes zero errors on the training data — or you hit a maximum number of epochs. For linearly separable data (like OR and AND gates), the perceptron is guaranteed to converge.

Example: OR gate

The perceptron finds a line (or plane) that separates the 0 output from the 1 outputs.

Hands-on walkthrough

Time to write the actual Python code. We'll implement a Perceptron class with fit() and predict() methods, and train it on the OR gate.

Step 1: The Perceptron class

import numpy as np

class Perceptron:
    def __init__(self, learning_rate=0.1, epochs=100):
        self.learning_rate = learning_rate
        self.epochs = epochs
        self.weights = None
        self.bias = None

    def activate(self, z):
        """Step activation function: 1 if z > 0 else 0"""
        return 1 if z > 0 else 0

    def fit(self, X, y):
        n_samples, n_features = X.shape

        # Initialize weights and bias
        self.weights = np.zeros(n_features)
        self.bias = 0

        # Training loop
        for epoch in range(1, self.epochs + 1):
            errors = 0
            for xi, target in zip(X, y):
                # Weighted sum
                z = np.dot(xi, self.weights) + self.bias
                prediction = self.activate(z)

                # Delta rule
                error = target - prediction
                self.weights = self.weights + self.learning_rate * error * xi
                self.bias = self.bias + self.learning_rate * error

                # Count errors
                errors += int(error != 0)

            # Early stop if no errors
            if errors == 0:
                print(f"Converged after {epoch} epoch(s)")
                break

    def predict(self, X):
        z = np.dot(X, self.weights) + self.bias
        return np.array([self.activate(val) for val in z])

Step 2: Train the perceptron on the OR gate

# Training data: [A, B] -> OR result
X = np.array([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
])
y = np.array([0, 1, 1, 1])

# Create and train
perceptron = Perceptron(learning_rate=0.1, epochs=10)
perceptron.fit(X, y)

# Test predictions
print("Predictions:")
for xi, target in zip(X, y):
    pred = perceptron.predict(np.array([xi]))[0]
    print(f"{xi} -> predicted {pred}, actual {target}")

Expected output:

Converged after 1 epoch(s)
Predictions:
[0 0] -> predicted 0, actual 0
[0 1] -> predicted 1, actual 1
[1 0] -> predicted 1, actual 1
[1 1] -> predicted 1, actual 1

Step 3: Test on a non-separable case (XOR)

Now let's try a famous failure: the XOR gate. It's not linearly separable — no single straight line can separate the outputs.

X_xor = np.array([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
])
y_xor = np.array([0, 1, 1, 0])

perceptron_xor = Perceptron(learning_rate=0.1, epochs=100)
perceptron_xor.fit(X_xor, y_xor)

# Test
print("XOR predictions:")
for xi, target in zip(X_xor, y_xor):
    pred = perceptron_xor.predict(np.array([xi]))[0]
    print(f"{xi} -> predicted {pred}, actual {target}")

Expected output:

XOR predictions:
[0 0] -> predicted 1, actual 0
[0 1] -> predicted 0, actual 1
[1 0] -> predicted 1, actual 1
[1 1] -> predicted 0, actual 0

The perceptron never converges. This is a critical limitation — a single perceptron can only learn linearly separable patterns. Multi-layer perceptrons (and non-linear activation functions) solve this, which you'll study in later lessons.

Compare options / when to choose what

You've built a perceptron from scratch, but in practice you have several options. Here's a quick comparison:

When to choose what:

• Use from-scratch when you're learning or need full control to debug.
• Use scikit-learn when you need a quick, trustworthy linear classifier in a pipeline.
• Use deep learning frameworks when your problem clearly requires non-linearity.

Variations on the perceptron:

• Stochastic vs batch updates: The code above updates weights after each sample (stochastic). Batch updates compute the average error over the whole epoch before updating — smoother but slower convergence.
• Different activation functions: ReLU or sigmoid instead of a step function leads to logistic regression and gradient descent — the foundation of modern neural nets.
• Data normalization: If features have wildly different scales, training becomes unstable. Always scale inputs for faster convergence.

Troubleshooting & edge cases

The perceptron is simple, but you'll hit pitfalls quickly. Here are the most common issues and how to fix them.

1. The model never converges

Symptom: The error count never reaches zero, even after many epochs.

Cause: The data is not linearly separable, or the learning rate is too high/low.

Fix: - Check if the data is linearly separable (plot it or try simplifying). If not, you need a non-linear model. - Try a smaller or larger learning rate. 0.1 is a good default, but 0.01 or 0.5 may work better. - Increase the number of epochs.

2. Predictions are always 0 or always 1

Cause: The bias and weights are initialized poorly, or the learning rate is way too small so learning stalls.

Fix: Use random initialization of weights (not zeros) to break symmetry, and ensure learning rate isn't 0.001 — too small a step means no progress.

3. Feature scaling matters

Problem: If one feature is on a scale of 0-1000 and another is 0-1, the weight updates become dominated by the large-scale feature.

Fix: Normalize or standardize features (mean 0, variance 1) before training.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

4. Confusing the step function threshold

Symptom: Model works on training data but fails on new data.

Cause: The step function z > 0 assumes a symmetric threshold. If you'd rather have a different bias, you can adjust it, but keep it consistent.

5. Using an unsupported label format

Symptom: TypeError or weird values.

Cause: The delta rule expects y to be exactly 0 or 1. Ensure your labels are integers/bools, not strings or floats like 1.0 (which still works, but be careful).

Pro tip: Always unit-test your perceptron on a known linearly separable problem, like the OR gate, before tackling real data. If it can't learn the OR gate, your implementation has a bug.

What you learned & what's next

Let's recap what you've accomplished in this lesson:

• You now understand what a perceptron is — a single neuron that makes binary decisions using a weighted sum and a step activation function.
• You can explain the perceptron learning rule (the delta rule) and trace how weights and bias are updated to minimize error.
• You successfully implemented a simple perceptron from scratch in Python, training it on the OR gate and witnessing its failure on XOR.
• You can troubleshoot common issues like non-convergence and know when the perceptron is the right tool versus when you need deeper models.

This is the foundational building block for everything that follows. In the next lesson, you'll take this single neuron and stack them into a multi-layer perceptron (MLP), introducing hidden layers and non-linear activation functions like ReLU — which will finally solve the XOR problem and unlock the power of deep learning.

You're no longer just a user of AI libraries — you're an engineer who understands what's under the hood. Keep going!