Activate Neurons with ReLU and Sigmoid

Building a neural network without activation functions is like building a symphony orchestra where every musician plays the same note at the exact same volume. Your model becomes a linear regression in disguise, no matter how many layers you stack — it simply cannot learn complex patterns. This lesson cuts through the confusion and shows you exactly how to activate neurons with ReLU and sigmoid, the two most essential activation functions in applied AI engineering. You'll not only understand why they matter but also when to use each one, complete with hands-on Python examples you can run today.

Let's dive in and unlock the true power of your neurons.

The problem this lesson solves

Imagine training a deep network to classify images of cats and dogs. You feed it raw pixels, and you expect it to learn features like edges, fur texture, and eye shapes. Without activation functions, each neuron computes a simple weighted sum:

[ z = w \cdot x + b ]

That output is then passed to the next layer, which computes another weighted sum. After three layers, your entire network still produces a linear combination of the input. No matter how deep, it can never approximate a curve like the boundary between a cat's ear and the background. That is the core pain: linear-only models have a severely limited capacity — they simply cannot express the complex, non-linear relationships that real-world data demands.

Activation functions inject non-linearity into every neuron. They decide whether a neuron should "fire" based on its input, mimicking the way biological neurons either activate or stay silent. With the right activation, a network can approximate virtually any function — this is the universal approximation theorem at work. Without it, you're stuck with a model that cannot learn anything beyond a straight line, no matter how massive your dataset or how long you train.

Core concept / mental model

Think of each neuron as a gatekeeper. The gatekeeper receives a raw score (the weighted sum (z)), and the activation function decides how much of that score should pass through to the next layer. It's like a faucet that adjusts water flow — not just on/off, but with nuanced control.

Sigmoid is the classic smooth gate. It squeezes any real number into the range ((0, 1)), producing an S-shaped curve. It's perfect for probabilities — think of it as the neuron saying, "I'm 0.8 confident this is a cat." The math is simple:

[ \sigma(z) = \frac{1}{1 + e^{-z}} ]

ReLU (Rectified Linear Unit) is the modern, fast gate. It outputs the input directly if positive and zero otherwise:

[ f(z) = \max(0, z) ]

Picture a gate that's fully closed for negative values but opens linearly for positive ones. It's computationally cheap and works remarkably well in deep networks, which is why it's the default choice in most modern architectures.

Key differences at a glance:

• Output range: Sigmoid outputs ((0,1)), ReLU outputs ([0, \infty)).
• Gradient behavior: Sigmoid can suffer from vanishing gradients; ReLU helps mitigate that.
• Use cases: Sigmoid for binary classification probability; ReLU for hidden layers.

How it works step by step

1. Compute the weighted sum (z) for each neuron: (z = w \cdot x + b), where (w) are the weights, (x) the inputs, and (b) the bias.
2. Apply the activation function to (z): this is where non-linearity enters. - For sigmoid, output (\sigma(z)) — a value between 0 and 1. - For ReLU, output (\max(0, z)) — a non-negative real number.
3. Pass the activated output to the next layer as input.
4. During backpropagation, the derivative of the activation function determines how much to update the weights. - For sigmoid: (\sigma'(z) = \sigma(z)(1 - \sigma(z))). - For ReLU: (f'(z) = 1) if (z > 0), else 0.

This cause-and-effect chain is what lets the network learn: the activation function shapes the signal, and the gradient shapes the learning.

Hands-on walkthrough

Let's bring these functions to life in Python. You'll need NumPy — install it with pip install numpy if you haven't already.

Example 1: Implement sigmoid and ReLU from scratch

import numpy as np

def sigmoid(z):
    """Sigmoid activation function."""
    return 1 / (1 + np.exp(-z))

def relu(z):
    """ReLU activation function."""
    return np.maximum(0, z)

# Test with a range of inputs
z_vals = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
print("Input:", z_vals)
print("Sigmoid:", sigmoid(z_vals))
print("ReLU:", relu(z_vals))

Expected output:

Input: [-2.  -0.5  0.   0.5  2. ]
Sigmoid: [0.11920292 0.37754067 0.5        0.62245933 0.88079708]
ReLU: [0.  0.  0.  0.5 2. ]

Notice how sigmoid squashes values into ((0,1)), while ReLU zeroes out negatives and passes positives through unchanged.

Example 2: Apply activations in a tiny neural network layer

import numpy as np

def dense_layer(X, W, b, activation):
    """A single dense layer with a specified activation."""
    z = np.dot(X, W) + b
    return activation(z), z

# Input features (3 samples, 2 features)
X = np.array([[0.5, 1.2],
              [1.0, -0.4],
              [-0.3, 2.0]])

# Weights (2 inputs -> 3 neurons) and bias
W = np.array([[0.2, -1.1, 0.5],
              [0.7, 0.3, -0.2]])
b = np.array([0.1, -0.2, 0.3])

# Apply sigmoid and ReLU activations
sigmoid_out, _ = dense_layer(X, W, b, sigmoid)
relu_out, _ = dense_layer(X, W, b, relu)

print("Sigmoid outputs:\n", sigmoid_out)
print("ReLU outputs:\n", relu_out)

Expected output (values will vary slightly due to floating-point):

Sigmoid outputs:
 [[0.77948942 0.3340163  0.63913176]
 [0.68572274 0.57261757 0.64565631]
 [0.59552026 0.2340056  0.53977172]]
ReLU outputs:
 [[0.8        0.         0.65      ]
 [0.4        0.         0.5       ]
 [0.         0.         0.6       ]]

Here you see each neuron's output after activation. In a real network, these values become the input to the next layer.

Example 3: Visualizing the vanishing gradient problem

import numpy as np
import matplotlib.pyplot as plt

z = np.linspace(-5, 5, 100)
sigmoid_deriv = sigmoid(z) * (1 - sigmoid(z))
relu_deriv = np.where(z > 0, 1.0, 0.0)

plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
plt.plot(z, sigmoid_deriv)
plt.title("Sigmoid derivative")
plt.xlabel("z"); plt.ylabel("Gradient")

plt.subplot(1, 2, 2)
plt.plot(z, relu_deriv)
plt.title("ReLU derivative")
plt.xlabel("z"); plt.ylabel("Gradient")

plt.tight_layout()
plt.show()

The sigmoid derivative peaks at 0.25 and quickly approaches zero for large or small (z). ReLU's derivative is exactly 1 for positive inputs, keeping gradients from vanishing. This is why deep networks often favor ReLU.

Compare options / when to choose what

When to choose sigmoid: - Final layer for binary classification (probability output). - When you need bounded outputs, like in certain gating mechanisms (LSTM gates).

When to choose ReLU: - Hidden layers in most deep networks. - When training speed matters and you have a modern optimizer. - In convolutional neural networks (CNNs) and most pre-trained models.

Variations worth knowing: - Leaky ReLU: allows small negative slope (e.g., 0.01) to avoid dead neurons. - Tanh: similar to sigmoid but outputs (-1, 1); often used in recurrent networks. - Softmax: generalization of sigmoid for multi-class classification.

Pro tip: For most modern neural networks, start with ReLU for hidden layers. Only use sigmoid when you need a probability output in binary classification.

Troubleshooting & edge cases

Problem: Gradients vanish during training.

You train a deep network with sigmoid activations and notice the loss barely decreases. This is the classic vanishing gradient.

Fix: Switch to ReLU (or its variants) for hidden layers. Ensure weights are initialized properly (e.g., He initialization for ReLU, Xavier for sigmoid).

Problem: Dead ReLU neurons.

Some neurons output zero for all inputs after training — the gradient never flows, so they stop learning. This happens if many inputs are negative.

Fix: Use Leaky ReLU or ReLU with a small positive slope. Reduce learning rate if neurons die early.

Problem: Sigmoid outputting 0.5 for all inputs.

This often occurs when weights are near zero, making (z) close to 0. The network isn't learning.

Fix: Re-initialize weights with a larger range or use a better optimizer like Adam.

Edge case: Overflow in sigmoid for large negative values.

np.exp(-z) can overflow to zero, causing division issues.

Fix: Use a stable implementation:

def stable_sigmoid(z):
    return np.where(z >= 0, 1 / (1 + np.exp(-z)), np.exp(z) / (1 + np.exp(z)))

This avoids overflow for very negative (z).

What you learned & what's next

You now understand the core idea behind activating neurons with ReLU and sigmoid. You saw how these functions introduce non-linearity, and you completed practical exercises implementing them from scratch and applying them in a dense layer. You can now select the right activation based on your problem: ReLU for hidden layers, sigmoid for binary classification output. Keep this mental model in mind — it's foundational for every neural network you'll build.

Next lesson in the track: You'll move on to building multi-layer perceptrons and understanding backpropagation. With ReLU and sigmoid under your belt, you're ready to stack layers and train deeper models with confidence.