Design Your First Feedforward Network

You understand convolutions, embeddings, and transformers, but when you need to build a custom model from scratch for a real-world tabular dataset, a feedforward network (also called a dense network or multilayer perceptron) is often the sharpest tool in the box. This lesson removes the mystery: you’ll learn exactly what a feedforward network is, how it processes information layer by layer, and how to design and train one in Python using PyTorch. By the end, you’ll be able to translate a business problem into a trained model that predicts outcomes reliably.

The problem this lesson solves

Many AI tutorials jump straight to convolutional or recurrent architectures, but most real-world datasets — sales figures, sensor readings, credit risk, customer churn — are tabular. Without a solid handle on the humble feedforward network, you’ll find yourself overcomplicating simple problems or worse, using a transformer when a small dense model would be faster and more interpretable.

This lesson addresses three practical pains:

• Design paralysis: You have the raw data but no idea how many layers or neurons you need.
• Training instability: Your loss won’t converge, or it explodes to NaN, leaving you guessing why.
• Poor generalization: The model memorizes the training data but fails on unseen examples.

We’ll tackle these head-on by building a feedforward network from scratch for a synthetic regression task, then discuss when to stick with this architecture versus alternatives.

Core concept / mental model

Think of a feedforward network as a multi-stage refining pipeline. Raw data enters at the input stage, gets refined in hidden stages, and exits as a prediction at the output stage. Each stage consists of neurons that hold a single number. Each neuron receives numbers from the previous stage, multiplies them by weights, adds a bias, then passes the sum through an activation function that introduces nonlinearity.

The key mental model is the flow of information: data moves strictly forward — no loops, no feedback. That’s what makes it feedforward.

Here’s the anatomy in words:

• Input layer: The number of neurons equals the number of features in your dataset (e.g., 10 features → 10 input neurons).
• Hidden layers: One or more layers that learn intermediate representations. Each hidden layer has a chosen width (number of neurons) and an activation function (usually ReLU).
• Output layer: The final prediction. For regression, it’s a single neuron with no activation (or linear). For classification, it’s one neuron per class with a softmax activation.

A network with two hidden layers of sizes 64 and 32 can be denoted as 10 → 64 → 32 → 1.

Pro tip: Think of each hidden layer as a feature extractor. The first hidden layer might detect simple patterns, the second layer combines them into more abstract features, and so on.

How it works step by step

Designing a feedforward network is a repeatable process. Follow these steps:

1. Define the task — Is it regression (predict a number) or classification (predict a class)? This sets the output layer.
2. Preprocess the data — Normalize numeric features to zero mean and unit variance. Handle missing values. Encode categorical variables.
3. Choose the architecture — Start simple: one hidden layer with 16–64 neurons. Increase depth only if needed.
4. Pick an activation function — ReLU for hidden layers (avoids vanishing gradients), linear for regression output, softmax for classification.
5. Select a loss function — Mean squared error (MSE) for regression, cross-entropy for classification.
6. Choose an optimizer — Adam is a robust default.
7. Train the model — Iterate over batches, compute loss, backpropagate, update weights.
8. Evaluate on a held-out test set — Use metrics like R² for regression or accuracy for classification.

Why normalization matters

Feedforward networks are sensitive to the scale of inputs. If one feature ranges 0–1 and another ranges 0–1,000,000, the network will struggle to learn. Normalizing each feature (subtract mean, divide by standard deviation) så that all inputs have a similar scale helps gradient descent converge faster and more reliably.

Hands-on walkthrough

Let’s build a feedforward network that predicts a continuous target from 8 features. We’ll use scikit-learn to generate a synthetic dataset, PyTorch to build and train the network, and matplotlib to visualize the loss curve. Make sure you have torch and scikit-learn installed.

Step 1: Generate synthetic data

import torch
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Generate synthetic regression data
X, y = make_regression(n_samples=1000, n_features=8, noise=0.2, random_state=42)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Normalize features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Convert to torch tensors
train_data = torch.tensor(X_train, dtype=torch.float32)
test_data = torch.tensor(X_test, dtype=torch.float32)
train_targets = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
test_targets = torch.tensor(y_test, dtype=torch.float32).unsqueeze(1)

Step 2: Define the network

We’ll build a small network with one hidden layer of 32 neurons and ReLU activation.

import torch.nn as nn

class FeedForwardNN(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim=1):
        super().__init__()
        self.layer1 = nn.Linear(input_dim, hidden_dim)
        self.activation = nn.ReLU()
        self.layer2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        x = self.layer1(x)
        x = self.activation(x)
        x = self.layer2(x)
        return x

# Instantiate model
model = FeedForwardNN(input_dim=8, hidden_dim=32)
print(model)

Output (parameter shapes may vary):

FeedForwardNN(
  (layer1): Linear(in_features=8, out_features=32, bias=True)
  (activation): ReLU()
  (layer2): Linear(in_features=32, out_features=1, bias=True)
)

Step 3: Train the model

We’ll use the Adam optimizer and mean squared error loss. Train for 100 epochs.

import torch.optim as optim

loss_fn = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

epochs = 100
for epoch in range(epochs):
    model.train()
    optimizer.zero_grad()
    predictions = model(train_data)
    loss = loss_fn(predictions, train_targets)
    loss.backward()
    optimizer.step()

    if (epoch+1) % 10 == 0:
        print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

Sample output:

Epoch 10, Loss: 2817.3423
Epoch 20, Loss: 1865.1104
...
Epoch 100, Loss: 63.1520

Step 4: Evaluate on test set

model.eval()
with torch.no_grad():
    y_pred = model(test_data)
    mse = loss_fn(y_pred, test_targets)
    print(f"Test MSE: {mse.item():.4f}")

# Show first few predictions
for pred, actual in zip(y_pred[:5], test_targets[:5]):
    print(f"Predicted: {pred.item():.2f}, Actual: {actual.item():.2f}")

Sample output:

Test MSE: 58.3421
Predicted: 10.34, Actual: 12.17
Predicted: -5.23, Actual: -6.01
Predicted: 0.45, Actual: 1.22
Predicted: 8.91, Actual: 7.85
Predicted: 3.67, Actual: 2.99

The test MSE is close to the training loss, indicating the model generalizes well.

Compare options / when to choose what

Feedforward networks aren’t the only choice for tabular data. Here’s how they stack up against common alternatives:

Rule of thumb: Start with a feedforward network if your data is tabular and you plan to iterate on the model. If you need a quick benchmark, train a random forest first. If your data has spatial or sequential structure, switch to specialized architectures.

Troubleshooting & edge cases

• Loss not decreasing → Check your learning rate. If too high, set it to 0.001 or lower. If too low, increase it gradually. Also verify that the data is normalized.
• Loss becomes NaN → This is often caused by exploding gradients. Lower the learning rate, add gradient clipping (torch.nn.utils.clip_grad_norm_), or use a different activation like Leaky ReLU.
• Overfitting (train loss low, test loss high) → Add dropout layers, increase training data, reduce model capacity, or apply L2 regularization (weight_decay in Adam).
• Input shape mismatch → Ensure your input tensor has shape (batch_size, input_dim). If you have a 1D tensor, add a batch dimension with unsqueeze(0).
• Wrong output dimension → For classification, the output should have the same number of neurons as classes. For regression, one neuron is expected.
• Normalization leakage → Fit the scaler on the training set only, never on the test set. Applying the scaler to the entire dataset introduces information leakage and skews evaluation.

What you learned & what's next

You now understand the anatomy of a feedforward network: input, hidden, and output layers, activation functions, and the forward pass. You can design a simple network, train it with PyTorch, and evaluate its performance on a regression task. You’ve also seen how to avoid common pitfalls like unnormalized data and exploding gradients.

This foundational knowledge sets you up for the next lesson in the Applied AI engineering track: building a deep neural network with dropout and regularization to improve generalization on more complex datasets. With a firm grasp of feedforward networks, you’ll tackle deeper architectures with confidence.

Now, take a moment to practice: try varying the hidden layer size to 16 or 128 and see how the test MSE changes. Observe the effect of learning rate on convergence. These small experiments will cement your understanding of how architectural choices impact model performance.