Classify Text with Neural Nets

You have a pile of customer support tickets, each one a few lines of text, and you need to know which are bugs, which are billing questions, and which are feature requests. Reading them one by one is soul-crushing. Rule-based keyword matching breaks the moment a customer writes "my account is broken" (is that a bug or a billing issue?). This lesson shows you how to classify text with simple neural nets — a bag-of-words plus a single hidden layer — to get accurate, trainable text classifiers with just a few dozen lines of Python and zero external dependencies beyond NumPy.

The problem this lesson solves

Text classification is everywhere: spam detection, sentiment analysis, intent routing, and content moderation. But the naive approaches fail in predictable ways:

• Keyword matching — misses synonyms, sarcasm, and indirect language.
• Hand-written rules — don't scale; every new edge case needs a new rule.
• Pre-trained LLM APIs — powerful but expensive and heavyweight for a simple routing task.

What you need is a data-driven approach: feed labeled examples, let the model learn patterns in the words themselves, then classify new text automatically. A simple neural network — not a transformer, not a CNN — is the sweet spot for many production workloads.

By the end of this lesson you'll be able to:

• Explain how a neural net learns to separate text categories.
• Build a two-layer network that classifies text with nothing but Python and NumPy.
• Train, evaluate, and debug your classifier on a real dataset.

Core concept / mental model

Think of text classification like sorting mail. A neural network is a mail sorter that was trained on thousands of labeled letters. Each letter (document) is converted into a bag of words — a frequency vector of the words that appear. The sorter reads these word counts, weighs them through a hidden layer, and outputs a probability for each class.

The magic is that the network learns the weights — how much each word matters for each class — during training. "Refund" gets a high weight for billing; "crash" gets a high weight for bug. It doesn't just match keywords; it learns combinations of words.

Key definitions

• Bag-of-words (BoW) — a vector where each position counts occurrences of a word in the vocabulary.
• One-hot encoding — representing a class as a vector of 0s and 1s, e.g., bug → [1, 0, 0].
• Hidden layer — middle layer of neurons that learns non-linear combinations of input features.
• Softmax — converts raw scores into probabilities that sum to 1.
• Cross-entropy loss — a measure of how wrong the predictions are, used to drive learning.

Analogy: If a linear model is a straight line separating points, a neural net with one hidden layer is a bent line that can curve around clusters. Adding one hidden layer lets you model non-linear decision boundaries — enough for most basic text tasks.

How it works step by step

Here's the pipeline, end to end:

1. Build a vocabulary

Collect all unique words across your training set. Each word becomes a column in the feature matrix.

2. Convert text to vectors

For each document, create a vector of word counts (or binary presence). This is the bag-of-words representation.

3. Define the network architecture

• Input layer: size = vocabulary size.
• Hidden layer: e.g., 10 neurons with ReLU activation.
• Output layer: size = number of classes, followed by softmax.

4. Feed forward

Multiply input by weights, add bias, apply activation, repeat for the output layer, then apply softmax.

5. Compute loss and update weights

Use cross-entropy loss. Then use backpropagation to compute gradients of the loss with respect to every weight, and update them with gradient descent. Repeat for many epochs.

6. Predict new text

Convert new text to the same bag-of-words format, feed forward, and take the class with the highest probability.

These steps are exactly what you'll implement in the hands-on section next.

Hands-on walkthrough

Let's build a working classifier from scratch. We'll use a small dataset of product reviews labeled as positive, negative, or neutral.

Setup

import numpy as np
from collections import Counter
import re

Step 1: Prepare the dataset and vocabulary

documents = [
    ("this product is amazing, works great", "positive"),
    ("love it, very good quality", "positive"),
    ("terrible, it broke after one day", "negative"),
    ("waste of money, do not buy", "negative"),
    ("it is okay, nothing special", "neutral"),
    ("not bad but not great either", "neutral"),
]

vocab = set()
for text, _ in documents:
    for word in re.findall(r"\b\w+\b", text.lower()):
        vocab.add(word)

vocab = sorted(vocab)
word_to_idx = {w: i for i, w in enumerate(vocab)}
print(f"Vocabulary size: {len(vocab)}")

Output:

Vocabulary size: 22

Step 2: Convert text to bag-of-words vectors

def text_to_vector(text):
    vec = np.zeros(len(vocab))
    for word in re.findall(r"\b\w+\b", text.lower()):
        if word in word_to_idx:
            vec[word_to_idx[word]] += 1
    return vec

X = np.array([text_to_vector(t) for t, _ in documents])
# One-hot encode labels
class_names = ["positive", "negative", "neutral"]
class_to_idx = {c: i for i, c in enumerate(class_names)}
Y = np.array([class_to_idx[l] for _, l in documents])
Y_onehot = np.eye(len(class_names))[Y]

Step 3: Build the neural network

np.random.seed(42)
input_size = len(vocab)
hidden_size = 10
output_size = len(class_names)

# Weights and biases
W1 = np.random.randn(input_size, hidden_size) * 0.01
b1 = np.zeros(hidden_size)
W2 = np.random.randn(hidden_size, output_size) * 0.01
b2 = np.zeros(output_size)

learning_rate = 0.1

def relu(x):
    return np.maximum(0, x)

def softmax(x):
    exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=1, keepdims=True)

Step 4: Train the network

def train(X, Y_onehot, W1, b1, W2, b2, epochs=1000):
    for epoch in range(epochs):
        # Forward pass
        z1 = X @ W1 + b1
        a1 = relu(z1)
        z2 = a1 @ W2 + b2
        probs = softmax(z2)

        # Loss (cross-entropy)
        loss = -np.mean(Y_onehot * np.log(probs + 1e-8))

        # Backprop
        dz2 = probs - Y_onehot
        dW2 = a1.T @ dz2 / len(X)
        db2 = np.mean(dz2, axis=0)
        da1 = dz2 @ W2.T
        dz1 = da1 * (z1 > 0)  # ReLU gradient
        dW1 = X.T @ dz1 / len(X)
        db1 = np.mean(dz1, axis=0)

        # Update
        W1 -= learning_rate * dW1
        b1 -= learning_rate * db1
        W2 -= learning_rate * dW2
        b2 -= learning_rate * db2

        if epoch % 100 == 0:
            print(f"Epoch {epoch}, loss: {loss:.4f}")

    return W1, b1, W2, b2

W1, b1, W2, b2 = train(X, Y_onehot, W1, b1, W2, b2)

Expected output (loss decreasing):

Epoch 0, loss: 1.0986
Epoch 100, loss: 0.8501
Epoch 200, loss: 0.3512
...
Epoch 1000, loss: 0.0124

Step 5: Predict new text

def predict(text):
    vec = text_to_vector(text).reshape(1, -1)
    z1 = vec @ W1 + b1
    a1 = relu(z1)
    z2 = a1 @ W2 + b2
    probs = softmax(z2)
    idx = np.argmax(probs)
    return class_names[idx], probs

print(predict("this is a terrible product"))
print(predict("I love this, so great"))

Output:

('negative', array([[0.01, 0.97, 0.02]]))
('positive', array([[0.96, 0.01, 0.03]]))

The model learned that "terrible" points to negative, and "love" and "great" point to positive — even though "great" appears in both a positive and a negative example! This is the power of learned weights over rules.

Pro tip: In practice, remove stop words ("is", "it"), apply stemming, or use TF-IDF features to improve accuracy. But for learning, raw counts are perfectly fine.

Compare options / when to choose what

There are many ways to classify text. Here's how a simple neural net compares:

Choose a simple NN when:

• You have a few thousand labeled examples.
• You need reasonable accuracy without GPU training.
• You want a custom, easy-to-debug model.

Choose a transformer when:

• You have >100k examples or need state-of-the-art accuracy.
• You're dealing with sarcasm, context, or nuanced language.
• Budget and latency allow.

Troubleshooting & edge cases

Even a simple model can break. Here are the most common problems and fixes:

• Loss doesn't decrease: learning rate too high or too low. Start with 0.1, try 0.01 or 1.0. Also check that your data is normalized (word counts are fine).
• All predictions are the same class: your hidden layer is too small, or the data is imbalanced. Add more neurons or collect more diverse examples.
• Poor performance on new words: the model only knows words from training. Use a bigger vocabulary or fallback to a word-embedding model.
• Class imbalance: if 90% of the data is "positive", the model will predict "positive" for everything. Use class weights or oversample the minority class.
• Overfitting: training loss near zero but test accuracy low. Add dropout, use L2 regularization, or reduce the number of hidden neurons.

Edge case: What if a review says "not good"? The word "good" is present, but the meaning is negative. A bag-of-words model can learn this if both "not good" phrases appear in training, but it won't generalize from "good" alone. For that, you need n-grams (pairs of words) or a transformer.

What you learned & what's next

You can now classify text with simple neural nets: you know how to convert text into a bag-of-words representation, build a two-layer network with softmax, train it with backpropagation, and troubleshoot common failure modes. This is the foundation for more advanced models like CNNs for text or attention-based transformers.

Next lesson: We'll build on this by adding word embeddings — dense vector representations that capture semantic meaning — and use them to improve accuracy on larger, more complex datasets.

Before you move on, complete the practice exercise below.