Reproducible Training Scripts

You've just spent the weekend tuning a model, only to discover that rerunning your training script produces a different accuracy score — or worse, a completely different model. The pain is real: unreproducible training scripts waste hours, erode trust, and make collaboration nearly impossible. In this lesson, you'll learn how to create reproducible training scripts that behave consistently across environments, so you can debug faster, share confidently, and move from 'it worked on my machine' to 'it works everywhere'.

The problem this lesson solves

Machine learning training is full of randomness: weight initialization, data shuffling, dropout layers, and even the order floating-point operations are performed. Without deliberate control, every run of your script becomes a unique snowflake. This causes irreproducible results — the same code, data, and parameters produce different outcomes on different machines or even on the same machine.

The consequences of irreproducibility are serious:

• Debugging nightmares: You can't tell if a change in accuracy came from your code edit or from random noise.
• Broken collaboration: Teammates can't validate your results if they can't reproduce them.
• Failed audits: In regulated industries (finance, healthcare), you may need to prove exactly how a model was trained.

Real-world example: In 2019, Google's researchers released a paper on BERT reproducibility and found that even with the same code and data, fine-tuning results varied by up to 10% accuracy depending on the random seed. This variability, when unmanaged, makes applied AI engineering unreliable.

Core concept / mental model

Think of a training script as a recipe. A recipe must list the ingredients (data, hyperparameters) and the steps (training loop). But if you don't specify the oven temperature or the altitude, the cake may rise differently each time. In machine learning, the 'altitude' is your hardware (GPU, CPU count), the 'oven temperature' is the random seed, and the 'baking time' is the number of epochs.

Reproducibility means that if you run the same recipe (script) on the same 'kitchen' (environment) under the same conditions, you get the same cake (model). In practice, this translates to three pillars:

1. Determinism: The same operations produce the same results every time.
2. Dependency locking: The software versions (Python, NumPy, PyTorch, etc.) are frozen.
3. Environment consistency: The hardware and OS don't change between runs, or you document them.

A key concept here is the random seed. By setting a seed for every library that uses randomness (Python's random, NumPy, PyTorch, TensorFlow), you force the initial weights and data shuffles to be identical across runs. But seeds alone are not enough — you must also disable GPU non-determinism and pin your library versions.

How it works step by step

To create reproducible training scripts, follow this logical sequence:

1. Set all random seeds at the very start of the script. Use a common seed, e.g., 42, for random, numpy, and your deep learning framework.
2. Ensure deterministic algorithms are used. For PyTorch, set torch.backends.cudnn.deterministic = True and torch.use_deterministic_algorithms(True) if possible. For TensorFlow, set tf.random.set_seed() and disable the XLA auto-tuning.
3. Pin dependency versions in a requirements.txt or pyproject.toml with exact versions (e.g., torch==2.1.0, not torch>=2.0).
4. Freeze the environment — the best approach is to use a container (Docker) or a virtual environment that you can recreate exactly.
5. Shuffle your data with a seeded RandomState rather than relying on global random state, to avoid interference between steps.
6. Log everything — hyperparameters, dataset version, seed, and commit hash — so you can later map what produced a given result.
7. Validate reproducibility by running the script twice and comparing the metrics (e.g., loss curves, final accuracy). If they differ, investigate the source.

The cause-and-effect relationship: Setting a seed controls the initial state, but without deterministic algorithms, GPU operations may still produce different results. Thus, the seed + deterministic flags + pinned dependencies work together as a chain of reproducibility.

Hands-on walkthrough

Let's build a minimal, reproducible training script for a neural network using PyTorch. We'll focus on the key elements that make it reproducible.

Step 1: Set the seed and deterministic flags

import random
import numpy as np
import torch


def set_seed(seed: int = 42) -> None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    # Ensure deterministic behavior on supported operations
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    # Raise errors on non-deterministic ops (use with caution)
    torch.use_deterministic_algorithms(True, warn_only=True)

set_seed(42)

Step 2: Use a seeded DataLoader with deterministic workers

Data shuffling happens inside the DataLoader, so we must also seed its random state. In PyTorch, you can pass a generator to the DataLoader and set the number of workers to a fixed value.

from torch.utils.data import DataLoader, TensorDataset

# Sample data
data = torch.randn(100, 10)
labels = torch.randint(0, 2, (100,))
dataset = TensorDataset(data, labels)

# Create a generator with the same seed
g = torch.Generator()
g.manual_seed(42)

# Shuffle with the generator, and pin worker count
train_loader = DataLoader(dataset, batch_size=16, shuffle=True, generator=g, num_workers=0)

Setting num_workers=0 ensures no parallel worker processes introduce subtle nondeterminism. For large datasets, you can use a deterministic sampler instead.

Step 3: Pin dependencies and log config

Create a requirements.txt with exact versions:

pip freeze > requirements.txt

This yields lines like torch==2.5.1 — commit this to version control. Better yet, use a Dockerfile to freeze the entire OS environment.

Inside your script, log the seed, hyperparameters, and dataset hash:

import hashlib
import json

config = {
    "seed": 42,
    "learning_rate": 0.001,
    "batch_size": 16,
    "epochs": 5,
    "dataset_sha256": hashlib.sha256(b"your_data_file").hexdigest()[:10],
}
print(json.dumps(config, indent=2))
# Save to a file for later comparison
with open("training_config.json", "w") as f:
    json.dump(config, f)

Step 4: Run twice and compare

Simply run python train.py twice, and compare the final loss or accuracy. If they match exactly, your script is reproducible. If not, go to the troubleshooting section.

Expected output (from the logging script):

{
  "seed": 42,
  "learning_rate": 0.001,
  "batch_size": 16,
  "epochs": 5,
  "dataset_sha256": "abc123def0"
}

Compare options / when to choose what

You can achieve reproducibility through different strategies. Here's a comparison:

Pro tip: Start with seeds + pinned dependencies for simplicity, then adopt Docker when you need to reproduce results across machines or after OS updates.

Troubleshooting & edge cases

Even with seeds set, you might still see variations. Here are common traps:

• Cudnn non-deterministic algorithms: Even with benchmark=False, some operations (like certain convolution algos) are nondeterministic. Use torch.use_deterministic_algorithms(True) to catch them, but note it may raise errors; use warn_only=True in development.
• Multi-GPU training: Setting a seed on one GPU doesn't make distributed training deterministic. You must set the seed on all processes and use torch.nn.DistributedDataParallel with careful ordering.
• DataLoader workers: num_workers > 0 introduces nondeterminism in shuffle order. Either set shuffle=False with a custom seeded sampler or use num_workers=0 for critical runs.
• Running on CPU vs GPU: CPU and GPU have different floating-point rounding, so results may differ. Document your hardware in the config.
• Missing dependencies: If torch==2.5.1 is not installed, your script fails before running. Use a Docker image with the same versions to avoid this.

If your two runs differ, check: nvidia-smi output (are you using the same GPU?), the requirements.txt diff, and enable the deterministic algorithms flag to see which operation is causing the issue.

What you learned & what's next

You now understand how to create reproducible training scripts — you can set random seeds, force deterministic algorithms, pin dependencies, log configuration, and validate reproducibility by running your script twice. You also know the trade-offs between simple seeds and full containerization. These skills are essential for any applied AI engineer: they let you debug effectively, share work confidently, and meet compliance requirements.

Next steps: In the next lesson, you'll explore model evaluation best practices — how to measure performance reliably once your training is reproducible. With a solid base, you'll be able to compare models without the confound of random variation.