Track Experiments with Weights & Biases

You've just spent 6 hours fine-tuning an LLM, but when you open the training logs, you're staring at a wall of console output that scrolls past too fast to read. Which hyperparameter combination gave you that 72% accuracy? You don't remember — and your results.json files are buried in a dozen folders with names like run_21_final_v3. If this sounds familiar, you're hitting the exact pain this lesson solves: tracking experiments in a way that's searchable, comparable, and shareable. Weights & Biases (W&B) gives you a central dashboard for every run, metric, and artifact, so you can stop guessing and start engineering.

The problem this lesson solves

Manually tracking ML experiments is a recipe for chaos. Here's what happens without a proper system:

• You lose context — a run's hyperparameters, code version, and dataset hash are scattered across notes, git commits, and your memory.
• You can't compare fairly — when you eyeball two runs side by side, you're often comparing different seeds, batch sizes, or preprocessing steps without realizing it.
• You waste compute — you retrain a model because you can't find the artifact from last week's best run.
• You can't collaborate — teammates ask, "Which run used the 7B model?" and you send them a screenshot of a terminal.

This is especially painful in LLM workflows, where a single fine-tuning run can cost hours and dollars. Every minute spent hunting for a metric is a minute you're not improving your model. Tools like W&B solve this by creating a single source of truth for every experiment — from quick prototype to large-scale sweep.

Core concept / mental model

Think of W&B as a version control system for experiments. Just as Git tracks changes to your code, W&B tracks changes to your data pipeline, model weights, and metrics. Each time you run a script, W&B creates a run — a lightweight object that collects:

• Hyperparameters (config)
• Metrics (loss, accuracy, etc.)
• System resources (GPU utilization, memory)
• Artifacts (model checkpoints, datasets)
• Source code and environment (git commit, Python packages)

You interact with W&B through a Python SDK. A typical flow looks like:

1. Initialize a run with wandb.init().
2. Log hyperparameters with config.
3. Log metrics in your training loop with wandb.log().
4. Watch the dashboard update in real time.
5. Compare runs and download artifacts later.

The dashboard lives on W&B's cloud (or your self-hosted instance), so you can access it from anywhere and share links with your team. The key idea is structured logging — instead of relying on console output, you send data to a central store that's designed for querying and visualization.

How it works step by step

Here's the mental workflow for integrating W&B into any experiment:

1. Install and authenticate — pip install wandb and log in (free for personal use).
2. Initialize a run — give it a name, project, and any config.
3. Define your config — all hyperparameters live in a dictionary that W&B stores and versions.
4. Log metrics inside the loop — at each step (epoch, batch), call wandb.log().
5. Log artifacts — when training finishes, save the model weights and any plots.
6. Compare runs — use the dashboard to overlay metrics, find the best config, and share results.

Pro tip: Run wandb.login() in a terminal once. The SDK stores your API key locally, so you won't be prompted in every script.

The power of W&B comes from its automated tracking — it hooks into your training loop without requiring you to change your logic. You just add a few lines, and the tool handles the rest.

Hands-on walkthrough

Let's build a complete example. First, install W&B:

pip install wandb
wandb login  # paste your API key when prompted

Logging metrics during a training loop

Here's a minimal script that tracks an LLM fine-tuning run. We'll log loss and learning rate at each epoch:

import wandb

# Initialize a run
wandb.init(
    project="llm-finetune",
    config={
        "batch_size": 16,
        "learning_rate": 1e-5,
        "epochs": 3,
        "model": "gpt2"
    }
)

# Simulate training
for epoch in range(wandb.config.epochs):
    for step in range(100):
        loss = 1.0 / (step + 1) + 0.1 * epoch
        # Log metrics at each step
        wandb.log({"epoch": epoch, "step": step, "loss": loss})

print("Training complete!")

When you run this, you'll see a URL in the output — open it to watch the loss curve update live.

Adding a custom metric and a plot

Now let's track accuracy and log a confusion matrix plot:

import wandb
import numpy as np

wandb.init(project="llm-eval", config={"seed": 42})

# Simulate evaluation results
num_classes = 5
y_true = np.random.randint(0, num_classes, size=100)
y_pred = np.random.randint(0, num_classes, size=100)
accuracy = (y_true == y_pred).mean()

# Log a scalar
wandb.log({"accuracy": accuracy})

# Log a plot
wandb.log({"confusion_matrix": wandb.plot.confusion_matrix(
    y_true=y_true, y_pred=y_pred, class_names=["A", "B", "C", "D", "E"]
)})

wandb.finish()

Saving and loading artifacts

After training, you can save your model checkpoint as an artifact:

import wandb

run = wandb.init(project="llm-finetune")

# Save a file as an artifact
artifact = wandb.Artifact("gpt2-finetuned", type="model")
artifact.add_file("model.pt")  # hypothetical checkpoint file
run.log_artifact(artifact)

# Later, download the artifact
run.use_artifact("gpt2-finetuned:latest").download()

run.finish()

You can also automatically track hyperparameters with config and let W&B log the environment (GPU, Python version, etc.) — no extra code needed.

Compare options / when to choose what

W&B is the go-to for experiment tracking, but it's not the only option. Here's a quick comparison:

When to choose W&B: - You want a polished UI with zero setup. - You're working with LLMs and need to log prompts, generated text, and evaluation metrics. - You want integrations with Hugging Face, PyTorch Lightning, and other tools.

When to choose alternatives: - You need fully open-source software (MLflow). - You're a solo developer and just want a local plot (TensorBoard). - You're on a strict budget and have infrastructure to maintain (MLflow).

Pro tip: W&B offers a free personal plan with unlimited runs and projects. It's perfect for learning and experimentation.

Troubleshooting & edge cases

Here are common pitfalls and how to fix them:

• wandb.login() fails — ensure your API key is correct. Re-run wandb.login() and paste a fresh key from settings.
• Run doesn't appear in dashboard — check that you called wandb.init() before logging, and that you didn't call finish() early. Also, if you're behind a firewall, set WANDB_MODE=offline to log locally and sync later.
• Logging None values — W&B can't serialize None. Convert to a number or use wandb.log({"metric": 0}) instead.
• Artifact version conflicts — each upload creates a new version. Use aliases (e.g., latest) to refer to the current one.
• High logging overhead — if logging every step slows training, log every N steps: if step % 10 == 0: wandb.log(...).

# Example of logging every 10 steps to reduce overhead
if step % 10 == 0:
    wandb.log({"loss": loss, "step": step})

What you learned & what's next

You've learned how to track experiments with Weights & Biases: initialize runs, log metrics, save artifacts, and compare results. The key takeaway is that W&B turns messy experiment logs into a structured, shareable dashboard. Now you can reproduce past runs, avoid wasting compute, and collaborate with teammates.

Next, you'll likely explore sweeps — W&B's automated hyperparameter optimization — or dive into evaluation tracking for your LLM pipelines. You're one step closer to building professional-grade AI systems.

Pro tip: Make W&B a habit — even small scripts benefit from logging. You'll thank yourself next week when you need to remember what you ran.

Practice: Re-run the hands-on example, but add your own metric (e.g., token length) and save an artifact. Then open the dashboard and explore the run details — note how the config and system metrics appear automatically.