Inspect Adapter Weights

You’ve trained your LoRA or QLoRA adapter, validation loss looks decent, and the model generates plausible outputs. But are your adapter weights actually healthy? Fine-tuned adapters can silently degrade — weights may beNaN, explode, collapse to near-zero, or overfit to a narrow distribution — and surface-level metrics won’t catch it until your model starts producing nonsense in production. This lesson shows you how to inspect adapter weights for quality issues using practical Python scripts, so you can catch problems early and ship a reliable model.

The problem this lesson solves

Training logs give you loss curves, but they rarely tell you the full story. A loss curve can look perfect while your adapter’s weight distribution collapses or diverges. Consider what happens when you’re fine-tuning a 7B parameter model with LoRA: the base model stays frozen, and only the low-rank A and B matrices update. If those matrices develop issues — like NaN values, extreme outliers, or uniform degradation — your outputs will be subtly wrong, often only detectable when you sample generations.

Quality issues in adapter weights manifest as:

• NaN or Inf values — Training may ‘complete’ if the loss is masked, but inference will crash or produce garbage.
• Exploding weights — A few weights dominate the adapter’s behavior, leading to overconfident or erratic outputs.
• Collapsed weights — All weights shrink toward zero, effectively making the adapter output a no-op, so the model ignores your fine-tuning.
• Distribution skew — Weights cluster in a narrow range, indicating the adapter hasn’t learned meaningful features.

Without inspection, these issues go straight to production. Inspection is your early warning system.

Core concept / mental model

Think of adapter weights as the sculptor’s chisel on the frozen base model. The base model is the raw block of marble; the adapter’s low-rank matrices are the fine carving tools. If the chisel is damaged (NaN or explosive), the sculpture becomes malformed. If it’s too dull (collapsed weights), you’re just polishing a stone that looks original but never improves.

The key idea is statistical health monitoring. Instead of looking at individual weights (which is overwhelming), you look at distributions and summary statistics. It’s like checking vital signs: temperature, blood pressure, heart rate. For adapters, the vitals are:

• Mean and standard deviation — Detect shifts from expected ranges.
• Percentiles and extremes — Spot outliers that could cause instability.
• NaN/Inf count — The easiest red flag.
• Histogram shape — Tell you if the distribution is healthy or skewed.

In practice, a healthy adapter trained with LoRA often has weights in a symmetric distribution around zero, with standard deviation around 0.01–0.1 (depending on rank and alpha). If you see a standard deviation over 1.0 or under 1e-5, something is wrong.

How it works step by step

Start by loading your trained adapter (usually a directory with adapter_model.bin or adapter_model.safetensors). Here’s the mental sequence:

1. Organize your project — Have your adapter checkpoint and training logs handy.
2. Load the adapter weights — Use Hugging Face’s PEFT library or directly with torch.load and safetensors.torch.load_file.
3. Compute summary statistics — For each weight tensor, get min, max, mean, std, and NaN/Inf counts.
4. Visualize distributions — Plot histograms (via matplotlib or seaborn) and, if the model is large, subsample.
5. Interpret the results — Compare against known healthy ranges or your own training baseline.
6. Decide action — If anomalies appear, consider retraining with adjusted hyperparameters or resetting the adapter.

The anatomy of an adapter checkpoint

PEFT adapters store only the low-rank matrices plus metadata. In a LoRA adapter, you’ll see layers named like base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight. The lora_A and lora_B tensors are your inspection targets.

Reading the signs like a doctor

Hands-on walkthrough

Let’s write Python code to inspect a real adapter. We’ll use a sample adapter path; replace with your own.

import torch
from safetensors.torch import load_file

# Load adapter weights (adjust path)
path = "./my_adapter/adapter_model.safetensors"
weights = load_file(path)

# Print available layers
print("Layers found:", len(weights))
print(list(weights.keys())[:5])

Expected output (list may vary):

Layers found: 24
['base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight', ...]

Now compute summary statistics per tensor:

import torch

def inspect_weights(weights):
    """Print per-tensor stats."""
    for name, tensor in weights.items():
        t = tensor.float()
        n_nan = torch.isnan(t).sum().item()
        n_inf = torch.isinf(t).sum().item()
        print(f"{name}: mean={t.mean():.4f}, std={t.std():.4f}, min={t.min():.4f}, max={t.max():.4f}, nan={n_nan}, inf={n_inf}")

inspect_weights(weights)

Watch for lines with nan or inf > 0. That’s your first red flag.

Visualizing the distribution

A histogram is worth a thousand numbers:

import matplotlib.pyplot as plt
import torch

# Flatten all weights into one vector
all_weights = torch.cat([t.flatten() for t in weights.values()]).float()

plt.hist(all_weights.numpy(), bins=100, alpha=0.7)
plt.title("All adapter weights distribution")
plt.xlabel("Weight value")
plt.ylabel("Frequency")
plt.grid(True)
plt.show()

A healthy LoRA adapter often shows a bell-shaped curve around zero. If you see spikes at very large values, that’s a warning.

Automating the check

Build a reusable function that returns a health score:

def adapter_health_check(weights, std_range=(0.001, 1.0)):
    """Return a list of anomalies."""
    issues = []
    for name, tensor in weights.items():
        t = tensor.float()
        if torch.isnan(t).any() or torch.isinf(t).any():
            issues.append(f"{name}: contains NaN/Inf")
        if t.std() > std_range[1] or t.std() < std_range[0]:
            issues.append(f"{name}: std={t.std():.4f} outside healthy range")
    return issues

issues = adapter_health_check(weights)
if issues:
    print("Issues found:")
    for i in issues:
        print(" -", i)
else:
    print("All good — adapter weights look healthy.")

Compare options / when to choose what

There are several ways to inspect adapter weights, each with its trade-offs:

Our recommendation: Use manual summary stats for iterative debugging, and automate threshold checks in your training pipeline to catch regressions early.

Troubleshooting & edge cases

• My adapter has no lora_A layers — You may have loaded the base model instead of the adapter. Ensure adapter_config.json is present and you’re reading the right file.
• All weights are zero — This happens if you loaded a model that was never trained (e.g., wrong checkpoint path). Verify the training run actually updated the adapter.
• Weights show NaN but training loss was fine — Sometimes the loss is computed after masking so NaN weights never cause a crash. Check the raw tensors during training callback.
• PyTorch vs safetensors mismatch — If you trained with torch but saved with safetensors, load accordingly. Use safetensors.torch.load_file for .safetensors files.
• Large model memory issues — When loading a 70B adapter, load only the tensors you need: load_file(path, device="cpu") then compute stats per tensor and free memory.

For serious cases, write a training-time hook to log weight stats every N steps and catch problems before they corrupt the checkpoint.

What you learned & what's next

You now know how to inspect adapter weights for quality issues by computing summary statistics, visualizing distributions, and building automated health checks. You can identify NaN/Inf, exploding or collapsed weights, and distribution skew — the most common quality traps in LoRA/QLoRA fine-tuning.

In the next lesson, you’ll learn how to quantize your adapter for efficient inference — using the healthy adapter you’ve verified to fit into production memory constraints. Keep your adapter health checks handy; they’ll save you from debugging downstream chaos.

Pro tip: Run this inspection script right after every training run, before you even look at generation samples. It takes 10 seconds and saves you hours of pain.