Set Training Hyperparameters for LLMs

Training a large language model is a bit like teaching someone to cook a complex dish. You can have the finest ingredients (your data) and the best recipe (the model architecture), but if you bake at the wrong temperature or for too long, you'll end up with a flavorful disaster. The dreaded 'loss plateau', 'catastrophic forgetting', or a model that just spits out nonsense are all classic symptoms of poorly set hyperparameters. In this lesson, you'll learn how to systematically set training hyperparameters for LLMs — from learning rate to batch size to epochs — so you can transform your raw data into a finely-tuned assistant that actually follows your instructions.

The problem this lesson solves

Imagine you've spent days cleaning and formatting your dataset, only to watch your fine-tuning run produce a model that repeats itself, forgets the general knowledge it was pretrained with, or never seems to improve. You're not alone. This is the single most common frustration in LLM fine-tuning.

Without a solid grasp of hyperparameters, you're essentially gambling with your compute budget. Too high a learning rate and the loss explodes; too low and the model barely moves. Wrong batch size can cause unstable training, and too many epochs can lead to overfitting on your narrow dataset. This lesson gives you the mental framework and Python code to set these values with confidence, so your training runs are predictable, efficient, and effective.

Core concept / mental model

Think of your pretrained LLM as a skilled but stubborn chef. Fine-tuning is the process of teaching it your specific restaurant's special recipes. Hyperparameters are the knobs you control during this apprenticeship:

• Learning rate – how aggressively the model changes its weights per step. A small learning rate is like a cautious apprentice who takes tiny, careful steps. A large one is like a reckless one who might knock over the whole kitchen.
• Batch size – how many recipes (samples) the model reviews at once before adjusting. A larger batch gives a more stable gradient estimate but uses more memory.
• Epochs – how many times the model goes through your entire dataset.
• Weight decay – a regularizer that discourages the model from becoming too confident in its weights.
• Warmup steps – a gradual increase from a low learning rate to the target rate, which helps avoid early instability.

You can visualize the training process as navigating a loss landscape. Hyperparameters determine the size of your steps (learning rate), how often you look at the map (batch size), and how many laps you take (epochs).

How it works step by step

The process of setting hyperparameters follows a logical sequence. Let's walk through it.

1. Initialize training arguments – You'll use the Hugging Face TrainingArguments class to define most hyperparameters.
2. Choose a learning rate – A common starting point for fine-tuning is 2e-5 for full fine-tuning and 1e-4 to 5e-4 for LoRA (Parameter-Efficient Fine-Tuning).
3. Set batch size – Depending on your GPU memory, start with 8 or 16. Use gradient accumulation if you need a larger effective batch.
4. Pick the number of epochs – 3 is a solid default, but you can monitor validation loss to detect overfitting.
5. Enable warmup – 10% of total steps is a common ratio.
6. Use weight decay – A value like 0.01 helps with regularization.
7. Consider advanced optimizers – Options like AdamW with a cosine learning rate schedule are widely used.

Hands-on walkthrough

Let's put this into practice. We'll set hyperparameters for a QLoRA fine-tuning run using the Hugging Face transformers library. Here’s a complete example to get you started.

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="./lora-llm",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    warmup_ratio=0.03,
    weight_decay=0.01,
    logging_steps=50,
    save_strategy="epoch",
    evaluation_strategy="epoch",
)

Expected behavior: The training loop will save a checkpoint after each epoch and log loss every 50 steps. The effective batch size is 4 * 4 = 16.

Now let's hook this into a real Trainer with a tiny LoRA model.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from transformers import Trainer

# Load a small model (for demo)
model_name = "microsoft/phi-2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(model_name, load_in_4bit=True)

# Configure LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05
)
model = get_peft_model(model, lora_config)

# Your datasets go here (dummy)
from transformers import DataCollatorForLanguageModeling
collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

# Assume train_dataset and eval_dataset are prepared

# Create the Trainer
trainer = Trainer(
    model=model,
    args=args,
    data_collator=collator,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

# Start training
trainer.train()

Expected output: You'll see a progress bar with the loss decreasing per logging step. For example:

Step: 50, loss: 1.2345
Step: 100, loss: 1.0056
...

Always monitor the loss curve. A decreasing trend indicates healthy learning.

Compare options / when to choose what

Not all hyperparameter choices are equal. Here's a quick comparison table to help you decide.

Pro tip: When in doubt, start with the low end of the learning rate range. It's easier to recover from a slow start than from a diverged run.

Different optimizers also come with their own quirks. The paged AdamW in QLoRA offloads optimizer states to CPU, which helps big models fit into small GPUs but can slow training.

Troubleshooting & edge cases

Here are the most common issues and how to fix them:

• Loss spikes or NaN – Often caused by too high a learning rate. Lower it (e.g., from 2e-4 to 5e-5) and add gradient clipping.
• Model doesn't improve – Check if your learning rate is too low. Try increasing by a factor of 10 or use a cosine schedule with a higher peak.
• Out-of-memory (OOM) – Reduce batch size, enable gradient accumulation, or use LoRA with 4-bit quantization (QLoRA).
• Overfitting – If validation loss starts increasing after a few epochs, reduce epochs, increase weight decay, or add more dropout.
• Warmup too short – If training is unstable in the first few steps, increase the warmup ratio to 10% or more.

What you learned & what's next

You've learned how to set the core training hyperparameters for LLMs: learning rate, batch size, epochs, warmup, and weight decay. You can now configure a training run using Hugging Face's TrainingArguments and Trainer, and you know how to troubleshoot common issues like overfitting and OOM errors.

Your next lesson in the LLM Fine-Tuning track will take you through Evaluating Fine-Tuned Models — where you'll use metrics like perplexity and human evaluation to see if your hyperparameter choices actually produced a better model.