Add DPO to Align With Human Feedback

You've spent hours curating datasets and fine-tuning your LLM until it nails the syntax, style, and domain knowledge you need. But when it actually generates responses, something feels off — it's knowledgeable, yet it argues with users, refuses harmless requests, or produces answers that a human reviewer would immediately reject. You could set up a full RLHF pipeline with PPO, but that means training a reward model, managing rollout sampling, and wrestling with unstable reinforcement learning. There's a simpler, more direct path: Direct Preference Optimization (DPO). In this lesson, you'll learn how to add DPO to align with human feedback — turning your supervised fine-tune into a model that consistently follows human preferences, without the complexity of RLHF.

The problem this lesson solves

Supervised fine-tuning (SFT) teaches a model to mimic the form of good responses, but it can't teach the model to prefer one response over another. You might have a dataset with high-quality examples, but what about the near-misses, the subtly wrong answers, or the responses that are technically correct but unhelpful? SFT treats all training examples as equally good, so the model never learns to differentiate between a 'good' response and a 'better' one.

Meanwhile, full RLHF with PPO requires you to train a separate reward model on human preference data, then use that reward model to guide the policy via reinforcement learning. This pipeline is notoriously finicky: it requires careful hyperparameter tuning, large memory footprints, and often suffers from instability. DPO offers a more elegant solution: it directly optimizes the policy to satisfy human preferences using a simple classification loss, eliminating the need for a reward model and the complexity of RLHF. This lesson will show you how to add DPO to your fine-tuning workflow so your model aligns with human feedback — efficiently and reliably.

Core concept / mental model

Think of DPO as teaching by comparison instead of teaching by imitation. With SFT, you show the model a single correct answer and say 'do this.' With DPO, you show the model two answers to the same prompt — a chosen response and a rejected response — and say 'prefer this over that.' The model learns to increase the likelihood of the chosen response and decrease the likelihood of the rejected response, all by adjusting its own parameters.

The magic of DPO is that it achieves what RLHF does without a separate reward model. In RLHF, you train a reward model on preference data, then use reinforcement learning to optimize the policy against that reward. In DPO, you directly derive the optimal policy from the preference data using a clever mathematical rearrangement of the RLHF objective. The result: you only need a reference model (a frozen copy of your SFT model) and your preference dataset, and you can train the policy with a simple cross-entropy-like loss.

Here's the mental model: you have a teacher (the reference model) and a student (your active model). The student tries to produce responses that humans prefer, while the teacher keeps the student from drifting too far into untamed territory. The DPO loss penalizes the student when it assigns higher probability to a rejected response than to a chosen one, and when it strays too far from the reference model. This dual objective keeps the model aligned and stable.

How it works step by step

To add DPO to your alignment pipeline, follow these steps:

1. Prepare your preference dataset — Each example must have a prompt, a chosen response, and a rejected response. The chosen response is the one humans (or an automated judge) prefer. Format it as a list of messages if you're using a chat model.
2. Load your SFT model and tokenizer — Use the model you fine-tuned in previous lessons (or any Hugging Face model).
3. Create a reference model — This is a frozen copy of your SFT model. It serves as the anchor to prevent the policy from collapsing.
4. Configure the DPO Trainer — Set parameters like beta (the temperature controlling how much you trust the reference model), learning rate, and batch size.
5. Train — Run the training loop. The trainer computes the DPO loss for each batch: it compares the log-probabilities of chosen and rejected responses under both the policy and the reference model.
6. Evaluate — After training, test on held-out prompts to check that the model's responses align with human preferences. Use metrics like win rate against a baseline, or simply eyeball a few examples.

Hands-on walkthrough

Let's implement DPO using the Hugging Face TRL library. First, install the required packages:

pip install transformers trl datasets peft accelerate bitsandbytes

Now, prepare a small preference dataset. In practice you'd have thousands of examples, but here's a simple synthetic one to illustrate the format:

from datasets import Dataset

preference_data = [
    {
        "prompt": "What is the capital of France?",
        "chosen": "The capital of France is Paris. It's known for the Eiffel Tower and its rich history.",
        "rejected": "Paris is the capital. It's a city in Europe."
    },
    {
        "prompt": "How do I make a cup of tea?",
        "chosen": "Boil water, add a tea bag to a cup, pour the water over it, let it steep for 3–5 minutes, then remove the bag. Add milk or sugar to taste.",
        "rejected": "Put tea in hot water."
    }
]

dataset = Dataset.from_list(preference_data)
print(dataset[0])  # see the structure

Now load your SFT model and set up DPO training:

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig

tokenizer = AutoTokenizer.from_pretrained("your-sft-model")
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained("your-sft-model")

# Format the dataset for DPO (assuming a chat template)
def format_dpo(example):
    return {
        "prompt": example["prompt"],
        "chosen": example["chosen"],
        "rejected": example["rejected"],
    }

dataset = dataset.map(format_dpo)

# Configure DPO
config = DPOConfig(
    output_dir="./dpo_model",
    beta=0.1,          # how much to trust the reference model
    learning_rate=1e-5,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    logging_steps=10,
    save_steps=100,
    max_length=1024,
    max_prompt_length=512,
)

trainer = DPOTrainer(
    model=model,
    ref_model=None,   # if None, the trainer creates a reference from the model
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=config,
)

trainer.train()

After training, you can evaluate how your model responds — it should now consistently prefer responses that align with human expectations:

from transformers import pipeline

gen_pipeline = pipeline("text-generation", model="./dpo_model", tokenizer=tokenizer)


def generate(prompt):
    out = gen_pipeline(prompt, max_new_tokens=50, do_sample=False)
    return out[0]["generated_text"][len(prompt):]

print("Prompt:\n", "What is the capital of France?\n")
print("DPO-aligned response:\n", generate("What is the capital of France?\n"))

Expected output: The model now gives a comprehensive answer, matching the style and completeness of the chosen response in your preference data, rather than the terse, incomplete rejected response.

Compare options / when to choose what

Troubleshooting & edge cases

• Reference model collapse: If your reference model is the same as your initial SFT model and you train too aggressively (high learning rate, low beta), the policy can drift and produce degenerate outputs. Lower the learning rate and increase beta (e.g., from 0.1 to 1.0) to anchor the policy closer to the reference.
• Imbalanced preference data: If most chosen and rejected responses are very similar, the model learns little. Ensure your preference pairs have meaningful diversity — the rejected response should be a plausible but clearly inferior alternative.
• Tokenizer issues: Ensure the tokenizer's padding token is set (we did tokenizer.pad_token = tokenizer.eos_token). Without it, the trainer may throw errors. Also, check that the prompt format matches your model's chat template; otherwise the model may generate garbage.
• Memory Out-of-Memory (OOM): DPO requires loading two models (policy + reference). Use LoRA or QLoRA via PEFT to reduce memory footprint, especially for large models like Llama-7B. See the variations section below.
• Loss not decreasing: If the DPO loss plateaus or increases, verify that your data pipeline is correct — the chosen/rejected outputs must be properly tokenized and aligned with the prompt. Also check that you're not accidentally shuffling the chosen/rejected columns.
• Training instability: Similar to RLHF, DPO can occasionally show instability. Reduce the learning rate, increase batch size via gradient accumulation, or lower beta to stabilize.

What you learned & what's next

You now understand how to add DPO to align with human feedback: you can prepare preference data, set up a DPOTrainer, and run training without the complexity of RLHF. You can explain the core idea behind DPO — direct policy optimization from preference pairs — and you've completed a practical exercise using Hugging Face TRL. You've also seen how DPO compares to other alignment methods and how to troubleshoot common issues.

Next step: In the next lesson, you'll move beyond a single alignment approach and learn to evaluate and iterate on your fine-tuned models. You'll design offline evaluations, track multiple quality dimensions, and use the results to decide when to adjust your data, your SFT recipe, or your DPO parameters. This closes the loop between training and deployment, ensuring your model not only aligns with human feedback but continuously improves.