Use PEFT for Fine-Tuning

Fine-tuning a modern large language model used to mean renting eight or more A100 GPUs, babysitting a memory-hungry training run for days, and hoping your single GPU doesn't OOM. That pain is real, and it blocks most developers from customizing models at all. In this lesson you'll remove that barrier by using PEFT (Parameter-Efficient Fine-Tuning) — a family of techniques that trains a tiny fraction of the model's parameters while keeping the original weights frozen. You'll walk away able to adapt a multi-billion-parameter model on a single consumer GPU, and you'll understand exactly why PEFT is the default choice for modern fine-tuning workflows.

The problem this lesson solves

Full fine-tuning updates every weight in the model. For a 7B-parameter model, that means storing and updating 14GB of gradients plus optimizer states (often another 56GB with Adam), and then you need to keep the original model in memory too. Realistically, full fine-tuning requires around 300–400GB of GPU memory for a 7B model. Few developers have access to that. Even if you do, the training is slow, and you end up with a full copy of the model — one fine-tuned checkpoint can be 14GB or larger. The operational cost of storing and serving one custom model per task becomes absurd.

PEFT solves this by freezing the original model weights and inserting a small number of trainable parameters (typically 0.1%–1% of the total). You only compute gradients for those new parameters, which slashes memory usage, speeds up training, and produces tiny adapter files (a few megabytes). Those adapters can be swapped in and out at inference time without copying the base model.

Core concept / mental model

Think of PEFT as adding a volume knob to a concert. The base model is the band — its music (knowledge) is fixed. PEFT adds a small, adjustable control that changes how the music sounds without re-training the band. You tweak the knob, not the musicians.

More concretely, the dominant PEFT technique is LoRA (Low-Rank Adaptation). LoRA works by adding small trainable matrices (the adapters) to specific layers of the frozen model. During training, only these adapters learn; the base weights stay untouched. Because the adapters are low-rank (few dimensions), they store surprisingly little information, yet they can steer the model's behaviour almost as effectively as full fine-tuning.

Key definitions to lock in

• Adapter: The small trainable module inserted into the model.
• Rank (r): The size of the adapter matrices. Higher rank = more capacity but more parameters.
• Alpha (alpha): A scaling factor that controls how strongly the adapter's output influences the frozen layer. Often set to 2× the rank.
• QLoRA: A variant that also quantizes the frozen base model (e.g., to 4-bit) to further reduce memory, enabling fine-tuning of even larger models on a single GPU.

How it works step by step

1. Load the pretrained model — usually from Hugging Face Hub.
2. Wrap the model with PEFT — call get_peft_model() with a LoraConfig to inject trainable adapters.
3. Verify trainable parameters — confirm only the adapter parameters are trainable (check print_trainable_parameters()).
4. Train — run your normal training loop (e.g., with Trainer). Gradients only flow into the adapters.
5. Save the adapter — use model.save_pretrained(); the resulting folder is just a few megabytes.
6. Load later — merge the adapter into the base model or load it separately for inference.

Why only a few parameters?

The magic is that the adapter's output is added to the frozen layer's output, and the adapter is a low-rank decomposition of a delta matrix. During training, the adapter learns the direction in which the base weights should change, not every individual weight. This compressed representation captures most of the benefit with a fraction of the cost.

Hands-on walkthrough

Let's put PEFT into action with a real example. We'll fine-tune the facebook/opt-350m model using LoRA. This model is small enough for a laptop GPU, but the workflow is identical for 7B or 70B models.

Setup

First, install the required libraries:

pip install peft transformers datasets accelerate bitsandbytes

Load a base model and apply LoRA

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

# Load the pretrained model and tokenizer
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m")
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")

# Configure LoRA
lora_config = LoraConfig(
    r=8,               # rank of the adapter matrices
    lora_alpha=16,     # scaling factor (often 2x r)
    target_modules=["q_proj", "v_proj"],  # which layers to adapt
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"   # for causal language modeling
)

# Wrap the model with PEFT
peft_model = get_peft_model(model, lora_config)

# See how many parameters are trainable
peft_model.print_trainable_parameters()
# Output (approximate):
# trainable params: 294,912 || all params: 665,649,152 || trainable%: 0.0443

Training with the Hugging Face Trainer

Now we'll fine-tune on a tiny custom dataset. In practice you'll use bigger datasets, but this shows the complete flow.

from transformers import Trainer, TrainingArguments

# Dummy dataset — replace with your own data
from datasets import Dataset

training_data = Dataset.from_dict({
    "text": [
        "What is the capital of France? Paris.",
        "What is 2+2? 4.",
        "The sky is blue."
    ]
})

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=32)

tokenized_dataset = training_data.map(tokenize_function, batched=True)

# Note: For LM training we need labels. For simplicity, we use the same text as labels.
# In a real project, you would have proper prompt/answer splits.

training_args = TrainingArguments(
    output_dir="./lora-opt",
    per_device_train_batch_size=1,
    learning_rate=2e-4,   # higher lr works well with LoRA
    num_train_epochs=3,
    logging_steps=1,
    save_strategy="epoch",
)

trainer = Trainer(
    model=peft_model,
    args=training_args,
    train_dataset=tokenized_dataset,
)

trainer.train()

Save and reuse the adapter

# Save only the adapter (a few KB)
peft_model.save_pretrained("./lora-opt-adapter")
tokenizer.save_pretrained("./lora-opt-adapter")

# At inference time, load the base model and the adapter separately
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m")
loaded_adapter = PeftModel.from_pretrained(base_model, "./lora-opt-adapter")

# Or merge the adapter into the base model once (for faster inference)
merged_model = loaded_adapter.merge_and_unload()

The saved adapter folder contains adapter_config.json and adapter_model.safetensors — typically just a few megabytes, even for a 7B model.

Compare options / when to choose what

PEFT isn't a single method — it's an umbrella. Here's how the main variants stack up:

When to prefer which

• LoRA is the default for most tasks. It's simple, well-supported, and gives excellent results.
• QLoRA is your answer if you want to fine-tune a 7B model on a 16GB laptop GPU. It quantizes the base model to 4-bit, and the adapter remains in 16-bit.
• Prefix tuning is niche but useful when you want to prepend learnable tokens to the input without any extra layers.

Troubleshooting & edge cases

target_modules is wrong or empty

If you get an error about not finding modules, or the trainable% is 0, you likely named the wrong layers. Check the model's architecture:

print(model)

Look for linear layers like q_proj, v_proj, k_proj, out_proj, or fc1, fc2. Some models (like Mistral) use q_proj and v_proj; others may have query, value. When in doubt, inspect and adjust.

OOM even with LoRA

If you still run out of memory, enable 4-bit quantization with bitsandbytes:

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-chat-hf",
    quantization_config=bnb_config,
    device_map="auto"
)

Then apply LoRA as before. This is the QLoRA workflow.

Learning rate too high / low

LoRA adapters are small, so they respond well to higher learning rates (1e-4 to 3e-4) compared to full fine-tuning (1e-5). If your loss fluctuates wildly, lower the LR; if it doesn't move, raise it.

Saving full model by mistake

model.save_pretrained() on a PeftModel saves only the adapter, which is what you want. But if you call merge_and_unload() before saving, you'll save the full base model — huge. Save the adapter first.

What you learned & what's next

You've mastered the core of parameter-efficient fine-tuning with PEFT. You now understand the pain of full fine-tuning and why PEFT solves it, how LoRA and QLoRA work conceptually, how to apply LoRA with transformers and peft, how to save a tiny adapter, and how to pick the right technique for your hardware. This directly sets you up for the rest of the track: evaluating your fine-tuned model, comparing PEFT with other methods, and deploying your adapted model in production.

Next, you'll dive into deeper practical territory — evaluating your fine-tuned model's outputs to make sure the adapter actually improved performance. You'll also get hands-on with more realistic datasets and training loops, so you can confidently fine-tune anything, anywhere.