What Is LLM Finetuning?

You've probably heard the phrase "fine-tune an LLM" tossed around, but what does it actually mean? Maybe you've tried using a generic model like GPT-3.5 for a specialized task—say, extracting medical terms from clinical notes—and gotten decent but not great results. Or perhaps you've felt frustrated by a model that just won't follow your company's tone guidelines no matter how clever your prompt is. That gap between a general-purpose model and your specific needs is the exact problem LLM fine-tuning solves. In this lesson, you'll understand what LLM fine-tuning means, why it's a game-changer for real-world applications, and how it fits into the larger ML lifecycle—setting you up for hands-on practice in the next lessons.

The problem this lesson solves

Generic pretrained LLMs are trained on massive, diverse internet text. They're brilliant at general language understanding, but they lack specialized knowledge and consistent behavior for your domain. Consider these real-world pain points:

• Domain jargon: A model trained on general web text may not know that "Indication" in a clinical report refers to the reason a treatment is given, not a clue in a mystery novel.
• Format adherence: You need structured JSON output, but the model keeps adding explanatory text.
• Tone and style: Your brand voice is witty and concise, but the model gives verbose, formal answers.

Prompt engineering can only go so far. You're limited by the model's pretrained knowledge and your ability to craft the perfect instruction. No amount of prompting will teach a model a new fact or a deeply embedded business rule. Fine-tuning addresses this by updating the model's weights on your own data, making the model learn from examples rather than merely follow instructions.

Core concept / mental model

Think of a pretrained LLM as a brilliant, book-smart graduate who has read everything but has no on-the-job experience. They know language, grammar, and world facts, but they've never filed a legal brief or written a product description in your specific style. Fine-tuning is the on-the-job training. You show them examples of the exact kind of output you want, and they adjust their behavior accordingly.

More technically, fine-tuning is supervised learning applied to a pretrained model. You take a model with pretrained weights, feed it your labeled dataset (input-output pairs), and continue the training process for a few epochs, updating the weights to minimize a loss function. The result is a new model that's specialized for your task.

Key definitions

• Pretrained model: A model trained on a huge corpus (e.g., BERT, GPT-3) to learn general language representations.
• Fine-tuning: The process of taking a pretrained model and training it further on a smaller, task-specific dataset.
• Base model vs. fine-tuned model: The base model is the out-of-the-box model; the fine-tuned model has adapted weights.
• Full fine-tuning vs. parameter-efficient fine-tuning (PEFT): Full fine-tuning updates all weights; PEFT updates a small subset or adds small adapters (e.g., LoRA).

How it works step by step

The process of fine-tuning an LLM follows a logical sequence. Here's the bird's-eye view:

1. Choose a pretrained base model - Pick a model appropriate for your task (e.g., BERT for classification, GPT for generation).
2. Prepare your dataset - Curate a set of input-output examples that represent your desired behavior. This is the most critical step.
3. Preprocess the data - Tokenize your inputs and outputs, convert them to tensors, and create batches.
4. Set up a training loop - Load the model, define a loss function (usually cross-entropy), an optimizer (e.g., AdamW), and a learning rate.
5. Train the model - Run forward passes, compute loss, backpropagate, and update weights over multiple epochs.
6. Evaluate - Measure performance on a held-out validation set using metrics like accuracy or ROUGE.
7. Deploy - Save the fine-tuned weights and use them for inference.

Let's break down the two main approaches:

Full fine-tuning

In full fine-tuning, you update all the parameters of the model. This gives maximum flexibility but requires substantial GPU memory and compute. For example, fine-tuning a 7B-parameter model fully can require 70+ GB of GPU memory, making it impractical for many teams.

Parameter-efficient fine-tuning (PEFT)

PEFT methods like LoRA (Low-Rank Adaptation) freeze the original weights and add small trainable matrices (adapters) to specific layers. This reduces trainable parameters to a fraction of the original (often <1%), allowing fine-tuning on a single consumer GPU. The adapters can later be merged into the base model for inference.

The choice between full and PEFT depends on your resources and how much you need to shift the model's behavior.

Hands-on walkthrough

Let's get concrete. We'll use the Hugging Face transformers library to fine-tune a small model—distilbert-base-uncased—for sentiment classification. This example is simplified but shows the core pipeline. We'll use a tiny synthetic dataset so you can run it on a laptop.

Setup

First, install the required libraries:

pip install transformers datasets torch

Load data and model

We'll create a tiny dataset with three examples:

from datasets import Dataset

# A tiny synthetic dataset
samples = [
    ("I love this course!", 1),  # positive
    ("I hate homework.", 0),     # negative
    ("It's okay.", 1)           # positive
]

dataset = Dataset.from_dict({"text": [s[0] for s in samples], "label": [s[1] for s in samples]})

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

# Tokenize the dataset
def tokenize(batch):
    return tokenizer(batch["text"], padding="max_length", truncation=True, max_length=64)

tokenized_dataset = dataset.map(tokenize, batched=True)
tokenized_dataset = tokenized_dataset.with_format("torch")

Training loop

Now we'll set up a simple training loop:

from torch.utils.data import DataLoader
from transformers import AdamW
from torch.nn.functional import cross_entropy
import torch

# Prepare dataloader
train_loader = DataLoader(tokenized_dataset.remove_columns("text"), batch_size=2)

# Set up optimizer
optimizer = AdamW(model.parameters(), lr=5e-5)

model.train()
for epoch in range(3):
    for batch in train_loader:
        # Move to GPU if available
        batch = {k: v.unsqueeze(0) if v.dim() == 0 else v for k, v in batch.items()}  # fix dimensions
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    print(f"Epoch {epoch} loss: {loss.item():.4f}")

Expected output (numbers may vary):

Epoch 0 loss: 0.6931
Epoch 1 loss: 0.5234
Epoch 2 loss: 0.4231

Save and test

After training, you can save the model and run inference:

model.save_pretrained("./my_finetuned_model")
tokenizer.save_pretrained("./my_finetuned_model")

# Inference
text = "I am excited to learn!"
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
prediction = torch.argmax(logits, dim=-1).item()
print("Positive" if prediction == 1 else "Negative")

Expected output:

Positive

Compare options / when to choose what

You have several ways to adapt an LLM to your task. Here's a quick comparison:

Pro tip: Start with prompt engineering. If that fails, move to LoRA. Only consider full fine-tuning when you have abundant data and compute—it's often overkill.

Troubleshooting & edge cases

Fine-tuning is a delicate process. Here are common pitfalls and how to fix them:

• Loss not decreasing: Your learning rate might be too high or too low. Try values like 2e-5 to 5e-5 for AdamW. Also check your data—label imbalances can stall training.
• Overfitting: If training loss decreases but validation loss increases, you're overfitting. Add regularisation (e.g., weight decay), use a smaller model, or get more data.
• GPU out of memory: Reduce batch size, use gradient accumulation, or switch to LoRA/QLoRA.
• Tokenizer mismatch: If you're using a model from Hugging Face, always use its corresponding tokenizer—don't mix and match.
• Data format issues: If your batches have varying lengths, ensure you pad/truncate consistently. Use padding=True and truncation=True in the tokenizer.
• Text generation models: For decoder-only models, you may need to format your data as instruction-following pairs with special tokens (e.g., <|im_start|>) to teach the model to respond correctly.

What you learned & what's next

In this lesson, you learned what LLM fine-tuning means: updating a pretrained model on your own data to specialize it for a task. You now understand the core concept, the step-by-step process, and have run a hands-on example. You also know how to choose between full and parameter-efficient fine-tuning and how to troubleshoot common issues.

Next, you'll dive deeper into data preparation—the foundation of any fine-tuning project. You'll learn how to collect, clean, and format your dataset to maximize model performance. Get ready to turn your raw data into a training-ready asset.

Remember: fine-tuning is not magic—it's supervised learning on top of a pretrained model. With the right data and a solid process, you can tailor LLMs to solve your specific problems.