Full Finetuning vs Feature Extraction

Your model is too big to retrain fully, too slow to run on every request, and the data you have is just a few hundred labeled examples. If you try to fine-tune every weight from scratch, you'll burn through GPU budget and risk catastrophic forgetting. The tension between full finetuning and feature extraction — deciding whether to update all model parameters or only use the pretrained backbone as a fixed feature extractor — is one of the most common forks in applied LLM work. This lesson gives you a practical framework to choose between them, with benchmarks you can run yourself on a small dataset.

The problem this lesson solves

When you deploy an LLM for a specific task — say, classifying support tickets or extracting entities from legal contracts — you have two broad adaptation strategies. One is full finetuning (also called full-model fine-tuning): you take the pretrained model and update all its weights using your labeled data. The other is feature extraction (sometimes called frozen-feature transfer): you keep the model's weights frozen and train only a small head (usually a linear classifier or a simple MLP) on top of the embeddings or hidden states the base model produces.

The pain is that beginners often pick the wrong one. They default to full finetuning because it sounds more "real," then run out of GPU memory, wait days for training, and end up with a model that worse than the original on out-of-distribution inputs. Or they use feature extraction when their data is large and expressive enough that a frozen backbone can't capture the new task's nuances — and they see stunted accuracy. This lesson gives you the mental model and a hands-on comparison so you can make the right call based on your data size, compute budget, and task complexity.

Core concept / mental model

Think of the pretrained LLM as a general-purpose map of language. Full finetuning is like redrawing the entire map for your specific city — you need a lot of source data and you risk losing the global context. Feature extraction is like adding a new legend on top of the existing map — you keep the terrain, but you add a marker that points directly to what matters for your task.

More formally:

• Feature extraction: the base model (e.g., bert-base-uncased, roberta-base, or a sentence-transformer) is frozen. You pass your inputs through it to get embeddings (fixed-length vectors). A small trainable head (often a linear layer or an MLP) is trained on those embeddings. Only the head's weights are updated.
• Full finetuning: you load the base model with its pretrained weights, then train the entire model (all transformer layers, attention, normalization, etc.) on your task. All weights are updated via backpropagation.

The key trade-off is capacity vs. constraint: full finetuning can adapt the model's internal representations to your task, but it exposes you to overfitting and higher compute costs. Feature extraction runs fast and needs less data, but its ceiling is lower because the base representations don't change.

When do you choose which? The decision follows a simple rubric:

Pro tip: If your task is a standard classification (sentiment, topic, type of defect), feature extraction is often enough — and it's dramatically cheaper to train and deploy.

How it works step by step

Full finetuning — what actually changes

When you run full finetuning, you initialize your model with pretrained weights (e.g., from HuggingFace's bert-base-uncased) and then run gradient descent over your labeled dataset. Because all parameters have gradients, the optimizer (like AdamW) updates every one of them. After a few epochs, the model's weights are tuned to minimize loss on your task.

The steps are: 1. Load a pretrained model and tokenizer. 2. Prepare your dataset into examples with input IDs, attention masks, and labels. 3. Define a loss function (often cross-entropy for classification, or language modeling loss for generation). 4. Train with a small learning rate (typically between 1e-5 and 5e-5) to avoid catastrophic forgetting. 5. Evaluate on a hold-out set.

Feature extraction — why it's different

Feature extraction uses the same base model, but you freeze its weights (set requires_grad = False for all parameters). You then compute the embeddings of your text examples (e.g., the [CLS] token for BERT, or the mean-pooled hidden state) and feed those to a fresh head. Only the head is trained.

The steps are: 1. Load the pretrained model and tokenizer (same as above). 2. Freeze the base model parameters. 3. Run forward passes to extract embeddings for all training samples (optionally cache them to disk). 4. Train a small classifier on those embeddings. 5. Evaluate on the hold-out set.

The difference in gradient flow is the core: in full finetuning, the optimizer touches every weight; in feature extraction, it only adjusts the head. This changes training speed, memory usage, and the minimum dataset size needed.

Hands-on walkthrough

For a concrete comparison, we'll use the AG News dataset (news article classification into 4 topics) — a manageable public dataset that shows both approaches well. We'll use Hugging Face Transformers and a tiny sample (e.g., 2,000 training examples) to keep it quick.

Setup

# pip install transformers datasets scikit-learn torch
torch.manual_seed(42)

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

Feature extraction in practice

We'll freeze the base model and train only the classifier head.

from tqdm import tqdm
import torch
import numpy as np
from sklearn.linear_model import LogisticRegression

# Load tokenizer + base model without the classification head (we'll add our own)
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=4,
    output_hidden_states=False
)

# Freeze the base model (everything)
for param in model.parameters():
    param.requires_grad = False

# Replace the classification head with a fresh, trainable linear layer
from torch import nn
model.classifier = nn.Sequential(
    nn.Dropout(0.1),
    nn.Linear(model.config.dim, 4)  # 768 for DistilBERT
)

# Load AG News (small subset for speed)
dataset = load_dataset("ag_news", split="train")
train_subset = dataset.select(range(2000))
test_subset = dataset.select(range(2000, 2400))

def preprocess(batch):
    return tokenizer(batch["text"], padding="max_length", truncation=True, max_length=64)

train_subset = train_subset.map(preprocess, batched=True, remove_columns=["text", "label"])
test_subset = test_subset.map(preprocess, batched=True, remove_columns=["text", "label"])

train_subset.set_format("torch")
test_subset.set_format("torch")

# Training arguments (only the head will train)
training_args = TrainingArguments(
    output_dir="./feat_extract",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    learning_rate=2e-3,   # Higher LR for the head, since we're only updating it
    logging_steps=200,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_subset,
    eval_dataset=test_subset,
)

trainer.train()
trainer.evaluate()

When you run this, you'll see the loss drop quickly. On a CPU-only machine, each epoch may take a few minutes. Expect accuracy around 83–86% on the test subset.

Full finetuning in practice

Now compare with full finetuning — the only change is we don't freeze the base model.

# Same imports and setup, but DO NOT set requires_grad = False
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=4
)

# Use a much smaller learning rate for full finetuning
training_args = TrainingArguments(
    output_dir="./full_finetune",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=3e-5,   # Typical for full finetuning
    logging_steps=200,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_subset,
    eval_dataset=test_subset,
)

trainer.train()
trainer.evaluate()

Expected output: you'll see 86–90% accuracy, but training is much slower and uses more memory. If you only had 200 training examples, you'd likely see the full finetuned model overfit and score worse than feature extraction.

Pro tip: Always save your embeddings when using feature extraction — you can reuse them across different heads without re-running the base model.

Measuring the difference

Here's a quick script that logs training time and final accuracy for both approaches:

import time

def train_and_evaluate(model, args, train, eval):
    t0 = time.time()
    trainer = Trainer(model=model, args=args, train_dataset=train, eval_dataset=eval)
    trainer.train()
    eval_result = trainer.evaluate()
    return eval_result["eval_accuracy"], time.time() - t0

# Assume you have models and args prepared
# acc_feat, time_feat = train_and_evaluate(model_feat, args_feat, train_subset, test_subset)
# acc_full, time_full = train_and_evaluate(model_full, args_full, train_subset, test_subset)
print(f"Feature extraction: {acc_feat:.2f} acc, {time_feat:.1f}s")
print(f"Full finetuning: {acc_full:.2f} acc, {time_full:.1f}s")

Compare options / when to choose what

Here's a detailed side-by-side for full finetuning vs feature extraction — use this table when deciding for your own project.

Why might you choose full finetuning? If your task's distribution is very different from the pretraining data (e.g., legal jargon with unusual sentence structures) and you have at least tens of thousands of examples, updating the base representations can yield significant gains. You're also always fine-tuning if your task is generative (like a summarization model) — you can't just slap a linear head on an autoregressive model.

Why might you choose feature extraction? When you have limited labeled data, your main goal is to get a quick, reliable baseline, or your deployment environment has tight constraints — a frozen model is smaller (no head), and you can cache embeddings offline to speed up inference. Feature extraction is also the go-to for creating custom embedding vectors for retrieval-augmented generation (RAG) pipelines.

Alternative: parameter-efficient finetuning (LoRA)

A middle ground exists: LoRA (Low-Rank Adaptation) freezes the base weights but adds small trainable low-rank matrices in parallel to the attention layers. This gives you the capacity of full finetuning with far fewer trainable parameters (often < 1% of the model), making it a favorite when you need deep adaptation without the full cost. We cover this in depth later in this track, but remember it as a third option when neither extreme feels right.

Troubleshooting & edge cases

Feature extraction gives poor accuracy

Symptom: The classifier plateaus early or underperforms a simple Bag-of-Words baseline.

Cause: The frozen representations don't contain enough task-specific signal (e.g., classifying complex legal terms vs. news topics).

Fixes: - Try a different, larger base model (e.g., roberta-large instead of distilbert-base-uncased). - Use pooled embeddings instead of just the [CLS] token — concatenate the mean and max pooling of all hidden states. - Move to LoRA or full finetuning if you have enough data.

Full finetuning overfits on small data

Symptom: Training loss drops to near 0, but validation performance is worse than the pretrained model.

Cause: The model memorizes the few hundred examples you gave it.

Fixes: - Reduce the learning rate (e.g., 1e-5 or lower). - Add regularization: weight decay, dropout, and early stopping. - Use k-fold cross-validation and monitor validation loss closely.

Full finetuning catastrophic forgetting

Symptom: The model's performance on general language tasks drops dramatically.

Cause: Updating all weights can overwrite pretrained knowledge.

Fixes: - Use a lower learning rate. - Use a smaller number of epochs (early stopping). - Consider LoRA, which limits weight updates to a small subspace.

Feature extraction on generative models

Gotcha: For models like GPT or Llama, you can't just attach a linear head. Instead, you'd extract embeddings from the last hidden state and train a separate classifier — but that often underperforms because autoregressive models aren't designed for classification. For generative tasks, full finetuning (or LoRA) is essentially required.

Memory issues

If you hit out-of-memory errors during full finetuning: - Use gradient accumulation to simulate a larger batch size. - Use torch.utils.checkpoint (gradient checkpointing) to trade compute for memory. - Default to LoRA if memory remains a problem.

What you learned & what's next

You now understand the fundamental trade-off between full finetuning and feature extraction. You've seen that feature extraction (freezing the base model and training a head) is fast, cheap, and works well with small datasets, while full finetuning (updating all weights) offers higher performance when you have abundant, task-specific data. You've also learned to spot and fix common pitfalls like overfitting, catastrophic forgetting, and insufficient signal from frozen representations.

Your next step in this track is Parameter-Efficient Finetuning with LoRA, where you'll learn to get close to full-finetuning performance at a fraction of the cost. You'll directly apply the comparative thinking from this lesson to decide when LoRA fits between the two extremes.

• [ ] Re-run the hands-on example with your own dataset (even 500 examples of email spam classification).
• [ ] Compare feature extraction vs. full finetuning accuracy and time on that dataset.
• [ ] Then move to LoRA and see if you can beat both with half the training time.