Run Your First QLoRA Script

Picture this: it's late Friday night, your dataset is cleaned, your GPU is humming, and you're about to fine-tune a 7-billion-parameter model. You've read that QLoRA is the magic bullet for efficient fine-tuning, but when you try to run a script, you hit a wall of cryptic errors, CUDA out-of-memory messages, and confusion about which parameters actually matter. If that sounds familiar, you're in the right place. This lesson walks you through running your first QLoRA training script step by step, so you can go from zero to a tuned model without the all-night debug session.

The problem this lesson solves

Fine-tuning a large language model (LLM) is often a game of trade-offs. Full fine-tuning of a 7B model requires a GPU with massive VRAM—think 60+ GB—which is out of reach for most developers, researchers, and startups. Even when you have the hardware, training is slow, expensive, and prone to catastrophic forgetting, where the model forgets its general knowledge while learning your specific task.

QLoRA (Quantized Low-Rank Adaptation) solves these problems by combining two powerful ideas: quantization (shrinking the model's memory footprint) and low-rank adaptation (training a tiny set of parameters instead of all of them). With QLoRA, you can fine-tune a 7B model on a single 24 GB consumer GPU—like an RTX 3090 or 4090—or even a 16 GB laptop GPU in some cases. This approach makes fine-tuning accessible, but it also introduces a new set of pitfalls: you need to configure the right quantization settings, choose the right LoRA rank, and manage your training loop properly.

If you've ever tried to write a QLoRA script from scratch, you know the pain: mismatched model versions, incorrect tokenizer settings, and subtle bugs that only appear after hours of training. This lesson cuts through the noise. You'll learn not only how to write and run a QLoRA training script, but also how to debug it when things go wrong—and why QLoRA is the go-to method for most fine-tuning tasks.

Core concept / mental model

Think of fine-tuning an LLM as teaching a new skill to a world-class chef. Full fine-tuning retrains every neuron in the chef's brain—expensive, slow, and you might lose their existing expertise. LoRA is like giving the chef a small, specialized notebook where they jot down new recipes and techniques without rewriting their entire culinary knowledge. The chef keeps their brain (the original weights) frozen, but they add a few extra notes (the LoRA adapters) that help them cook your specific dishes.

QLoRA takes this a step further. Instead of keeping the original weights in full precision (32-bit floats), QLoRA quantizes them to 4-bit or 8-bit integers, drastically reducing memory usage. The LoRA adapters, however, remain in high precision (usually 16-bit), so you still get accurate gradients during training. The result is a model that's roughly 4x smaller in memory but still capable of high-quality fine-tuning.

Here are the key components you'll encounter in a QLoRA script:

• Base model: The pretrained LLM you're adapting, e.g., meta-llama/Llama-2-7b-hf or mistralai/Mistral-7B-v0.1.
• Tokenizer: Converts text into tokens (and back), handling padding and truncation.
• Quantization config: Parameters like load_in_4bit=True, bnb_4bit_quant_type='nf4', and bnb_4bit_compute_dtype=torch.float16 that control how the model is quantized.
• LoRA config: Parameters like r (rank), alpha (scaling factor), and target_modules that define the adapters.
• Training arguments: A TrainingArguments object that controls learning rate, batch size, number of epochs, and logging.
• Trainer: The Trainer (or SFTTrainer) that orchestrates the training loop.

Pro tip: Think of QLoRA as high-precision surgery. The original model is frozen in a compressed form, and you're only fine-tuning a tiny set of parameters—the adapters—that capture the task-specific patterns.

How it works step by step

Here's the logical flow from raw data to trained model:

1. Load the base model and tokenizer: Use AutoModelForCausalLM and AutoTokenizer from Hugging Face Transformers, along with a quantization config.
2. Apply quantization: The model is loaded in 4-bit using bitsandbytes, which compresses the weights and reduces memory.
3. Prepare the data: Format your dataset into a single text column, e.g., text, that the tokenizer can process. Handle padding and truncation to keep sequences short.
4. Configure LoRA: Define a LoraConfig with the rank r, alpha, and target modules (typically attention layers like q_proj, v_proj).
5. Set up the trainer: Use SFTTrainer (supervised fine-tuning) or Trainer with a data collator, training arguments, and the LoRA adapter model.
6. Train: Run the training loop, monitor loss, and save checkpoints.
7. Save and merge: Save the adapter weights, and optionally merge them back into the base model for inference.

Each step depends on the previous one. If your quantization config is wrong, your training loop will fail. If your dataset isn't formatted correctly, you'll get poor results. This step-by-step flow is the foundation of any QLoRA script.

Hands-on walkthrough

Let's write and run your first QLoRA training script. We'll use the popular transformers, peft, and bitsandbytes libraries. Install them first:

pip install transformers datasets peft bitsandbytes accelerate

Now, create a Python file named qlora_train.py. Here's a complete script that fine-tunes a small model (e.g., microsoft/phi-2 or bigscience/bloom-560m) on a toy dataset. We'll use a tiny model so you can run it on almost any GPU.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import Dataset

# 1. Load tokenizer
model_id = "microsoft/phi-2"  # or "gpt2" for an even faster test
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

# 2. Quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# 3. Load base model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
model = prepare_model_for_kbit_training(model)

# 4. LoRA config
lora_config = LoraConfig(
    r=8,                # rank
    lora_alpha=16,      # scaling factor
    target_modules=["q_proj", "v_proj"],  # modules to apply LoRA to
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Should show a tiny % of trainable params

# 5. Toy dataset (replace with your own)
data = [
    {"text": "Question: What is the capital of France?\nAnswer: Paris"},
    {"text": "Question: What is 2+2?\nAnswer: 4"},
    {"text": "Question: What is the largest planet?\nAnswer: Jupiter"},
]
dataset = Dataset.from_list(data)

# 6. Training arguments
training_args = TrainingArguments(
    output_dir="./qlora-output",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_steps=50,
    report_to="none",
)

# 7. Trainer
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    dataset_text_field="text",
    max_seq_length=256,
    packing=True,
    peft_config=lora_config,
)

# 8. Train!
trainer.train()

trainer.save_model("./qlora-adapter")

Expected output: You should see logs showing the number of trainable parameters, a training loss that decreases each step, and a final model saved in ./qlora-adapter. For example:

trainable params: 8,388,608 || all params: 2,789,767,168 || trainable%: 0.3006%
Step 10: loss=0.82, lr=2e-4
...
Training complete!

Pro tip: If you don't have a GPU, you can still run this on a CPU with device_map="cpu" and fp16=False — it'll be slow but works for learning.

Compare options / when to choose what

Now, you might wonder: when should I use QLoRA versus plain LoRA or full fine-tuning? Here's a quick comparison:

For most developers, QLoRA is the sweet spot because it lets you experiment with large models without a big budget. The quality loss compared to full fine-tuning is often negligible, especially for instruction tuning and task-specific adaptation.

Another comparison within QLoRA: nf4 vs. fp4 quantization. nf4 (normal float 4) is designed for neural networks and generally yields better accuracy, while fp4 is simpler but slightly less accurate. We'll stick with nf4 in this tutorial.

Troubleshooting & edge cases

Here are the most common pitfalls when running your first QLoRA script and how to fix them:

• CUDA out of memory: Reduce batch size, use gradient accumulation, or lower max_seq_length. Also ensure load_in_4bit=True and device_map="auto" are set.
• bitsandbytes not found: Make sure you have a compatible version installed. On Windows, you may need to install bitsandbytes-windows or use WSL.
• Tokenizer pad token error: Set tokenizer.pad_token = tokenizer.eos_token before training.
• ValueError: Please specifytarget_modules: Check your model's architecture. For some models, you need to list all attention modules (e.g.,q_proj,k_proj,v_proj,o_proj`).
• Loss doesn't decrease: This could be due to a too-high learning rate, wrong dataset format, or not enough epochs. Try lowering the learning rate to 1e-4 or 2e-4 and verify your dataset has proper text field.
• AttributeError: 'PeftModel' object has no attribute 'print_trainable_parameters': Make sure you call get_peft_model and then use model.print_trainable_parameters(). It should work after that.

Pro tip: Always test with a tiny dataset and a small model (e.g., gpt2) first to verify your script works before running on a large model.

What you learned & what's next

You've just successfully run your first QLoRA training script! Let's recap the key takeaways:

• You understand how quantization reduces memory footprint while LoRA adapters capture task-specific patterns.
• You can write a complete training script using transformers, peft, and bitsandbytes.
• You know how to configure BitsAndBytesConfig and LoraConfig for optimal performance.
• You can debug common issues like CUDA OOM and tokenizer pad errors.
• You're ready to apply this knowledge to your own dataset and fine-tune a model for your specific use case.

As a next step, you'll learn how to evaluate your fine-tuned model and measure its performance on a held-out validation set. This is crucial to ensure your model isn't just memorizing the training data but actually generalizing to new inputs. You'll also learn how to deploy your fine-tuned model to production, integrating it into an API or an application.

Before moving on, try this mini-exercise: Use your own dataset (e.g., a few hundred examples of customer support conversations) and run the script above. Experiment with different ranks (r=4, r=16) and compare the training loss. This hands-on practice will solidify your understanding of QLoRA and prepare you for the next lesson on evaluation.