Create an Instruction Dataset

You've trained a base model, and it's still answering like a general-purpose chatbot — not your product. The difference between a generic response and a precise, on-brand answer isn't the model. It's the data. If you're fine-tuning an LLM, your instruction-style dataset is the single most important asset you'll create. It determines whether your model follows your format, your tone, and your rules. A poorly built dataset can silently degrade performance, forcing you into endless retraining cycles. In this lesson, you'll learn how to create an instruction-style dataset that turns raw examples into the exact behavior you want — without guessing.

The problem this lesson solves

Raw text data doesn't teach a language model to follow instructions. If you hand a base model a bunch of customer reviews, it won't automatically learn to classify sentiment or generate a structured response. It learns patterns, but it doesn't know the task you expect it to perform. Without explicit instructions, the model treats your input as open-ended text and produces something unpredictable. The result: you spend hours on prompt engineering, yet the output remains inconsistent.

Fine-tuning on an instruction-style dataset solves this by pairing each example with a clear task. You're not just feeding text; you're teaching the model the mapping between an instruction (like "Classify this review as positive, negative, or neutral") and the desired output ("positive"). This transforms a generic language model into a task-specific assistant.

Pro tip: The quality of your instruction dataset directly correlates with the model's instruction-following ability. Garbage in, garbage out — even for fine-tuning.

Core concept / mental model

Think of an instruction-style dataset as a set of flashcards. Each card has two sides: the instruction (the question or command) and the response (the expected answer). When you fine-tune, you're showing the model thousands of these cards so it learns the rule behind each mapping.

When you create an instruction-style dataset, you're creating the foundation for instruction tuning. It transforms raw data into structured (input, output) pairs that teach a language model to follow specific commands. This is the essential preparation step before any supervised fine-tuning run.

The key components of an instruction-style dataset are:

• Instruction: The task description, e.g., "Summarize the following email."
• Input (optional): The context or data the instruction operates on, e.g., the email text.
• Output: The expected response, e.g., the summary.

A mental model: each instance is a contract between you and the model. You specify what you want, and the model learns to fulfill that specification. The more consistent you are with your phrasing and format, the better the model internalizes the rule.

How it works step by step

Creating an instruction-style dataset follows a structured pipeline. Let's break it down.

Step 1: Define the task and output format

Before you write a single example, decide what behavior you want. What is the input? What is the expected output? For a classification task, the output is a label. For a generation task, the output is a paragraph or a JSON object.

Write a task description that a human could follow without additional context. For example: "Given a product review, classify the sentiment as positive, negative, or neutral."

Step 2: Collect or generate raw data

Gather examples that represent the real-world inputs your model will see. Sources include:

• Logs from your existing application
• Public datasets from your domain
• Synthetic data generated by a teacher LLM
• Manually crafted examples

Pro tip: Include edge cases and borderline examples. A classifier trained only on clear-cut cases will fail on ambiguous ones.

Step 3: Create instruction–output pairs

For each raw data point, write the instruction and the expected output. Ensure the instruction is clear, specific, and consistent across all examples. For example:

Step 4: Apply a prompt template

To maintain consistency, wrap each pair in a standard template. This helps the model learn the exact format it should follow. A common template for chat-based models is:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:
{output}

Step 5: Split and validate

Split your dataset into train, validation, and test sets. Validate the data for missing fields, duplicate examples, and template correctness. A quick script can catch errors before you spend hours training.

Hands-on walkthrough

Let's build a small instruction-style dataset in Python. We'll use the datasets library to structure our data and a simple validation script.

First, create a CSV file with your examples:

review,sentiment
"This product is amazing!",positive
"The battery died after a week.",negative
"It's okay, but not great.",neutral

Now load and transform it into an instruction-style dataset:

import pandas as pd
from datasets import Dataset

# Load raw data
df = pd.read_csv("reviews.csv")

# Apply instruction template
def format_instruction(review):
    return f"Classify the sentiment of this product review: {review}"

df["instruction"] = df["review"].apply(format_instruction)
df["output"] = df["sentiment"]

# Create Hugging Face dataset
dataset = Dataset.from_pandas(df[['instruction', 'output']])
print(dataset[:3])

Expected output:

{'instruction': 'Classify the sentiment of this product review: This product is amazing!', 'output': 'positive'}
{'instruction': 'Classify the sentiment of this product review: The battery died after a week.', 'output': 'negative'}
{'instruction': 'Classify the sentiment of this product review: It's okay, but not great.', 'output': 'neutral'}

Next, validate the dataset for common issues:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Check token lengths
lengths = [len(tokenizer.tokenize(f"{instr} {out}")) for instr, out in zip(dataset['instruction'], dataset['output'])]
print(max(lengths), min(lengths))

# Verify all outputs are in expected labels
valid_outputs = {'positive', 'negative', 'neutral'}
assert all(out in valid_outputs for out in dataset['output']), "Invalid output label found"
print("Validation passed")

This script catches token overflows and invalid labels before training.

Compare options / when to choose what

You can source your instruction-style dataset in several ways. Here's a table of options:

When to choose what: - If you need high accuracy for a narrow task, invest in human-curated data. - If you're exploring or prototyping, start with a public dataset. - If you have limited real data, generate synthetic examples using a powerful model.

Troubleshooting & edge cases

Here are common problems and how to fix them when creating an instruction-style dataset:

Missing or empty fields

Some examples may have missing instructions or outputs. This happens when your source data has nulls. Fix: Drop or fill them with a placeholder during preprocessing.

Inconsistent label formatting

If some labels are "positive" and others are "pos", the model will get confused. Fix: Normalize all labels to a single canonical form.

Token length exceeding model limit

Long instructions get truncated, losing context. Fix: Set a max token length and truncate or filter out overly long examples.

Overlapping instruction phrasing

If you use different phrasings for the same task (e.g., "classify sentiment" vs. "what is the sentiment?"), the model may not generalize well. Fix: Stick to one template per task type.

What you learned & what's next

You now understand how to create an instruction-style dataset — the backbone of any fine-tuning project. You learned the core concept: mapping instructions to outputs using a consistent template. You applied this in a hands-on exercise, building a small dataset and validating it. You also know how to choose between human-curated, synthetic, or public datasets, and how to troubleshoot common issues.

Key takeaways from this lesson:

• An instruction-style dataset maps prompts to desired outputs, teaching a model to follow instructions.
• Data quality and consistency matter more than raw volume.
• Step-by-step creation includes collection, templating, splitting, and validation.
• Choose your data source based on your task constraints.
• Validate token lengths and label consistency to avoid training failures.

Next step: Now that your instruction-style dataset is ready, you're prepared for the next lesson in this track: Supervised fine-tuning with Hugging Face Trainer. You'll load this dataset, tokenize it, and train a model on it. Master this data preparation, and the training will be smooth sailing.