Version Datasets with DVC

Your model's accuracy is only as good as the data that trains it, but if you can't reproduce the exact dataset version behind a breakthrough result, your "reproducible pipeline" is fiction. Git can track your code, but it collapses when you try to commit a 50GB image folder or a CSV that changes every hour. That's where DVC (Data Version Control) steps in — it gives your datasets, models, and experiments the same versioning superpowers Git gives your code, without bloating your repository or slowing down your team.

The problem this lesson solves

Every AI engineer has hit this wall: you train a model, it performs brilliantly, and two weeks later you can't figure out why — because the dataset changed, someone overwrote the "final" CSV, or your teammate's "updated" version has different preprocessing. Git alone can't handle large binary files; try git add on a 10GB dataset and your repo becomes a monster that breaks cloning, slows CI, and eventually forces a forceful history rewrite.

DVC solves this by treating your data like code: it tracks metadata in Git and stores the actual file content in a separate remote (S3, GCS, SSH, or even a local folder). When you commit code, you also commit a tiny pointer file that says "this exact dataset, this exact version." Now every experiment is tied to a specific dataset snapshot, and you can roll back to the exact data that produced your best model.

But DVC isn't just about storage — it's about reproducibility. When you run dvc repro, DVC rebuilds your pipeline from scratch, ensuring every artifact is derived from the correct data versions. This turns "works on my machine" into "works on any machine, anywhere."

Core concept / mental model

Think of DVC as a time machine for your data. While Git is a version control system for text, DVC is a version control system for binary assets. It works by:

• Tracking metadata: A small .dvc file (or an entry in dvc.yaml) records the checksum (MD5) of your dataset.
• Storing content separately: The actual data lives in a remote cache (a directory, cloud bucket, or server).
• Linking versions: When you commit code, Git commits the tiny pointer, not the data. When you need a specific version, DVC pulls the exact bytes from the remote.

Key terms

• DVC remote: A storage location for your data (e.g., s3://my-bucket, ssh://server/path, or a local dir).
• .dvc file: A small YAML file that tracks a single data file or directory.
• Cache: DVC's internal storage (usually .dvc/cache) where data content is stored by checksum.
• Pipeline: A series of DVC stages (dvc.yaml) that define how data transforms into models.

Analogy: Git for code, DVC for data

How it works step by step

DVC's versioning flow follows a simple pattern, mirroring Git's workflow but applied to data.

1. Initialize DVC in your project: dvc init (creates .dvc/ and adds it to .gitignore).
2. Add your dataset to DVC: dvc add data/raw/ — this creates data/raw.dvc and a cache entry.
3. Configure a remote (if you want to share): dvc remote add -d myremote s3://mybucket/dvcstore (or gcs://, ssh://, or a local path).
4. Push data to remote: dvc push (uploads contents to the remote).
5. Commit the metadata to Git: git add data/raw.dvc .dvc/config then git commit -m "Add dataset v1".
6. When data changes, run dvc add data/raw/ again — it will detect changes, update the .dvc file, and you commit a new version.
7. To reproduce an old result, git checkout <commit> and dvc checkout — DVC pulls the exact dataset version from the remote.

Versioning multiple datasets and models

You can version not just raw data, but also processed datasets, trained models, and evaluation metrics. Each becomes a DVC-tracking target. This lets you answer: "Which dataset version produced this model accuracy?"

Hands-on walkthrough

Let's walk through a complete workflow — from installing DVC to reproducing an experiment. We'll use a local directory as our remote for simplicity, but the commands translate directly to cloud storage.

Prerequisites

• Python 3.10+
• pip install dvc (or pip install 'dvc[s3]' for S3 support)

Step 1: Initialize a Git repo and DVC

mkdir my-ai-project && cd my-ai-project
git init
dvc init

Step 2: Add a dataset

Create a fake dataset, then add it to DVC:

mkdir -p data/raw
for i in $(seq 1 100); do echo "$i,$((RANDOM % 100))" >> data/raw/data.csv; done
dvc add data/raw

After dvc add, you'll see data/raw.dvc created. Its content looks like:

outs:
- md5: a1b2c3...
  size: 12345
  path: data/raw

Step 3: Configure a remote and push

dvc remote add -d myremote /tmp/dvcstore
dvc push

Now commit the metadata to Git:

git add .
git commit -m "Add raw dataset v1"
git tag -a v1-data -m "Dataset v1"

Step 4: Modify the dataset and create a new version

Append more rows, then re-add:

for i in $(seq 101 200); do echo "$i,$((RANDOM % 100))" >> data/raw/data.csv; done
dvc add data/raw
git add data/raw.dvc
git commit -m "Add raw dataset v2"
git tag -a v2-data -m "Dataset v2"
dvc push

Step 5: Reproduce an old version

Forget which dataset you used for your best model? Check out the tag and run dvc checkout:

git checkout v1-data
dvc checkout
cat data/raw/data.csv | wc -l  # Should show 100, not 200

Pro tip: dvc checkout will update your working tree to match the .dvc file's checksum. If you have local changes, dvc checkout may overwrite them — commit or stash first!

Step 6: Define a pipeline (optional but powerful)

Create dvc.yaml to track a data processing step:

stages:
  preprocess:
    cmd: python src/preprocess.py data/raw data/processed
    deps:
    - data/raw
    - src/preprocess.py
    outs:
    - data/processed

Now run dvc repro to execute the stage. DVC will only rerun if dependencies changed, and it will track the output version too.

Compare options / when to choose what

DVC isn't the only tool for dataset versioning. Here's a comparison:

When to choose DVC:

• You need pipeline reproducibility along with data versioning.
• You're already using Git and want a minimal learning curve.
• You want to version models and metrics as well as datasets.
• You need to collaborate across teams with large datasets (using cloud storage).

When to choose Git LFS:

• You only need to version a few large files, not full datasets.
• You don't need pipeline stages or experiment tracking.
• Your datasets are under a few GB.

When to avoid DVC:

• You need real-time data streaming or continuous updates (DVC is snapshot-based).
• You have an existing data lake with its own versioning.

Troubleshooting & edge cases

Even with DVC, things go wrong. Here's how to fix the common issues:

1. dvc add is slow or hangs

• Cause: Large dataset or slow filesystem.
• Fix: Use --no-compute-checksum to skip checksum computation initially, or use a dataset on a faster disk. Also, ensure your .dvcignore excludes temporary files (e.g., *.tmp).

2. dvc checkout overwrites my changes

• Cause: DVC sees the working tree as dirty.
• Fix: Commit or stash your changes in Git first, then run dvc checkout. Or use --force if you're sure.

3. Remote push fails with authentication error

• Cause: Missing credentials for cloud provider.
• Fix: Set environment variables (e.g., AWS_ACCESS_KEY_ID) or use dvc remote modify to set credentials. For S3, consider using ~/.aws/credentials.

4. DVC tracks a file inside .dvcignore

• Cause: Wrong ignore rules.
• Fix: Ensure .dvcignore is correctly formatted; patterns like data/*.tmp work. Test with dvc status.

5. Version conflicts between teammates

• Cause: Two people updated the same dataset and pushed to remote.
• Fix: Use Git tags to mark versions, and always pull (dvc pull) before making changes. Use dvc status to see if your local cache is in sync.

6. DVC doesn't detect a dataset change

• Cause: File contents changed but permissions or metadata changed, not content.
• Fix: DVC uses MD5 of content — if you change metadata like file timestamps, it won't trigger. That's usually fine. If you need to force an update, use dvc add --force.

What you learned & what's next

You now understand use dvc for dataset versioning — from the pain of untracked data to setting up DVC in a project, adding datasets, pushing to a remote, and rolling back to exact versions. You can explain the core idea behind DVC and complete a practical exercise that mirrors real AI workflows.

Next steps: In the next lesson, you'll learn how to integrate DVC with experiment tracking tools like MLflow or Weights & Biases. You'll also dive into dvc.yaml pipelines and how to track model performance across dataset versions. This is key for building reproducible AI pipelines that your team can trust.

Now go version your data — your future self will thank you when you reproduce that state-of-the-art result.