Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 41 ยท Data and Experiment Tracking

Data Versioning and Fingerprints

Learn why Git cannot version big datasets, how content hashes identify a dataset exactly, and how tools like DVC and lakeFS apply the same idea. Includes a runnable mini data store.

Code has Git. Data usually has a folder called data_new_FINAL. When a model behaves strangely, the first question is "what data was it trained on?", and without versioning nobody can answer. Data versioning gives every dataset an identity that is stable, comparable and linked to the models trained from it.

Git is not built for this. It stores full copies of changed files, so a 5 GB dataset edited ten times becomes a huge repository, and diffs of binary data are meaningless. Data versioning tools keep the large files outside Git and store small pointer files inside it.

The core idea: content addressing

A content hash (for example SHA-256) turns any bytes into a short fixed-length fingerprint. Identical content always gives the identical hash; a single changed byte gives a completely different one. If you name a dataset by its hash, the name is the content.

Python A hash identifies content

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fingerprinting a DataFrame

For tables, hash the content in a canonical form so row order or index quirks do not change the identity unless you want them to.

Python Fingerprint a pandas DataFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Decide whether row order is part of the dataset's identity. If not, sort by a key before hashing.

A mini data store

Here is a stand-in for what DVC does under the hood. Files are stored under their hash (the cache), and a tiny pointer (name plus hash) is what you commit to Git. Rolling back means loading an older pointer.

Python A content-addressed data store

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pointers are tiny text files, so Git handles them well. The matching big files live in cheap object storage (S3, GCS, Azure Blob, a shared disk) and are fetched on demand. Identical files are stored once.

What the real tools look like

DVC (Data Version Control) uses exactly this design. These commands are read-only because they need Git and a filesystem:

Terminal
dvc init
dvc add data/train.csv          # creates data/train.csv.dvc (the pointer)
git add data/train.csv.dvc data/.gitignore
git commit -m "Add training data v1"
dvc remote add -d storage s3://my-bucket/dvc
dvc push                        # upload data to the remote
dvc pull                        # on another machine, download it
git checkout v1.0 && dvc checkout   # go back to the old data

Other approaches:

Approach Idea Good for
DVC Pointers in Git, data in remote storage File-based datasets, simple stacks
lakeFS Git-like branches over an object store Large data lakes, atomic data commits
Delta Lake / Iceberg Table formats with time travel Tables on Spark or warehouses
Dataset snapshots in a warehouse Dated tables or views SQL-based teams

Linking data version to model version

The point of all this is a record like: "model churn-7 was trained on data a3f9c2d1". Store the hash in the run record so you can move from model to data and back.

flowchart LR
  D["Dataset hash a3f9c2d1"] --> R["Run 42"]
  C["Code commit 9be21f0"] --> R
  P["Params config 7c1b"] --> R
  R --> M["Model churn v7"]

A model is identified by the triple of data, code and parameters.

Practical rules

  • Never overwrite a dataset in place. Write new versions, keep old ones.
  • Version raw data and derived data differently. Raw data is immutable; derived tables can be re-created from raw data plus code.
  • Record the query or code that produced a snapshot, not only the result.
  • Keep a small sample in the repo for tests and demos.
  • Plan retention. Old versions cost money; decide how long to keep them, and mind privacy rules that may require deleting personal data.

Sampling and train/test membership are part of the dataset's identity. Version the split too (see the reproducible script).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is Git alone a poor fit for large datasets?

    Repositories balloon in size and file diffs of data are meaningless.
  2. 2. What does a content hash guarantee?

    Hashes identify content; they say nothing about quality.
  3. 3. What is committed to Git when using DVC?

    Pointers go in Git; the data goes to remote storage.
  4. 4. Which triple identifies a trained model for reproducibility?

    Data, code and configuration together determine the model.

Summary

  • Datasets need identities; content hashes provide exact, comparable ones.
  • Keep big files outside Git and commit small pointers; DVC and lakeFS follow this pattern.
  • Canonicalize before hashing and decide whether row order matters.
  • Record the data hash in every run so models trace back to their data.

Next: Data Validation and Schemas