Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 38 ยท Datasets and Preprocessing

Splits, Streaming and Custom Data

Create reliable train, validation and test splits, avoid data leakage, stream huge datasets, build a Dataset from your own records and push it to the Hub.

A model is only as trustworthy as the data it is evaluated on. This lesson covers how to split data correctly, how to process datasets that do not fit on disk, and how to turn your own records (a spreadsheet, a database export, logs) into a proper Dataset ready for training and sharing.

Train, validation and test

  • Train: the model learns from it.
  • Validation: used while developing, to choose settings and when to stop.
  • Test: touched once at the end, to report an honest score.

If you tune on the test set, it stops being a fair measure. Many Hub datasets ship train and test only; carve a validation set from train.

Python
split = ds["train"].train_test_split(test_size=0.1, seed=42, stratify_by_column="label")
train_ds, val_ds = split["train"], split["test"]

stratify_by_column keeps the class proportions equal in both parts, which matters for imbalanced labels. Always fix the seed so the split is reproducible.

The stand-in below is a plain-Python stratified split so you can see why stratification helps.

Python Stand-in: stratified versus plain split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With only 10% positives, a random split can easily give the test set far fewer (or more) positives than expected; the stratified version always gets about 2 of 10.

Avoiding leakage

Data leakage means information from the test data sneaks into training, producing scores that fall apart in production. Typical causes:

  • Duplicate or near-duplicate rows on both sides of the split. Deduplicate before splitting.
  • Several rows from the same user, document or patient in both train and test. Split by the group (for example by user id), not by row.
  • Time series split randomly. Split by time so the test period is later than the training period.
  • Preprocessing statistics (such as normalisation constants) computed on all data. Compute them on train only.
Python Stand-in: group-aware split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Streaming huge datasets

Some datasets are hundreds of gigabytes. With streaming=True, load_dataset returns an IterableDataset that reads examples over the network as you iterate, with no download and no disk cache:

Python
stream = load_dataset("allenai/c4", "en", split="train", streaming=True)

for example in stream.take(3):
    print(example["text"][:80])

stream = stream.shuffle(seed=42, buffer_size=10_000)   # approximate shuffle with a buffer
tokenized = stream.map(tokenize, batched=True)

An iterable dataset has no length and no random access (stream[5] does not work). You can take, skip, filter, map and shuffle with a buffer. The Trainer needs max_steps instead of epochs when the length is unknown.

The following stand-in shows the idea with a Python generator: processing happens one item at a time, so memory stays constant however long the stream is.

Python Stand-in: streaming with generators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Nothing is computed until you ask for items, which is exactly how an IterableDataset behaves.

Building a dataset from your own data

From a Python dict, a list of records or a pandas DataFrame:

Python
from datasets import Dataset, DatasetDict, ClassLabel, Features, Value

records = {
    "text": ["refund not received", "love the new design", "app crashes on start"],
    "label": [0, 2, 1],
}
features = Features({
    "text": Value("string"),
    "label": ClassLabel(names=["billing", "bug", "praise"]),
})
ds = Dataset.from_dict(records, features=features)

# pandas: Dataset.from_pandas(df); a generator for big data: Dataset.from_generator(fn)
dataset = DatasetDict({"train": ds, "validation": ds})

Declaring Features with a ClassLabel stores readable class names with the data, so the model's id2label can be taken straight from them. For big inputs, Dataset.from_generator builds the Arrow file incrementally without holding everything in memory.

Reading messy files first

Real files need cleaning before they become a dataset. The stand-in reads a CSV with Python's csv module, drops bad rows and builds columns, the same steps you would do before Dataset.from_dict.

Python Stand-in: from CSV text to clean columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Saving and sharing

Python
ds.save_to_disk("my_dataset")                  # Arrow files, reload with load_from_disk
ds.to_parquet("data.parquet")                  # portable file
dataset.push_to_hub("my-name/support-tickets")  # needs a write token

When you share data, add a dataset card that states the source, licence, collection method, labelling process, known biases and any personal data. Remove or anonymise personal information before publishing.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use stratify_by_column in train_test_split?

    Stratification matters most when classes are imbalanced.
  2. 2. Rows from the same user appear in both train and test. What is this?

    The model can memorise a user's style, so test scores are too optimistic. Split by group instead.
  3. 3. What does streaming=True return?

    Streaming reads examples lazily with no full download and no random access.
  4. 4. Which step should come before splitting data?

    Deduplicating first prevents identical examples from landing in both train and test.

Summary

  • Keep separate train, validation and test sets; fix seeds and stratify when classes are imbalanced.
  • Prevent leakage: deduplicate first, split by group or time when examples are related.
  • Use streaming for huge data; use from_dict, from_pandas or from_generator for your own records.
  • Declare features with ClassLabel, save to disk or the Hub, and write a dataset card.

Next: The Trainer and TrainingArguments