Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 38 ยท Datasets and Preprocessing

Loading Datasets with the datasets Library

Load datasets from the Hub and local files with load_dataset, understand Dataset and DatasetDict, Arrow memory mapping, features, indexing, filter and select.

Models are only half of machine learning; the other half is data. The datasets library gives you a uniform way to load text, image and audio datasets from the Hub or from your own files, and to process them quickly without running out of memory.

Its secret is Apache Arrow: a dataset is stored on disk as columnar Arrow files that are memory-mapped, so even a dataset much larger than your RAM can be indexed and iterated without loading it all. Real library code is read-only on this site; each idea also has a runnable stand-in.

load_dataset

Python
from datasets import load_dataset

ds = load_dataset("imdb")                  # from the Hub
print(ds)
# DatasetDict with 'train', 'test' and 'unsupervised' splits

The result is a DatasetDict: a dictionary of splits, each a Dataset. Files are downloaded once and cached under HF_HOME.

You can load part of a split, or load local files:

Python
small = load_dataset("imdb", split="train[:2000]")          # first 2000 rows
csv   = load_dataset("csv", data_files="reviews.csv")
js    = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})
txt   = load_dataset("text", data_files="corpus.txt")

Supported local formats include CSV, JSON and JSON Lines, text, Parquet, Arrow and folders of images or audio (imagefolder, audiofolder).

A Dataset is a table

A Dataset is a list of rows with named columns and typed features:

Python
ds = load_dataset("imdb", split="train")
ds.column_names          # ['text', 'label']
ds.features              # text: string, label: ClassLabel(names=['neg', 'pos'])
len(ds)                  # number of rows
ds[0]                    # one row: a dict
ds[:3]                   # three rows: a dict of lists (columns)
ds["text"][:3]           # one column

Notice the difference: an integer index gives a row as a dict, a slice or column name gives a dict of lists / list. Indexing is lazy, reading from disk only what you ask for.

The class below is a stand-in: a tiny pure-Python imitation of the Dataset access patterns. It stores data column by column, like Arrow does.

Python Stand-in: a mini Dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The real Dataset has the same shapes (row dict, dict of lists, column list) plus the Arrow storage and many more methods.

Features and class labels

ds.features describes each column. A ClassLabel maps integers to names, which is how the label 1 becomes "pos":

Python
labels = ds.features["label"]
labels.names                 # ['neg', 'pos']
labels.int2str(1)            # 'pos'
labels.str2int("neg")        # 0

You will pass these names to the model as id2label/label2id so pipelines print readable labels.

Selecting, filtering, sorting, shuffling

Python
ds.select(range(100))                    # first 100 rows
ds.filter(lambda row: len(row["text"]) > 200)
ds.sort("label")
ds.shuffle(seed=42)                      # always set a seed
ds.rename_column("label", "labels")      # the Trainer expects "labels"
ds.remove_columns(["text"])
ds.set_format("torch")                   # return PyTorch tensors when indexing

These return new datasets and, thanks to Arrow, do not copy all the data. Results are cached on disk, so the second run of an expensive step is instant. If you change a function but the cache hides it, pass load_from_cache_file=False.

Inspecting data before modelling

Always look at your data first: class balance, text lengths, duplicates and missing values will explain many model problems.

Python Stand-in: quick data audit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Duplicates between training and test data cause inflated scores (data leakage), so check for them before trusting an evaluation.

Other modalities

Images and audio use the same API with different feature types: Image() decodes to a PIL image on access, Audio(sampling_rate=16000) decodes and resamples audio. Large media stays on disk and is decoded lazily when you read a row.

Python
from datasets import load_dataset, Audio

speech = load_dataset("PolyAI/minds14", "en-US", split="train")
speech = speech.cast_column("audio", Audio(sampling_rate=16_000))

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can datasets handle data larger than RAM?

    Memory mapping lets the library read just the parts you access.
  2. 2. What does ds[0] return?

    An integer index gives a row dict; a slice or column name gives columns.
  3. 3. What is a DatasetDict?

    load_dataset returns a DatasetDict when the data has several splits.
  4. 4. Why shuffle with a fixed seed?

    A seed makes experiments repeatable.

Summary

  • load_dataset loads Hub datasets and local CSV, JSON, text, Parquet and media folders.
  • A Dataset is a typed, memory-mapped table; DatasetDict groups splits.
  • Row access gives dicts, slices give dicts of lists, column names give lists.
  • Inspect balance, lengths, duplicates and empties before modelling.

Next: Preprocessing with map