Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 45 ยท Training

Datasets and DataLoaders

Learn how PyTorch's Dataset and DataLoader feed data to a model: batch_size, shuffle, num_workers, collate_fn, samplers, random_split and IterableDataset.

A model never sees your whole dataset at once. It sees it in batches: a few dozen or few hundred examples, stacked into tensors, in a fresh random order each epoch. Producing those batches is the job of two small classes in torch.utils.data. A Dataset answers "how many examples are there, and what is example number i?" A DataLoader wraps a dataset and turns it into an iterator of batches, with shuffling, parallel loading and padding if you need them.

PyTorch itself cannot run in the browser, so the PyTorch code in this lesson is read-only. The live examples are a pure-Python, NumPy-based mini DataLoader that follows the same contract, so you can read, run and change every step that happens between your files and your training loop. It continues the series after Learning Rate Schedules and Gradient Clipping.

Two classes, one pipeline

Dataset Knows its length and returns one sample.
DataLoader Groups samples into batches.

Your training loop only ever talks to the DataLoader.

1 / 2

The Dataset contract

There are two kinds of dataset. The common one is map-style: you subclass Dataset and implement two methods.

Python
from torch.utils.data import Dataset

class SquaresDataset(Dataset):
    def __init__(self, n):
        self.x = torch.arange(n, dtype=torch.float32)

    def __len__(self):
        return len(self.x)                 # how many samples

    def __getitem__(self, index):
        return self.x[index], self.x[index] ** 2    # one (feature, label) pair

__len__ tells the sampler how many indices exist, and __getitem__(i) returns the i-th sample, as a tensor, a tuple, a dictionary, whatever your model needs. Heavy work belongs in __getitem__: opening an image file, decoding it, applying augmentation. The DataLoader calls it lazily, one sample at a time, so memory use stays small even for datasets that do not fit in RAM. Put cheap, one-time setup, such as reading a CSV of file names, in __init__.

For data that is already a set of tensors, TensorDataset saves you the class:

Python
from torch.utils.data import TensorDataset

ds = TensorDataset(features, labels)       # ds[i] -> (features[i], labels[i])

The second kind is iterable-style, covered near the end of this lesson.

The DataLoader

Python
from torch.utils.data import DataLoader

loader = DataLoader(ds, batch_size=32, shuffle=True)

for x_batch, y_batch in loader:
    ...                                    # x_batch has shape (32, ...)

Each iteration over loader is one epoch. Behind the scenes a sampler produces indices, the dataset turns each into a sample, and a collate function stacks the samples into batch tensors.

flowchart LR
    S["Sampler<br/>(indices)"] --> D["Dataset<br/>__getitem__(i)"]
    D --> C["collate_fn<br/>(stack samples)"]
    C --> B["Batch<br/>tensors"]
    B --> T["Training loop"]

The DataLoader chains a sampler, the dataset and a collate function.

The arguments you will use most:

Argument Meaning
batch_size samples per batch (default 1)
shuffle new random order every epoch; use True for training, False for validation
drop_last drop the final batch if it is smaller than batch_size
num_workers worker processes that load batches in the background (default 0 = main process)
collate_fn how a list of samples becomes a batch
sampler, batch_sampler take full control of which indices are drawn
pin_memory page-locked host memory for faster GPU copies
generator a seeded torch.Generator to make shuffling reproducible

A mini DataLoader you can run

This stand-in implements the heart of it: a Dataset base class, an __iter__ that shuffles indices, slices them into batch-sized groups, fetches each sample, and collates. It uses NumPy arrays where PyTorch uses tensors.

Python A Dataset and a mini DataLoader

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ten samples with batch_size=4 give three batches: two full ones and a last one with just two samples. drop_last=True discards that short batch, which matters for layers such as BatchNorm that dislike tiny batches (see Initialization, Normalization and Dropout). With shuffle=True the two epochs visit the samples in different orders. In PyTorch, pass generator=torch.Generator().manual_seed(0) to make that order reproducible.

Do not shuffle the validation or test loader. The order does not change the average metric, but a fixed order makes results easier to compare and debug.

Why mini-batches?

You could compute the gradient on the whole dataset (batch gradient descent), on one sample at a time (stochastic gradient descent) or on a small group (mini-batch). Mini-batches are the standard compromise, because:

  • Memory. The whole dataset rarely fits on the GPU; a batch of 32 to 512 samples does.
  • Speed. GPUs multiply big matrices efficiently, so a batch of 64 costs little more than a batch of 1, but you get many updates per epoch instead of one.
  • Useful noise. A mini-batch gradient is a noisy estimate of the full gradient. That noise helps escape sharp minima and acts as mild regularization.
  • Too large is also bad. Very large batches give smoother but fewer updates, need a larger learning rate and can generalize worse; very small ones are noisy and underuse the hardware. Start with 32 to 128 and tune the learning rate together with the batch size.

The DataLoader creates mini-batches by asking its sampler for batch_size indices, fetching each sample from the dataset and collating them into stacked tensors. The last batch of an epoch is smaller unless you pass drop_last=True.

Python Batch size versus gradient noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mean of the mini-batch gradients matches the full gradient (they are unbiased), while the spread shrinks roughly as one over the square root of the batch size.

Writing your own sampler

A sampler is any iterable of indices with a length. To change the order or selection of samples, subclass torch.utils.data.Sampler and pass it as sampler=:

Python
from torch.utils.data import Sampler, DataLoader

class EvenFirstSampler(Sampler):
    """Yield all even indices, then all odd ones (a toy curriculum)."""
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        yield from range(0, self.n, 2)
        yield from range(1, self.n, 2)

    def __len__(self):
        return self.n

loader = DataLoader(dataset, batch_size=16, sampler=EvenFirstSampler(len(dataset)))

Do not combine sampler= with shuffle=True; the sampler already decides the order. Transformations (resizing, augmentation) belong inside the Dataset, applied in __getitem__, and batch-level work such as padding belongs in collate_fn.

Collating: from samples to a batch

The collate function receives a list of samples (whatever __getitem__ returned) and must return one batch. PyTorch's default, default_collate, handles the common cases: it stacks tensors and NumPy arrays along a new first dimension, converts Python numbers to tensors, and recurses into tuples and dictionaries, so a list of (image, label) pairs becomes a tuple (images, labels).

The default breaks when samples have different shapes. Sentences have different lengths, and you cannot stack a length-3 sequence with a length-5 one. The fix is a custom collate_fn that pads to the longest item in the batch and also returns a mask marking which positions are real:

Python
from torch.nn.utils.rnn import pad_sequence

def pad_collate(batch):
    seqs, labels = zip(*batch)
    lengths = torch.tensor([len(s) for s in seqs])
    padded = pad_sequence([torch.as_tensor(s) for s in seqs], batch_first=True)
    return padded, lengths, torch.tensor(labels)

loader = DataLoader(text_ds, batch_size=16, collate_fn=pad_collate)

Here is the same idea in NumPy, including the failure of a naive stack.

Python Padding variable-length sequences with collate_fn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The batch is padded to the longest sentence in this batch (five tokens), not to a global maximum. A different batch can have a different width. The mask lets later layers ignore the zeros, a topic that returns in Embeddings and Text Data and Attention from Scratch.

Splitting and sampling

random_split

Most of the time you start with one dataset and need training and validation parts. random_split returns non-overlapping Subset views of the same dataset (no data is copied):

Python
from torch.utils.data import random_split

generator = torch.Generator().manual_seed(42)
train_ds, val_ds = random_split(ds, [0.8, 0.2], generator=generator)

Passing a seeded generator makes the split reproducible, which you need so that a validation example does not turn into a training example on the next run. lengths can be counts ([8000, 2000]) or fractions that sum to 1.

Samplers

A sampler decides which indices the loader draws and in which order. shuffle=True quietly uses a RandomSampler; shuffle=False uses SequentialSampler. To change the strategy, pass your own, for example WeightedRandomSampler to oversample a rare class:

Python
from torch.utils.data import WeightedRandomSampler

class_counts = torch.bincount(labels)
sample_weights = 1.0 / class_counts[labels]          # rare class -> big weight
sampler = WeightedRandomSampler(sample_weights, num_samples=len(labels), replacement=True)
loader = DataLoader(ds, batch_size=32, sampler=sampler)   # do not also pass shuffle=True

Splits and sampling, live

Python random_split, weighted sampling and streaming

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data is 90% class 0 and 10% class 1. Drawing with inverse-frequency weights makes the two classes roughly equally likely (0.47 and 0.53 here, sampling noise included), so each batch the model sees is far more balanced. The split is the same every time you reuse the seed.

IterableDataset

Some data has no meaningful length and no random access: a live sensor, a huge log file, lines streamed from a network source. For those you subclass IterableDataset and implement __iter__, usually as a generator. The DataLoader batches whatever the stream yields:

Python
from torch.utils.data import IterableDataset

class LogLines(IterableDataset):
    def __init__(self, path):
        self.path = path

    def __iter__(self):
        with open(self.path) as f:
            for line in f:
                yield parse(line)

loader = DataLoader(LogLines("events.log"), batch_size=64)

Two consequences: shuffle=True is not allowed (there are no indices to shuffle, so shuffle inside your stream with a buffer if you need it), and with num_workers > 0 each worker gets its own copy of the stream and would yield every item, so duplicates appear unless you shard the data by torch.utils.data.get_worker_info(). The last half of the example above shows the batching idea with a plain Python generator.

Parallel loading with num_workers

With num_workers=0, the same process that trains the model also opens files, decodes images and builds batches. While it does, the GPU waits. Setting num_workers=4 starts four worker processes that run __getitem__ and collate_fn ahead of time and hand finished batches to the training process.

Python
loader = DataLoader(
    ds, batch_size=64, shuffle=True,
    num_workers=4,            # background processes
    pin_memory=True,          # faster host-to-GPU copies
    persistent_workers=True,  # keep workers alive between epochs
)

A few practical rules:

  • Good values are roughly the number of free CPU cores, found by experiment. More workers cost memory, because each one holds a copy of the dataset object.
  • On Windows and macOS, worker processes are started with spawn, so training code must sit under if __name__ == "__main__":, and the dataset must be picklable.
  • Random augmentation in workers should be seeded per worker if you care about exact reproducibility (worker_init_fn).
  • If a small dataset already sits in memory as tensors, workers add overhead and num_workers=0 is faster.

Processes cannot run inside this lesson's browser sandbox, so the next TryIt is a timing model, not a measurement: it shows how loading and training overlap. Loading takes 40 ms per batch and a training step takes 15 ms.

Python Why num_workers helps: a timing model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this model the epoch drops from 5.50 s with no workers to 1.54 s with four. Eight workers do not help further, because training itself (15 ms per batch) has become the bottleneck. Real numbers depend on your disk, CPU and augmentations, so measure with a profiler, as covered in Making Training Faster.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which two methods must a map-style Dataset implement?

    A map-style dataset answers how many samples there are (len) and what sample i is (getitem). iter is for IterableDataset.
  2. 2. A DataLoader over 10 samples with batch_size=4 and drop_last=False yields how many batches, and how big is the last one?

    10 divided by 4 gives two full batches and a remainder of 2. With drop_last=True the short last batch would be discarded, leaving 2 batches.
  3. 3. Why do variable-length sequences usually need a custom collate_fn?

    default_collate stacks samples along a new dimension, which needs identical shapes. A custom collate_fn pads each batch to its longest sequence and can return a mask.
  4. 4. You pass a WeightedRandomSampler to a DataLoader. What should you not also pass?

    shuffle and sampler are mutually exclusive: shuffle=True just builds a RandomSampler for you, so DataLoader raises an error if you give it both.
  5. 5. What is the risk of using an IterableDataset with num_workers=4 and no extra code?

    Every worker gets its own copy of the dataset object. Unless iter splits the work using get_worker_info(), each copy yields the entire stream.

Summary

  • A Dataset provides __len__ and __getitem__ (map-style) or __iter__ (iterable-style); do the heavy per-sample work in __getitem__ so loading is lazy.
  • DataLoader adds batching (batch_size, drop_last), shuffling per epoch, a sampler, a collate_fn and optional parallel workers.
  • The default collate stacks equal-shaped samples; use a custom collate_fn to pad ragged data and return a mask.
  • random_split with a seeded generator gives reproducible train and validation subsets; WeightedRandomSampler balances rare classes.
  • num_workers, pin_memory and persistent_workers hide data-loading time; wrap training code in if __name__ == "__main__": on Windows and macOS.

Next, combine everything in The Complete Training Loop.