Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 73 ยท Training in Practice

Mini-Batches, Epochs and Data Loading

Learn how shuffling, batch size, epochs and iterations fit together, write a reusable batch generator, and see why mini-batch gradients are noisy.

So far our training loops pushed the whole dataset through the network for every update. That is fine for 200 points, but a real dataset can hold millions of images that do not fit in memory at once. The standard solution is the mini-batch: take a small random slice of the data, compute the gradient on just that slice, update the weights, and move on to the next slice.

This lesson sorts out the vocabulary (sample, batch, iteration, epoch), shows you how to write a batch generator you can reuse for the rest of the tutorial, and explains with numbers why mini-batch gradients are noisy and how the batch size trades speed against stability. It builds on Gradient Descent, where batch, stochastic and mini-batch descent were first compared.

The vocabulary

Sample One example, such as one image and its label.
Batch A group of samples used for one gradient.
Iteration One update step, on one batch.
Epoch One full pass over the training data.

iterations per epoch = ceil(samples / batch size)

1 / 2

Counting: samples, iterations, epochs

Suppose you have 10,000 training samples and use a batch size of 100. One epoch takes ceil(10000 / 100) = 100 iterations, because each iteration consumes 100 samples. Training for 30 epochs means 3,000 weight updates in total. If the batch size is 128 and the dataset has 1,000 samples, an epoch has ceil(1000 / 128) = 8 iterations: seven full batches of 128 and one last batch of 104.

That last, smaller batch is normal. Most data loaders let you choose to keep it or to drop it (drop_last). Keeping it uses all the data; dropping it keeps every batch the same shape, which matters for a few operations such as batch normalization statistics.

Shuffling and a reusable batch generator

Mini-batches must be random. If your data is sorted (all cats first, then all dogs), unshuffled batches contain a single class, and every update drags the model toward that class and then back again. The usual routine is to reshuffle the order at the start of every epoch, so no two epochs see the batches in the same arrangement.

In NumPy the cleanest way is to shuffle an array of indices instead of copying the data:

Python
order = rng.permutation(len(X))      # a random ordering of 0..n-1
batch_idx = order[start:start + batch_size]
xb, yb = X[batch_idx], y[batch_idx]

Wrap that in a generator and you have a data loader. The next example builds one, prints two epochs on a toy dataset of 10 numbers with a batch size of 4, and then shows that the same seed always gives the same batches.

Python A reusable batch generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every epoch visits all 10 numbers exactly once, in a different order, and the last batch holds only two of them. Because the generator takes the random number generator as an argument, you control the randomness from one place.

Reproducible seeds

Training involves several random choices: the initial weights (Weight Initialization), the shuffling, dropout masks and any data augmentation. To make a run repeatable, create generators from a fixed seed and pass them around, as the example does with np.random.default_rng(5). Two details matter:

  • Use a separate generator for shuffling and for initialization if you want to change one without disturbing the other.
  • Reproducible does not mean identical across machines: floating point sums on different hardware or library versions can differ in the last digits. Aim for results that match closely, not necessarily bit for bit. Experiments and Reproducibility goes deeper.

Why mini-batch gradients are noisy

The gradient from a batch is an estimate of the gradient over the whole dataset. It points roughly the right way, but not exactly, and the smaller the batch, the rougher the estimate. This is statistics, not a bug: averaging B independent samples shrinks the random error in proportion to 1 / sqrt(B). Quadrupling the batch size halves the noise.

We can measure it. The next example makes a linear regression problem with 4,096 samples, computes the exact full-dataset gradient at the starting weights, then draws 400 random batches for each batch size and measures how far the batch gradients fall from the exact one.

Python Measuring gradient noise against batch size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error shrinks from about 15.5 with a single sample to about 0.57 with 1,024. The column error * sqrt(B) stays nearly constant (between 15.5 and 18.3), which confirms the 1 / sqrt(B) law, and on the plot the measured line runs parallel to the dashed theory line. The exact gradient has components of size up to about 6, so a single-sample gradient is mostly noise, while a 256-sample gradient is already a decent estimate.

Notice the cost side of the bargain: going from 1 to 1,024 samples multiplies the computation per step by 1,024 but reduces the noise only about 27-fold (32-fold in theory). Past a certain point, extra samples per step buy very little.

The batch size trade-off

Is noise bad? Not entirely. Small batches give noisy updates, but they give many of them. Large batches give accurate updates, but fewer per epoch, and each costs more. Which wins depends on the hardware and the problem. A rough guide:

Batch size Updates per epoch Gradient quality Typical effect
Very small (1 to 8) Many Very noisy Slow on GPUs (poorly used), needs a small learning rate
Medium (32 to 256) Moderate Good The usual sweet spot
Very large (thousands) Few Very accurate Fast per epoch on big hardware, can need warmup and a larger learning rate

The slight noise of mid-sized batches also acts as a mild regularizer, a reason why training with very large batches sometimes generalizes a little worse unless the learning rate is adjusted.

Let us see this in a real training run. The next example trains the same small network on a noisy moons dataset for 10 epochs with batch sizes 2, 8, 32 and 600 (the full dataset), all with a learning rate of 0.5. It prints the number of updates and the loss after 1, 5 and 10 epochs.

Python The same epochs, four batch sizes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every run looks at exactly the same amount of data (10 epochs of 600 samples), but they differ a lot in how many updates they perform. Full-batch training makes only 10 updates and is still at 0.349 at the end; after the first epoch it had barely started (0.610). A batch of 8 makes 750 updates and reaches 0.273, the best here. Batch 32 (190 updates) ends at 0.325, and batch 2 (3,000 updates) at 0.339: with a rate of 0.5, each of its updates is so noisy that its first-epoch loss (0.436) is worse than the batch-8 run's (0.341) despite four times as many updates. A smaller rate would suit the tiny batch better, which is why the learning rate and the batch size must be tuned together.

When you double the batch size, try a somewhat larger learning rate, and when you halve it, try a smaller one. A common starting point is to scale the rate in proportion to the batch size, then verify with a learning rate range test.

Data loading in practice

Real loaders do more than slice an array:

  • Prefetching and workers: while the GPU trains on batch k, other processes prepare batch k + 1 (decoding images, applying augmentation). Our browser runner has no threads, so everything is in memory.
  • Streaming: huge datasets live on disk or in object storage and are read in shards, with shuffling done in a buffer.
  • Collation: variable-length samples (sentences) are padded to a common length per batch; see Working With Sequences.
  • Stratified or weighted sampling: for imbalanced classes, batches can be built to contain enough of the rare class.
  • Evaluation batches: validation and test data do not need shuffling, and can use any batch size that fits in memory, since no gradients are computed.

PyTorch calls this machinery DataLoader, Keras uses tf.data, and both are conceptually the generator you wrote above.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A dataset has 1,000 samples and the batch size is 128. How many iterations are in one epoch, keeping the final partial batch?

    ceil(1000 / 128) = 8: seven full batches of 128 plus a last batch of 104 samples.
  2. 2. Why should the data be reshuffled at the start of every epoch?

    Unshuffled or sorted data creates biased batches. Reshuffling gives different batch compositions in every epoch.
  3. 3. By roughly how much does the gradient noise change if you multiply the batch size by 4?

    The error of an average falls like 1 / sqrt(B), so a 4 times larger batch halves it.
  4. 4. What is the main downside of very small batches (such as 1) with a large learning rate?

    A single-sample gradient is mostly noise. Small batches usually need a smaller learning rate.
  5. 5. How can you make a shuffling step repeatable between runs?

    Passing a seeded generator, such as np.random.default_rng(5), gives the same permutations every run.

Summary

  • A batch is the group of samples used for one gradient; an iteration is one update; an epoch is one full pass over the training data.
  • Iterations per epoch is ceil(samples / batch size); the last batch may be smaller (or dropped).
  • Reshuffle an index array each epoch and slice it; a generator that takes a seeded random generator is reproducible.
  • Batch gradients are noisy estimates; the error shrinks like 1 / sqrt(batch size).
  • Small batches give many noisy updates, huge batches give few accurate ones, and 32 to 256 is a common sweet spot.
  • Tune the learning rate together with the batch size.

Next: Overfitting, Underfitting and Regularization.