Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Training and Scaling

Scaling Laws and Compute

Learn how loss follows power laws in parameters, data and compute, estimate training FLOPs, apply the Chinchilla token-per-parameter rule and fit a power law in Python.

Training a frontier model costs millions of dollars, so teams cannot afford to guess. Scaling laws are the empirical finding that makes planning possible: a model's loss falls smoothly and predictably as you increase parameters, data, and compute. Small experiments can therefore forecast the result of a huge run, and the same laws tell you how to split a fixed budget between a bigger model and more training tokens.

Loss is a power law

Researchers trained many models of different sizes and found that, when other factors are not the bottleneck, test loss follows a power law plus a floor:

L(N) โ‰ˆ E + A / N^ฮฑ

N is the number of parameters, E is the irreducible loss (the entropy of text itself), and ฮฑ is a small exponent (roughly 0.3 for parameters). A power law is a straight line on a log-log plot. Each doubling of size reduces the reducible part of the loss by the same multiplicative factor, which is why gains continue but with diminishing absolute returns.

Python Fit a power law to loss versus model size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data here are synthetic (generated from a known law), because real training curves belong to specific labs. The method is exactly what researchers do: fit small runs, then extrapolate to a size you have not trained.

Estimating training compute

A transformer's forward pass costs about 2N floating-point operations (FLOPs) per token (one multiply and one add per parameter), and the backward pass costs about twice that. Training therefore costs about:

C โ‰ˆ 6 ยท N ยท D

where N is parameters and D is training tokens. This rule of thumb ignores attention's quadratic term, which is small except at very long contexts.

Python Compute, time and cost estimates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real throughput depends on hardware and software efficiency, so treat the GPU-day numbers as an order-of-magnitude illustration, not a quote.

The compute-optimal split (Chinchilla)

For a fixed compute budget C, should you train a bigger model on fewer tokens or a smaller one on more? Early large models were trained on relatively few tokens per parameter. The Chinchilla study found that loss is minimized by scaling parameters and tokens together, with roughly 20 tokens per parameter. Substituting D = 20 N into C = 6 N D gives N = sqrt(C / 120).

Python Compute-optimal model size for a budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why many modern models over-train

Chinchilla optimizes training cost. But a model is trained once and then served many times, so inference cost matters too. A smaller model trained on many more tokens than 20 per parameter costs more to train but is cheaper to run for every later user. This is why recent open models of 7B to 70B parameters are trained on several trillion to over ten trillion tokens, hundreds of tokens per parameter, far beyond the compute-optimal ratio. Loss still keeps improving, just with diminishing returns.

Python Tokens per parameter in practice

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Emergence and its caveats

Some abilities appear to switch on suddenly as models grow, such as multi-step arithmetic or following unusual instructions. Part of this is a measurement effect: accuracy that requires every step to be right can stay near zero and then jump even when the underlying per-step skill improves smoothly. Smooth metrics (like per-token loss on the answer) often show gradual improvement. Predicting which capabilities appear, and when, is still an open research question.

Limits of the laws

  • They describe loss, not usefulness; benchmarks and real tasks can behave less smoothly.
  • They assume data quality stays constant. Better data shifts the curve (see Data Pipelines for Pretraining).
  • High-quality text is finite, which raises the question of repeating data (a few epochs help, many hurt) and of synthetic data.
  • Recent work scales inference-time compute too: letting a model think longer can substitute for a larger model on some tasks.

Planning a training run

Power laws Loss falls predictably with size, data and compute.
C = 6ND Estimate FLOPs from parameters and tokens.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What shape does loss versus model size take on a log-log plot?

    A power law is linear in log-log coordinates.
  2. 2. About how many FLOPs does training cost for N parameters and D tokens?

    Forward costs about 2ND and backward about 4ND, totaling 6ND.
  3. 3. What did the Chinchilla result suggest?

    For a fixed training budget, parameters and tokens should grow in proportion.
  4. 4. Why are many open models trained on far more than 20 tokens per parameter?

    Training is paid once, inference many times, so over-training a smaller model lowers lifetime cost.

Summary

  • Loss follows power laws in parameters, data and compute, so small runs predict large ones.
  • Training compute is about 6ยทNยทD FLOPs.
  • Chinchilla suggests roughly 20 tokens per parameter for compute-optimal training; over-training smaller models reduces serving cost.
  • "Emergent" abilities are partly a measurement effect, and scaling laws describe loss, not usefulness.
  • Data quality and inference-time compute are additional levers.

Next: Fine-tuning vs Prompting