Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 37 ยท Getting Started

A Short History of LLMs

Trace language modeling from n-grams and RNNs to the Transformer, BERT, GPT and instruction-tuned assistants, with a small RNN versus attention demo.

Large language models look like a sudden arrival, but they are the end of a long line of ideas: count tables, neural networks that read text one token at a time, attention, and finally models trained at huge scale. Knowing the sequence explains why today's architecture looks the way it does and what problem each step solved.

A timeline

Dates and model names are given for orientation. Sizes are approximate and vendors do not always publish exact figures.

flowchart LR
    A["1950s-2000s<br/>N-gram models"] --> B["2013<br/>Word vectors"]
    B --> C["2014-2016<br/>RNN, LSTM, seq2seq<br/>+ attention"]
    C --> D["2017<br/>Transformer"]
    D --> E["2018<br/>BERT and GPT-1"]
    E --> F["2020<br/>GPT-3, scaling laws"]
    F --> G["2022<br/>Instruction tuning, RLHF,<br/>chat assistants"]
    G --> H["2023 onward<br/>Open weights, long context,<br/>multimodal, reasoning models"]

Each arrow removed a bottleneck: data sparsity, fixed word meanings, sequential processing, then the lack of instruction following.

Stage 1: counting (n-grams)

Statistical language models counted word sequences, as you built in Next-Token Prediction. They powered early speech recognition and spelling correction. Their limits were sparsity and a context of a few words.

Stage 2: dense word vectors

Around 2013, methods such as word2vec learned a vector for each word from its neighbors so that similar words got similar vectors. This solved the "no sharing" problem for single words, but each word still had one vector regardless of meaning (the same vector for a river bank and a savings bank). Token Embeddings shows how vectors work and how they are made context dependent.

Stage 3: recurrent networks

A recurrent neural network (RNN) reads tokens one at a time and keeps a hidden state that summarizes everything so far. LSTMs and GRUs added gates so information could survive longer. Sequence-to-sequence models used an RNN to encode a sentence and another to decode a translation, and attention let the decoder look back at every encoder state instead of one compressed vector.

RNNs have two weaknesses. They are sequential, so they cannot use a GPU's parallelism across the time dimension during training, and information must travel through every intermediate step, which makes very long-range dependencies hard to learn.

Python Why long-range information fades in an RNN

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With these weights the state shrinks quickly: the signal from the first token is gone after a few dozen steps. Real RNNs are trained, and gates help, but information still passes through many nonlinear steps, so the same shrinking (or exploding) gradient problem limits them.

Stage 4: the Transformer (2017)

The Transformer discarded recurrence. Every token can attend directly to every other token in a single layer, so the path between any two positions is one step long, and all positions are processed in parallel. Parallel training made it practical to use far more data and far larger models. The complete mechanism is in Self-Attention Step by Step.

Python Path length: RNN versus attention

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trade is visible: a path of one step, but a number of pairwise scores that grows with the square of the length. A large part of the later engineering story (see Attention Variants) is about taming that cost.

Stage 5: pretraining, then adapting

In 2018 two families appeared that used the Transformer in different ways.

  • BERT (an encoder) learned by filling in masked words and used both left and right context. It was fine-tuned for classification and extraction tasks.
  • GPT (a decoder) learned by predicting the next token and generated text left to right.

Decoder models turned out to scale and generalize better for open-ended use. GPT-2 (2019) showed fluent text; GPT-3 (2020, 175 billion parameters) showed in-context learning: given a few examples in the prompt, it performed new tasks with no weight updates. Scaling laws (see Scaling Laws and Compute) explained why bigger, longer-trained models kept improving.

Stage 6: assistants

A base model continues text; it does not obey. Instruction tuning and reinforcement learning from human feedback (RLHF) taught models to follow requests, refuse harmful ones, and converse. This step, covered in the fine-tuning chapter, is what turned raw next-token predictors into assistants.

Stage 7: the current wave

Recent years added open-weight models that anyone can download, mixture-of-experts layers that grow parameter count without growing cost per token, context windows reaching hundreds of thousands of tokens, multimodal models that read images and audio, tool use, and models trained to produce long reasoning before answering. The Transformer core is recognizably the 2017 design, polished by many small improvements.

Four big shifts

Counts to vectors Share strength between similar words.
Sequential to parallel The Transformer trains on GPUs efficiently.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What limitation of RNNs did the Transformer remove?

    Attention connects any two tokens directly and processes all positions in parallel.
  2. 2. Which pairing is correct?

    GPT-style models are decoders trained on next-token prediction. BERT is an encoder trained on masked-word prediction.
  3. 3. What was notable about GPT-3's in-context learning?

    In-context learning means the behavior changes with the prompt while the weights stay fixed.
  4. 4. What is the cost of letting every token attend to every other token?

    n tokens create n times n pairs, so long contexts get expensive.

Summary

  • Language modeling progressed from n-grams to word vectors, RNNs, and then the Transformer.
  • RNNs process tokens sequentially and struggle with long-range information; attention gives direct, parallel connections at a quadratic cost.
  • BERT (encoder) and GPT (decoder) split the design space; decoders dominate generation.
  • Scale produced in-context learning; instruction tuning and RLHF produced assistants.
  • Modern models refine the same core with open weights, mixture-of-experts, long context and multimodality.

Next: LLM Families and Model Sizes