Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 63 ยท Neural NLP and Transformers

Pre-trained Language Models: BERT, GPT and Beyond

Learn how pretraining and fine-tuning work, how encoder models like BERT differ from decoder models like GPT, and what model size means for memory and cost.

Before 2018, most NLP projects trained a model from scratch on a labeled dataset, often with only a few thousand examples. Every new task meant starting over, and the model knew nothing about language except what those few examples showed. The big change since then is transfer learning: first train a huge Transformer on enormous amounts of plain text, then reuse what it learned for your own task.

That is the idea behind BERT, GPT, T5 and every model in the same family. In this lesson you will learn the two stages (pretraining and fine-tuning), the three architecture shapes, what the training objectives look like, and how to estimate the size, memory and cost of a model. The Hugging Face code is shown read-only, because the models are far too big to download in a browser tab, but the numbers and objectives are runnable.

Pretraining and fine-tuning

Two stages

Pretraining Learn language from huge unlabeled text.
Fine-tuning Adapt to one task with a small labeled set.

Pretraining is expensive and done once; fine-tuning is cheap and done often.

1 / 2

Pretraining uses a self-supervised objective: the labels come from the text itself, so no human annotation is needed. The model sees trillions of words from books, web pages and code, and it must solve a prediction puzzle over and over. To solve it well, it has to learn grammar, facts, some reasoning patterns and the meaning of words in context.

Fine-tuning takes the pretrained weights and trains them a little more on a labeled dataset for your task, such as spam detection or entity recognition. Usually a small new output layer (a head) is added on top, and either only the head or all the weights are updated with a small learning rate. Compare that with the classifiers you built earlier: those started from a blank slate, so they needed to learn the language and the task together.

Three architecture shapes

The Transformer has an encoder part and a decoder part, and different models keep different pieces.

  • Encoder-only (BERT, RoBERTa): every token sees the whole sentence, left and right. Great for understanding tasks: classification, entity recognition, semantic search embeddings. It does not generate text naturally.
  • Decoder-only (GPT family, Llama, most chat models): each token sees only the tokens to its left (the causal mask from the previous lesson). It generates text one token at a time. With enough scale and good prompting, it handles nearly every task as text in, text out.
  • Encoder-decoder (T5, BART): an encoder reads the input, and a decoder writes the output while attending to the encoder. A natural fit for translation and summarization, where input and output are different texts.

Two training objectives

The architectures differ because their objectives differ. An encoder is trained with masked language modeling: hide about 15% of the tokens and ask the model to fill them in using context on both sides. A decoder is trained with next-token prediction: given the tokens so far, predict the next one. The code builds training examples for both from one sentence.

Python Masked and next-token training examples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The masked version gives two training signals from one sentence, and to guess tasted the model can use words on both sides (chef before it, soup after it). The next-token version produces eight training pairs from the same sentence, and each prediction only sees the left context. The decoder's objective is exactly what you do at generation time, which is why decoder models are the ones that write text. This is also the same idea as the n-gram language models from earlier, with a neural network that can use thousands of tokens of context instead of two.

Real models predict subword tokens, not whole words, using vocabularies such as the ones from BPE and WordPiece. Our example splits on spaces only to keep the output readable.

Using a pretrained model

In practice you rarely train a base model yourself. The Hugging Face transformers library downloads weights and gives you a few lines of code for common jobs. The snippets below are read-only: they need the library, a download of hundreds of megabytes and usually a GPU for fine-tuning.

Python
from transformers import pipeline

# Zero setup: a ready-made model for one task
classifier = pipeline("sentiment-analysis")
print(classifier("The update fixed every bug I reported."))

# Encoder model asked to fill in a masked word
fill = pipeline("fill-mask", model="bert-base-uncased")
print(fill("The chef added more [MASK] to the soup."))

For your own labeled data you fine-tune. The skeleton looks like this:

Python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments

name = "distilbert-base-uncased"            # a small encoder model
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)

def encode(batch):
    return tokenizer(batch["text"], truncation=True, padding="max_length", max_length=128)

# train_ds and eval_ds are labeled datasets with "text" and "label" columns
args = TrainingArguments(output_dir="out", num_train_epochs=3, learning_rate=2e-5,
                         per_device_train_batch_size=16)
trainer = Trainer(model=model, args=args,
                  train_dataset=train_ds.map(encode, batched=True),
                  eval_dataset=eval_ds.map(encode, batched=True))
trainer.train()

The tokenizer must match the model: a model pretrained with one vocabulary is meaningless with another. The small learning rate (about 2e-5) is typical for fine-tuning, because large steps would erase what the model learned in pretraining.

Model size, memory and cost

Model size is counted in parameters (weights). For a standard Transformer, each layer holds roughly 12 ร— dยฒ parameters (4 dยฒ in the attention projections and 8 dยฒ in the feed-forward network, where d is the model width), plus a table of vocabulary ร— d token embeddings. Memory follows directly: 2 bytes per parameter in 16-bit precision, 1 byte at 8-bit, half a byte at 4-bit. Training cost is commonly estimated at about 6 floating-point operations per parameter per training token. The next program applies these rules of thumb.

Python Estimating parameters, memory and training cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimates land close to the published sizes: about 0.11 billion parameters for a BERT-base-like model, 1.55 billion for GPT-2 XL and about 175 billion for the GPT-3 shape (174.56 here). The memory columns explain a lot of practical decisions. A 175-billion-parameter model needs roughly 349 GB just for its weights in 16-bit precision, so it cannot fit on one GPU, while a model with 1.5 billion parameters fits in about 3 GB. Quantization (storing weights at 8 or 4 bits) shrinks memory by two to four times at a small cost in quality, which is why many models run on a laptop. The last line shows why pretraining is a big-lab activity: even a small 124M model on 10 billion tokens needs around 7.4e18 operations, which is dozens of GPU-hours under optimistic assumptions, and the largest models cost tens of thousands of times more. Fine-tuning, by contrast, touches far fewer tokens and often finishes in minutes to hours.

Choosing and adapting models

A practical decision guide:

  • Small labeled classification task, no GPU? A TF-IDF with logistic regression baseline is often within a few points of a fine-tuned model and costs almost nothing. Always build it first.
  • Need better accuracy on understanding tasks? Fine-tune a small encoder such as DistilBERT; models with 100 million parameters are fast on a single GPU.
  • Need generation, flexible tasks or no training data at all? Use a decoder model with prompting, covered in Large Language Models and Prompting.
  • Fine-tuning a huge model is too heavy? Parameter-efficient methods such as LoRA train a tiny number of extra weights while the base stays frozen.

Remember the limits too. Pretrained models absorb biases from their training text, may reflect outdated facts, and need a license check before commercial use; see Evaluation, Bias and Responsible NLP.

BERT in detail

BERT (Bidirectional Encoder Representations from Transformers, 2018) is the model that made pretraining standard in NLP, so it is worth knowing its parts.

Sizes. BERT comes in two classic sizes. BERT-base has 12 Transformer layers, hidden size 768 and 12 attention heads, about 110 million parameters. BERT-large has 24 layers, hidden size 1024 and 16 heads, about 340 million. Both use a 30,000-token WordPiece vocabulary (see Subword Tokenization) and a maximum input of 512 tokens.

Input format. BERT reads one or two segments: [CLS] segment A [SEP] segment B [SEP]. The [CLS] vector at the front is used for sentence-level tasks, [SEP] marks boundaries, and a segment embedding tells the model which part a token belongs to. Each token's input is the sum of its token, segment and position embeddings.

Combined pretraining. BERT was pretrained with two objectives at once. Masked language modeling hides 15% of the tokens; of the chosen tokens, 80% are replaced by [MASK], 10% by a random token and 10% stay unchanged, so the model cannot rely on [MASK] being a hint that exists only in training. Next sentence prediction asks whether segment B really followed segment A in the source text, to teach relations between sentences. Later work (RoBERTa) found that next sentence prediction adds little and dropped it.

Word meaning in context. The vector for a token depends on the whole sentence, so bank in "river bank" and in "bank loan" gets different vectors. Static embeddings such as word2vec cannot do this (see Word Embeddings). Using these contextual vectors is the basis of Sentence Embeddings.

Python BERT-base versus BERT-large: parameters and attention cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The parameter count of 109.5 million for BERT-base and 335.1 million for BERT-large matches the published sizes. In fp16 the whole base model needs only about 0.22 GB, which is why it fits on a laptop, while the attention-score count (every token against every token) grows with the square of the length, from about 16 thousand at 128 tokens to over 4 million at 2048. The last line confirms that the masking recipe gives roughly 80/10/10.

Limitations of BERT

  • High computational requirements: pretraining BERT-large took days on dozens of accelerators, and inference is slower than a classic model such as logistic regression on TF-IDF features.
  • Fixed length: inputs beyond 512 tokens must be cut into chunks, which loses long-range context.
  • Not a text generator: it is an encoder, so it fills in and classifies but does not write long text (decoder models, described earlier in this lesson, do that).
  • Pretraining mismatch: [MASK] never appears when you fine-tune or run the model, a gap the 80/10/10 recipe only reduces.
  • Inherited bias: it learns the associations present in its training text.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main benefit of pretraining followed by fine-tuning?

    Pretraining on huge unlabeled text teaches grammar, facts and context. Fine-tuning then adapts that knowledge with a small labeled dataset.
  2. 2. Which objective is used to train BERT-style encoder models?

    Masked language modeling hides some tokens and asks the model to recover them, so every token can look at both its left and right neighbors.
  3. 3. Which model shape is best suited to translation and summarization, where the output is a different text from the input?

    An encoder-decoder model reads the input with an encoder and writes the output with a decoder that attends to it, as in T5 and BART.
  4. 4. Roughly how much memory do the weights of a 7-billion-parameter model need at 16-bit precision?

    Each parameter takes 2 bytes at 16 bits, so 7 billion parameters need about 14 GB. At 4 bits it would be roughly 3.5 GB.
  5. 5. Why is a small learning rate such as 2e-5 typical when fine-tuning?

    Large updates would quickly destroy the useful pretrained weights. Small steps adjust them gently toward the new task.

Summary

  • Pretraining learns language from huge unlabeled text; fine-tuning adapts the result to a task with few labels.
  • Encoder models (BERT) see both directions and suit understanding; decoder models (GPT) predict the next token and suit generation; encoder-decoder models (T5) map one text to another.
  • Training objectives are masked-token prediction (encoders) and next-token prediction (decoders).
  • Parameters are about 12 ร— layers ร— dยฒ plus embeddings; memory is 2 bytes per parameter at 16-bit and less when quantized.
  • Always build a simple baseline first, and check bias, license and cost before choosing a large model.

Next, look at how decoder models generate text and how to steer them in Large Language Models and Prompting.