Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 37 Ā· Getting Started

LLM Families and Model Sizes

Compare encoder, decoder and encoder-decoder models, dense and mixture-of-experts designs, open and closed weights, and estimate memory from parameter count.

"LLM" covers several different kinds of model. They differ in architecture (which tokens each position may look at), in training objective, in how they are released, and in size. Knowing the vocabulary lets you read a model card and quickly judge what a model is for and whether you can run it.

Three Transformer shapes

All three use the same building blocks, but the attention mask and objective differ.

Family Attention Trained to Typical use
Encoder-only Every token sees every token (bidirectional) Fill in masked tokens Classification, search embeddings, tagging
Decoder-only Each token sees only earlier tokens (causal) Predict the next token Text generation, chat assistants, code
Encoder-decoder Encoder bidirectional; decoder causal and also attends to the encoder Turn an input sequence into an output sequence Translation, summarization

Most headline LLMs today are decoder-only. The causal mask lets one training pass score every position at once, and the same model can both read a prompt and continue it. The mask itself is explained in Multi-Head Attention and Masking.

Python Attention masks for the three shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Parameters: what the number means

A parameter is one learned number (a weight or bias). Model names often carry the count: "7B" means about seven billion. Parameters live mostly in matrices: the embedding table, the attention projections, and the feed-forward layers. For a standard decoder with model width d, L layers, and vocabulary V, a good approximation is:

parameters ā‰ˆ V * d + 12 * L * d²

The 12 * d² per layer comes from four attention matrices (4d²) plus a feed-forward block that expands to 4d and projects back (8d²). The Decoder-Only Architecture End to End derives it exactly.

Python Estimate parameters of well-known shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimates land close to the published sizes (about 124 million for the small GPT-2 shape and 7 billion for the 4096-wide, 32-layer shape). The small differences come from details such as gated feed-forward layers and tied embeddings.

Memory: the practical constraint

To run a model you must keep its weights in memory. Memory in bytes is the parameter count times the bytes per parameter, which depends on the numeric precision.

Python Memory needed just for the weights

Press Run (or Ctrl + Enter) to see the result here.

Program input Ā· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 7B model in 16-bit needs about 14 GB just for weights, which is why quantization to 8 or 4 bits (covered in the inference chapter) makes models fit on consumer hardware.

Dense versus mixture-of-experts

A dense model uses all its parameters for every token. A mixture-of-experts (MoE) model has many parallel feed-forward "experts" per layer and a small router that sends each token to only a few of them. The model can hold a very large total number of parameters while using only a fraction (the active parameters) for each token, so compute per token stays modest. The catch is that all experts must still be stored in memory.

Base, instruct, and reasoning variants

The same architecture is released in several flavors:

  • Base (pretrained) models continue text. They are the raw product of next-token training.
  • Instruct or chat models are further tuned to follow instructions and hold conversations.
  • Reasoning models are trained to write long intermediate reasoning before the final answer.
  • Embedding models (often encoder-only) output vectors for search rather than text.

Open weights versus closed

Open-weight Closed (API only)
You can download weights Yes No
Run offline, fine-tune, inspect Yes No
Pay for Your own hardware Per token
Typical trade More control, you own operations Strong quality with no infrastructure

"Open weights" is not always "open source": licenses differ on commercial use, and training data and code are often not released. Always read the license.

Reading a model name

Size 7B, 70B: parameter count.
Variant Base, instruct, reasoning.
1 / 2

Check your understanding

Quick check Ā· 4 questions

of 4 answered You scored / 4 Ā· perfect!

  1. 1. Which family is trained to predict the next token and is used by most chat assistants?

    Decoder-only models with a causal mask predict the next token and generate text left to right.
  2. 2. Roughly how much memory do the weights of a 7B model take in 16-bit precision?

    7 billion parameters times 2 bytes is about 14 GB.
  3. 3. What is an advantage of mixture-of-experts?

    A router activates a few experts per token. All experts must still be stored.
  4. 4. Which is true about open-weight models?

    Open weights means downloadable weights. Licenses and data availability vary.

Summary

  • Encoder-only models read bidirectionally, decoder-only models predict the next token, encoder-decoder models map inputs to outputs.
  • Parameters are learned numbers; a decoder has roughly VĀ·d + 12Ā·LĀ·d² of them.
  • Weight memory equals parameters times bytes per parameter, so precision matters.
  • Mixture-of-experts adds capacity without proportional compute but still needs all experts in memory.
  • Base, instruct, reasoning and embedding variants share architecture but differ in training; check the license before using open weights.

Next: Tokenization Basics