Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 38 ยท Getting Started

Model Families and Tasks

Map tasks to model families: BERT-style encoders, GPT-style decoders, T5 and BART, vision transformers and Whisper, and learn how to choose the right size.

The Hub lists hundreds of thousands of models, but they fall into a handful of families. Once you recognise the family from a name, you know what the model is good at, which tokenizer it uses and which AutoModel class to load. This lesson gives you that map.

Task first, model second

Always start from the task, not the model name. Ask what goes in and what comes out.

You have You want Task Typical family
a sentence a label text classification BERT, RoBERTa, DeBERTa
a sentence a label for every word token classification (NER) BERT-style encoders
a question and a passage an answer span extractive question answering encoders
a long text a short text summarization BART, T5, large decoders
a prompt more text text generation GPT-style decoders
a sentence a vector embeddings / search sentence encoders
an image a label image classification ViT, ConvNeXt
audio text speech recognition Whisper, wav2vec 2.0
image and text text captioning, visual QA multimodal models

The main families

Encoders (BERT family). BERT, RoBERTa, DistilBERT, ALBERT, DeBERTa, ModernBERT. They read the whole input in both directions, are small (tens to hundreds of millions of parameters) and are excellent when you need a decision or a vector rather than free text. DistilBERT is a smaller, faster student trained to imitate BERT.

Decoders (GPT family). GPT-2, Llama, Mistral, Qwen, Gemma, Phi. They continue a prompt one token at a time. With instruction tuning they follow requests and chat. Sizes range from half a billion to hundreds of billions of parameters.

Encoder-decoders. T5, FLAN-T5, BART, mBART, Marian. They map one text to another, so they are natural for translation and summarization. T5 treats every task as text to text with a prefix such as summarize:.

Vision. ViT (Vision Transformer) cuts an image into patches and treats them as tokens. DETR does object detection, SegFormer segmentation, CLIP links images and text.

Audio. Whisper (encoder-decoder) transcribes and translates speech. wav2vec 2.0 and HuBERT learn speech representations.

Reading a model name

Names carry information. Take distilbert-base-uncased-finetuned-sst-2-english:

  • distilbert: the architecture family.
  • base: the size.
  • uncased: the tokenizer lowercases text.
  • finetuned-sst-2: further trained on the SST-2 sentiment dataset.
  • english: the language.

A name such as meta-llama/Llama-3.1-8B-Instruct reads: organisation, family and version, 8B (eight billion parameters), Instruct (tuned to follow instructions).

Python Stand-in: decode a model name

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a stand-in heuristic: real names do not follow a strict rule, so always read the model card.

Choosing a size

Bigger models are usually more accurate and always slower and more expensive. A rough memory estimate for loading weights is parameters times bytes per number: 2 bytes in 16-bit precision, 4 bytes in 32-bit. Training needs several times more because of gradients and optimizer state.

Python Memory needed to load a model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 7-billion-parameter model needs roughly 13 GiB in 16-bit but only about 3.3 GiB in 4-bit. That is the whole motivation for the quantization techniques covered later in the course.

flowchart TD
  A["Task"] --> B{"Output type"}
  B -->|"label or vector"| C["Small encoder"]
  B -->|"new text from text"| D["Encoder-decoder or decoder"]
  B -->|"open-ended chat"| E["Decoder, instruct version"]
  B -->|"image or audio"| F["ViT or Whisper style"]

Choose the smallest family that fits the output type, then scale up only if quality is too low.

Practical advice

  1. Start with a small, well-downloaded model for your task and measure it.
  2. Check the licence on the model card before commercial use.
  3. Prefer models with recent updates and clear documentation.
  4. For a fixed label set with training data, a fine-tuned small encoder often beats a prompted giant model on both cost and accuracy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which family is the most natural choice for assigning a label to every word in a sentence?

    Encoders see the whole sentence and give a vector per token, which a token-classification head labels.
  2. 2. What does the 7B in a model name usually mean?

    B stands for billions of parameters.
  3. 3. Roughly how many GiB do the weights of a 7B model take in 16-bit precision?

    7e9 parameters times 2 bytes is 14e9 bytes, about 13 GiB.
  4. 4. T5 treats every task as:

    T5 uses a text prefix to say which task to perform and always produces text.

Summary

  • Pick the task first, then the family: encoders for labels and vectors, decoders for generation, encoder-decoders for text transformation.
  • Vision and audio use the same ideas with patches and spectrogram frames.
  • Names reveal architecture, size, tuning and language; model cards give the facts.
  • Memory is roughly parameters times bytes per parameter; lower precision shrinks it.

Next: The Model Hub