Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 38 ยท Hub, Models and Tokenizers

The Model Hub: Repositories, Model Cards and Versions

Learn how the Hugging Face Hub works: Git-based repositories, model cards, files, revisions, searching and downloading models safely with from_pretrained.

The Hub is where Hugging Face models live. Each model is a Git repository with large-file support, so it has branches, commits and tags like any code repository. A repository is named owner/name, for example google-bert/bert-base-uncased, and older famous models also have short names without an owner.

Understanding what is in a repository explains most of what from_pretrained does: it downloads a few specific files from the right revision and builds objects from them.

What a model repository contains

File Purpose
config.json architecture settings: layers, hidden size, label names
model.safetensors the weights, in the safe, fast safetensors format
tokenizer.json, tokenizer_config.json, vocab files everything the tokenizer needs
generation_config.json default settings for text generation
README.md the model card

Prefer safetensors over old pickle-based .bin checkpoints. Pickle files can execute arbitrary code when loaded, while safetensors files contain only numbers. Never load pickle weights from a source you do not trust.

The model card

The README.md begins with a YAML block (metadata) and continues with free text. The metadata powers the Hub's search and filters: licence, languages, task (pipeline_tag), base model and datasets. A good card also explains intended use, training data, limits, biases and evaluation results. Read it before you rely on a model.

The code below is a stand-in: a small parser for a model card's metadata block using only standard Python. Real cards use full YAML; this one handles simple key: value and list entries.

Python Stand-in: parse model card metadata

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Licence checking is the most important use of metadata. Some model licences forbid commercial use or require attribution.

Downloading with from_pretrained

Real code, read-only:

Python
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
model = AutoModel.from_pretrained("google-bert/bert-base-uncased")

The first call fetches config.json, the tokenizer files and the weights into the cache; the next calls read from disk. You can also pass a local folder path instead of a repository id, which is how you load a model you saved yourself.

Revisions

A repository has branches and commits. main moves whenever the owner pushes. To make a deployment reproducible, pin a commit hash or a tag:

Python
model = AutoModel.from_pretrained(
    "google-bert/bert-base-uncased",
    revision="86b5e0934494bd15c9632b12f734a8a67f723594",
)

Without a pinned revision, an update to main could change your results overnight.

Python Stand-in: why pinning matters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Finding models

On the website you can filter by task, library, language, licence and size, and sort by downloads or likes. In code, huggingface_hub lets you search:

Python
from huggingface_hub import list_models

for m in list_models(filter="text-classification", sort="downloads", limit=5):
    print(m.id)

Signals of quality: many downloads, recent updates, a detailed card, evaluation results, and a trustworthy organisation. Treat unknown repositories with caution, especially those that require trust_remote_code=True, which runs Python from the repository on your machine.

Uploading your own model

After fine-tuning, you can share a model (read-only code; it needs a write token):

Python
model.push_to_hub("my-name/my-sentiment-model")
tokenizer.push_to_hub("my-name/my-sentiment-model")

Repositories can be public or private. Add a model card describing data, limits and the licence of the base model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What kind of thing is a Hub model repository?

    Repositories have commits, branches and tags, which is why revisions can be pinned.
  2. 2. Why prefer safetensors files over pickle-based .bin files?

    Pickle can run code on load. Safetensors holds only tensor data.
  3. 3. How do you make a deployment reproducible when loading from the Hub?

    A pinned revision always returns the same files even if main changes.
  4. 4. Where do you find a model's licence and intended use?

    The card's metadata and text describe licence, data, limits and use.

Summary

  • A Hub model is a Git repository holding config, weights, tokenizer files and a model card.
  • from_pretrained downloads those files into the cache and builds objects from them.
  • Pin revision for reproducibility; prefer safetensors; be careful with trust_remote_code.
  • Read the model card for licence, data and limits before using a model.

Next: Tokenization Basics