Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 37 ยท Tokens and Embeddings

Token Embeddings

Understand how an embedding table turns token ids into vectors, how cosine similarity measures meaning, and how vectors are learned from context, with NumPy demos.

A token id is just an integer, and the number 7,421 is not "closer" to 7,422 in any useful sense. To compute with tokens, the model first replaces each id with a vector of real numbers, called its embedding. These vectors are learned during training so that tokens used in similar ways end up with similar vectors. Embeddings are the model's first layer and the foundation for everything that follows.

The embedding table

The embedding layer is a matrix E of shape V ร— d: one row for every token in the vocabulary (V rows) and d numbers per row (the model width, such as 768 or 4096). Looking up token i simply returns row i. Mathematically this equals multiplying a one-hot vector by E, but implementations use direct indexing.

Python An embedding lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a real model the table is large: a 50,000-token vocabulary at width 4096 is over 200 million parameters before a single attention layer.

Similarity of vectors

Once tokens are vectors, relationships can be measured geometrically. The usual measure is cosine similarity: the cosine of the angle between two vectors. It is 1 for the same direction, 0 for perpendicular, and -1 for opposite, and it ignores vector length.

Python Cosine similarity with hand-made vectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real embeddings have hundreds or thousands of dimensions and no human-readable axes, but the principle holds: nearby vectors mean tokens that behave alike.

Where the vectors come from

Embeddings are not designed; they are learned. The table starts random and is updated by gradient descent together with the rest of the network, because predicting the next token well requires representing what each token means and how it can be used. The earlier word2vec method made this very visible: words appearing in similar contexts get similar vectors. You can reproduce the intuition with plain co-occurrence counts and a low-rank factorization.

Python Vectors from co-occurrence (SVD)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Words that occupy the same slots (king/queen, cat/dog) get similar vectors. A neural network learns the same idea but end to end for the actual prediction task.

Static versus contextual embeddings

The table gives every token one fixed vector wherever it appears. The word bank has the same input vector in "river bank" and "bank account". The model fixes this through its layers: after each attention layer a token's vector is updated using its neighbors, so by the middle of the network the vector for bank encodes which meaning is intended. These layer outputs are called contextual embeddings, and they are what embedding models for search use.

Weight tying and scale

Many models reuse the embedding matrix at the output: the final hidden vector is multiplied by Eแต€ to produce a score for every token. This weight tying removes a large matrix and works because input and output representations of a token should be related. Implementations often also scale embeddings (for example by sqrt(d)) before adding position information, which you will meet in Positional Encodings.

Python Tied output projection

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Embeddings in three steps

Id An integer from the tokenizer.
Lookup One row of a learned matrix.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the shape of an embedding table for vocabulary V and width d?

    There is one d-dimensional row per token in the vocabulary.
  2. 2. What does cosine similarity measure?

    It is the cosine of the angle: 1 for the same direction, 0 for orthogonal, -1 for opposite.
  3. 3. How do embedding vectors become meaningful?

    The table starts random and is trained with the rest of the network.
  4. 4. Why is the input embedding of 'bank' the same in two different sentences?

    The table is a fixed lookup. Contextual meaning is built by later layers.

Summary

  • An embedding table has one learned d-dimensional vector per token; lookup equals multiplying a one-hot vector.
  • Cosine similarity compares directions; similar usage yields similar vectors.
  • Embeddings are learned jointly with the model; co-occurrence factorization shows the intuition.
  • Input embeddings are static; attention layers produce contextual embeddings.
  • Weight tying reuses the table to score output tokens.

Next: Positional Encodings