Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท Tokens and Embeddings

Positional Encodings

Why attention needs position information, and how learned, sinusoidal, rotary (RoPE) and ALiBi encodings work, with runnable NumPy demos you can plot.

Attention treats its input as a set: if you shuffle the tokens, every token still gets the same set of other tokens to look at, so without extra information the model cannot tell "dog bites man" from "man bites dog". Position information must be injected. The way a model does it affects how well it handles long inputs, so it is a major design choice.

The problem, shown

Here is a one-line demonstration that plain attention without positions is permutation-equivariant: reorder the inputs and the outputs are reordered identically, nothing more.

Python Attention cannot see order by itself

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is True: shuffling changes nothing except the order of the rows. A Transformer therefore needs positions in the data.

Option 1: learned absolute positions

The simplest fix is a second embedding table with one row per position (0, 1, 2, ... up to a maximum). The position vector is added to the token embedding. It is easy and works well, but the model only knows positions it saw in training, and it cannot be used beyond the maximum length without retraining.

Option 2: sinusoidal encodings

The original Transformer used a fixed formula. Each position gets a vector of sines and cosines at geometrically spaced wavelengths: fast-changing dimensions encode fine position, slow-changing ones encode coarse position, like the digits of a clock.

PE(pos, 2i) = sin(pos / 10000^(2i/d)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Python Sinusoidal encodings as a picture

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Nearby positions have similar vectors and distant ones differ. Because the formula works for any position, sinusoidal encodings can be computed for lengths never seen in training, although models do not always use them well there.

Option 3: rotary position embeddings (RoPE)

Most modern open models use RoPE. Instead of adding a position vector, RoPE rotates the query and key vectors by an angle proportional to the position. Treat each pair of dimensions as a 2D point and rotate it by pos * ฮธ, with a different ฮธ per pair. The remarkable property is that the dot product between a rotated query at position m and a rotated key at position n depends only on the relative distance m - n, not on the absolute positions. Relative distance is what language mostly cares about.

Python RoPE: dot products depend only on distance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three rows with distance 3 print the same score even though the absolute positions differ (5/2, 15/12, 105/102), which is exactly the property the table claims. RoPE adds no parameters and is applied only to queries and keys, not values.

Option 4: ALiBi

ALiBi skips vectors entirely. It subtracts a penalty from each attention score proportional to the distance between query and key, with a different slope for each head. Far tokens are therefore less likely to be attended to, and the trick extrapolates to longer sequences than were seen in training.

Python ALiBi bias matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Long contexts and position scaling

A model trained on 4,096 positions may fail at 32,000 because it never saw those position values. Techniques such as position interpolation squeeze the new, longer range into the trained range, and variants of RoPE scaling (adjusting the base frequency) let a model be extended with a short period of extra training. Context length is therefore tied to position design, which is why the maximum context is listed on a model card.

Ways to give order to attention

Learned A table of position vectors, added to embeddings.
Sinusoidal Fixed sines and cosines, any length.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a Transformer need positional information?

    Without positions, attention is permutation-equivariant and cannot tell word order.
  2. 2. What property makes RoPE useful?

    Rotating queries and keys by position-proportional angles makes their dot product depend on the difference of positions.
  3. 3. How does ALiBi encode position?

    ALiBi adds a linear bias that grows with the query-key distance.
  4. 4. A model trained with 4,096 positions fails at 32,000. What is a typical remedy?

    Position interpolation and RoPE scaling let a model handle longer contexts with a little extra training.

Summary

  • Attention treats tokens as a set, so position must be injected.
  • Learned tables are simple but bounded; sinusoidal encodings are fixed formulas for any length.
  • RoPE rotates queries and keys so attention depends on relative distance; it is widely used.
  • ALiBi penalizes attention scores by distance.
  • Context-length extension relies on scaling or interpolating positions.

Next: Self-Attention Step by Step