Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 73 ยท Sequences, Attention and Transformers

Working With Sequences

Prepare sequence data for neural networks: windowing time series, tokenizing text, one-hot and index encoding, padding and masking, and why order matters.

Images have a fixed size and a grid structure, so a convolutional network can treat every picture the same way. Many of the most useful kinds of data are different: a sentence, a stock price over a year, the notes of a melody, a sensor reading every second, a DNA strand. Each is a sequence, an ordered list whose length varies and whose meaning depends on the order of the items.

Before you can build recurrent networks or transformers, you need to know how to turn raw sequences into arrays a network can eat. This lesson covers the preprocessing that all sequence models share: cutting a time series into windows, turning text into integer tokens, one-hot and index encoding, padding variable lengths to a rectangle and masking the padding, and a small demonstration of what is lost when order is ignored.

Sequences are everywhere

Time series Prices, sensors, weather. One value per time step.
Text Words or characters in a particular order.
Audio and music Samples or notes that unfold over time.
Biology DNA and protein strings.
1 / 2

Time series: windows

A network needs fixed-size inputs. A common way to turn one long series into training examples is the sliding window: take T consecutive values as the input and the next value (the target) as the output. Slide the window one step at a time to get many examples from a single series.

Two rules matter:

  • Split chronologically. Train on the earlier part and test on the later part. A random shuffle would put windows from the future into the training set, so the test would look easier than real forecasting.
  • Compare with a naive baseline. The simplest forecast, "tomorrow equals today", is surprisingly hard to beat. A model that cannot beat it has learned nothing useful.

sliding_window_view from NumPy builds all windows without copying data (see views and copies).

Python Windowing a time series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A series of 300 points gives 290 windows of length 10. The naive forecast has a small error because consecutive values of a smooth series are close, while the constant mean is much worse. These two numbers are the bar that recurrent networks and later models will have to clear.

Text: tokens and a vocabulary

A network can only process numbers, so text goes through three steps:

  1. Tokenize: split the text into units called tokens. Tokens can be characters, words, or sub-words (pieces like "un", "break", "able"; you will meet the sub-word method BPE in How Large Language Models Work).
  2. Build a vocabulary: assign every distinct token an integer id. Reserve ids for special tokens: <pad> (filler), <unk> (any word not in the vocabulary), and often <bos>/<eos> for the beginning and end of a sentence.
  3. Encode: replace each token with its id.

Here is a word-level tokenizer in a few lines. Note that the vocabulary is built from the training text only; new words at test time become <unk>.

Python A tiny word tokenizer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Word-level tokens give short sequences but a large vocabulary and trouble with unseen words. Character-level tokens give a tiny vocabulary and no unknown symbols, but sequences are several times longer. Sub-word tokenizers sit in between, which is why modern language models use them.

One-hot versus index encoding

An integer id is only a label. The id 5 is not "bigger than" or "close to" the id 4, so feeding raw ids to a neuron would be meaningless. There are two standard fixes:

  • One-hot vectors. Each token becomes a vector of length V (the vocabulary size) with a single 1 at its id. A sequence of T tokens becomes a (T, V) matrix. This is simple and explicit, but wasteful when V is large.
  • Index lookup (embeddings). Keep the integer ids and let the first layer look up a learned dense row for each id. Multiplying a one-hot vector by a weight matrix W of shape (V, D) just picks row id of W, so lookup computes the same thing without the huge intermediate array. Embeddings covers this in depth.
Python One-hot versus index lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a real vocabulary of 50,000 tokens, a 1,000-token passage as one-hot would hold 50 million numbers, against 1,000 integers for the lookup. That is why embedding layers take ids directly.

Padding and masking

Sentences differ in length, but a batch must be a rectangular array. The solution is padding: append the <pad> id to shorter sequences until all have the length of the longest one (or a fixed maximum, cutting longer ones, which is called truncation).

Padding creates a new problem: the filler tokens are not data, and must not influence the result. A mask is a boolean array of the same shape, True where the position is a real token. Anything that averages, sums, takes a max or computes a loss over a sequence has to use the mask. Forgetting to do so is a classic source of silent bugs: for example, the "average" of a short sentence silently becomes smaller because of the zeros mixed in.

Python Padding and a masked average

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The masked averages match the true ones exactly, while the naive averages are too small for every sequence that was padded. Later lessons use the same idea in more places: an RNN stops at each sequence's true length, and attention uses a mask to avoid looking at padding positions (and, in language models, at future tokens).

Why order matters

A tempting shortcut is to ignore order: count how often each word appears (a bag of words) and feed the counts into an ordinary network. It works for easy tasks such as spotting spam, but it throws away information. The sentences "the dog bit the man" and "the man bit the dog" use exactly the same words and mean different things. For time series the situation is similar: shuffling a series destroys the very relationship between neighbors that a forecast depends on.

Python What a bag of words forgets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two bags are identical, but the two one-hot sequences differ, because the sequence form keeps the position of each word. And the neighbors in the real series are strongly correlated (about 0.9), while after shuffling the correlation drops to about zero: the "memory" in the data is gone. Sequence models are built to read data in order, carrying information from earlier steps to later ones, and that is the subject of the next lessons.

Sequence batches usually have the shape (N, T, D): examples, time steps, features per step. Some libraries put time first, (T, N, D), so check the convention before you reshape. Whatever the order, T is the axis that models move along.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should a time series be split chronologically into train and test sets, instead of shuffling the windows?

    A forecasting model must predict data that comes after what it has seen. Mixing future windows into training leaks information and inflates the test score.
  2. 2. Which token is used for words that were not in the vocabulary when it was built?

    <unk> stands for unknown. <pad> is filler for equal lengths, and <eos> marks the end of a sequence.
  3. 3. A batch of sentences is padded with zeros to the same length. What goes wrong if you average over time without a mask?

    The zeros are included in both the sum and the count, so shorter sequences get a smaller average than their real tokens would give.
  4. 4. What is the main reason for using an embedding lookup instead of one-hot vectors with a large vocabulary?

    A one-hot vector times W just selects one row of W. Indexing does it directly and saves memory and compute.

Summary

  • Sequences have variable length and meaningful order: text, time series, audio, DNA and more.
  • Time series become training examples with sliding windows; split chronologically and compare against a naive "last value" baseline.
  • Text is tokenized (characters, words or sub-words), mapped to integer ids through a vocabulary with special tokens such as <pad> and <unk>, and the vocabulary is built from training data only.
  • Integer ids are labels, not quantities: use one-hot vectors or, better, embedding lookups.
  • Pad sequences into a rectangular batch of shape (N, T) and use a mask so padding never affects averages, losses or attention.
  • Ignoring order (a bag of words, a shuffled series) discards information that sequence models are designed to use.

Next: Recurrent Neural Networks.