Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 40 ยท Sequences and Text

Text Vectorization and Embeddings

Turn text into numbers for neural networks: tokenization, vocabularies, padding, the Keras TextVectorization and Embedding layers, with plain Python and NumPy demos.

Neural networks only understand numbers, so before a model can read a review or a tweet, the text has to be converted. The process has three steps: split the text into tokens, map each token to an integer using a vocabulary, and turn each integer into a dense vector with an embedding. This lesson walks through all three.

TensorFlow does not run in the browser here. The runnable blocks implement the same ideas in plain Python and NumPy; the Keras equivalents are read-only.

Tokenization and a vocabulary

The simplest tokenizer lowercases the text, strips punctuation and splits on spaces. The vocabulary lists the most frequent tokens and reserves two special entries: index 0 for padding and index 1 for unknown words (out-of-vocabulary, OOV).

Python Build a vocabulary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

soundtrack was never seen in training, so it maps to the unknown token. Capping the vocabulary (say, the 20,000 most common words) keeps models small and sends rare words to [UNK].

Padding to equal length

A batch is a rectangular tensor, but sentences differ in length. Pad shorter ones with zeros (or truncate longer ones) to a fixed length.

Python Pad and truncate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The boolean mask marks real tokens. Keras layers can use it (mask_zero=True in Embedding) so that RNNs and attention ignore padding.

The Keras TextVectorization layer

In Keras all of the above can live inside the model as one layer, so raw strings go in and your deployed model needs no separate preprocessing code.

Python
import keras
from keras import layers

vectorize = layers.TextVectorization(
    max_tokens=20000,
    output_mode="int",
    output_sequence_length=200,
)
vectorize.adapt(train_texts)           # learns the vocabulary
print(vectorize.get_vocabulary()[:10])
print(vectorize(["I loved this movie"]))

adapt() scans your training texts once to build the vocabulary. Other output modes include "multi_hot", "count" and "tf_idf", which produce one fixed-size vector per text (a bag of words) that ignores word order.

Bag of words

Bag of words counts how many times each vocabulary word appears. It loses order ("dog bites man" equals "man bites dog") but works surprisingly well for simple classification.

Python Bag of words by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Embeddings

An integer like 4 has no meaning to a network: 4 is not "twice" 2. An embedding layer is a trainable table with one row per vocabulary word. Looking up the row for a token gives a dense vector (say 64 numbers). During training the rows move so that words used in similar ways get similar vectors.

From text to vectors

Tokenize Split into words
Index Vocabulary ids
Pad Equal length
Embed Dense vectors

Every step except the last has no learned weights.

1 / 2

An embedding lookup is just row selection, which is the same as multiplying a one-hot vector by the table, only much faster.

Python An embedding is a lookup table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Keras layer behaves the same way:

Python
layers.Embedding(input_dim=20000, output_dim=64, mask_zero=True)
# input (batch, 200) integers  ->  output (batch, 200, 64)

Parameters: input_dim * output_dim, here 1.28 million. Embeddings usually hold most of a text model's weights.

A complete text classifier

Python
model = keras.Sequential([
    keras.Input(shape=(1,), dtype="string"),
    vectorize,
    layers.Embedding(20000, 64, mask_zero=True),
    layers.GlobalAveragePooling1D(),
    layers.Dense(32, activation="relu"),
    layers.Dense(1, activation="sigmoid"),
])
model.compile("adam", "binary_crossentropy", metrics=["accuracy"])

GlobalAveragePooling1D averages the word vectors over the sequence, exactly like the sentence vector in the demo (and it respects the mask). Later lessons replace that pooling with recurrent layers or attention, which also use word order.

Pretrained embeddings such as GloVe can initialize the table, and you can freeze or fine-tune them just like a pretrained CNN base.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do we reserve index 0 and 1 in a vocabulary?

    Index 0 is conventionally padding (and the mask) and index 1 stands for out-of-vocabulary words.
  2. 2. What is the output shape of Embedding(1000, 16) applied to integer input of shape (32, 50)?

    Each of the 32*50 token ids is replaced by a vector of length 16.
  3. 3. What does bag of words discard?

    Counts are kept, but the sequence order is lost.
  4. 4. How many parameters does Embedding(20000, 64) have?

    One 64-dimensional row per word: 20000 * 64 = 1,280,000.

Summary

  • Text becomes numbers by tokenizing, indexing with a vocabulary, and padding to equal length.
  • Index 0 is padding and 1 is unknown by convention; TextVectorization.adapt() learns the vocabulary.
  • An embedding is a trainable lookup table that maps ids to dense vectors; output shape is (batch, length, dim).
  • Bag of words ignores order; embeddings plus sequence layers use it.
  • Put TextVectorization inside the model so deployment accepts raw strings.

Next: Recurrent Networks