Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 53 ยท Beyond Text

Audio, Video and Multimodal Models

See how speech-to-text, text-to-speech, vision-language models and video generation work, using spectrograms, image patches and shared embeddings.

Text is only one kind of data. Phones record voices, cameras record scenes, and documents mix words with pictures and tables. Multimodal models work with more than one of these modalities: they can listen to speech, describe a photo, read a chart or generate a clip. The recurring trick is the same everywhere: turn the signal into a sequence of vectors, then let a transformer or a diffusion model work on that sequence.

This lesson walks through audio, vision, video and the shared embedding spaces that tie them together. The big models cannot run in the browser, so the live examples build the inputs those models see: a spectrogram and image patches.

One idea, many modalities

Audio Waveform to spectrogram frames.
Images Pixels to patch vectors.
Video Frames plus time.
Text Tokens, as always.

Everything becomes a sequence of vectors.

1 / 2

Audio: from waves to spectrograms

Sound is a waveform: air pressure sampled thousands of times per second (16,000 times per second is typical for speech). Raw samples are long and hard to model, so most systems compute a spectrogram: slice the signal into short overlapping windows (around 25 milliseconds) and measure how much energy is present at each frequency in each window. The result is an image with time on one axis and frequency on the other. A mel spectrogram spaces the frequencies the way human hearing does, with finer detail at low pitches.

Python A toy spectrogram of a rising tone

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The bright diagonal line is the rising tone, and the horizontal line near 2,500 Hz is the hum. A speech model sees the same kind of picture, with the energy patterns of vowels and consonants.

Speech-to-text (ASR)

Automatic speech recognition turns audio into text. Modern systems use an encoder-decoder transformer: the encoder reads spectrogram frames and the decoder writes text tokens one at a time, like a language model conditioned on the audio. Large multilingual models can also translate and detect the language. Typical issues:

  • Accents, noise and overlapping speakers raise error rates; measure with word error rate (WER), the share of words that are substituted, inserted or deleted.
  • Rare names and jargon are often misspelled; a custom vocabulary or a post-processing step helps.
  • Timestamps and speaker labels need extra components (alignment and diarization).

Text-to-speech (TTS)

Text-to-speech goes the other way. Common designs predict a mel spectrogram from text (or from phonemes) and then use a vocoder, a neural network that converts the spectrogram into a waveform; newer systems generate audio tokens directly with a language-model-style decoder or with diffusion. Quality questions are naturalness, prosody (rhythm and emphasis), latency for streaming, and voice cloning, which raises consent and fraud concerns, so check the rules in Bias, Privacy, Copyright and Responsible Use. Music and sound-effect generators follow similar recipes.

Vision-language models

A vision-language model (VLM) answers questions about images, reads screenshots and documents, and describes scenes. The usual design has three parts:

  1. A vision encoder (typically a Vision Transformer) that splits the image into square patches and turns each into a vector.
  2. A projector, a small network that maps those vectors into the language model's embedding space.
  3. A language model that reads the projected patch vectors together with the text tokens, as if the image were a run of special tokens.
flowchart LR
    I["Image"] --> P["Split into patches"]
    P --> V["Vision encoder"]
    V --> J["Projector"]
    J --> L["Language model"]
    T["Text prompt tokens"] --> L
    L --> O["Text answer"]

Image patches become vectors that sit in the prompt next to the text tokens.

Because each image becomes many tokens, images cost money and context space (see Counting Tokens and Estimating Cost); a larger image or higher detail setting means more patches.

Python Cut an image into patches and embed them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The patches covering the bright square stand out in the grid. A real encoder is trained so that patch vectors capture objects, text and layout rather than raw brightness.

Video

Video is a stack of frames in time, so the cost explodes: a 10-second clip at 24 frames per second is 240 images. Understanding models therefore sample frames (for example one per second), embed each as above, add time information and pass the tokens to a language model. Generation models extend image diffusion with a time dimension: they denoise a block of frames together, often in a compressed latent space that is compressed across time as well as space, and use attention across frames to keep objects consistent. Hard problems include temporal consistency, physics, long durations, audio that matches the picture, and heavy compute. Many systems generate short clips and extend them, and moderation and labeling of synthetic video are especially important.

Shared embeddings across modalities

Embeddings can place different modalities in one space. CLIP-style models train an image encoder and a text encoder together on image-caption pairs with a contrastive loss: matching image-text pairs are pulled together, mismatched pairs are pushed apart. Afterwards, the sentence "a photo of a dog" and a photo of a dog have a high cosine similarity (see Cosine Similarity and Semantic Search). This enables:

  • Searching photos with a text query, or finding similar images.
  • Zero-shot classification: compare an image with text labels and pick the closest.
  • Conditioning for image and audio generators, as in the previous lesson.
  • Multimodal RAG: index images, slides or audio transcripts and retrieve them for a question.
Python Zero-shot classification in a shared space

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Using multimodal APIs

APIs usually accept a list of content parts in one message: text parts, image parts (a URL or base64 data), and sometimes audio. The call looks like the one in Calling an LLM API, with a richer message. This is read-only because it needs a key and network access:

Python
# Not runnable here.
response = client.chat(
    model="some-multimodal-model",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is the total on this receipt?"},
            {"type": "image", "source": {"type": "base64", "data": receipt_b64}},
        ],
    }],
)
print(response.text)

Tips: crop to the relevant region, ask for specific fields in a structured format, and verify numbers, because models can misread blurry text and hallucinate details in pictures just as they do in text.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a spectrogram?

    It measures how much energy each frequency has in each short window, giving a time-frequency image that speech models read.
  2. 2. In a typical vision-language model, what does the projector do?

    The projector lets the language model treat image patches like extra tokens in its own embedding space.
  3. 3. Why is video understanding more expensive than image understanding?

    Each frame becomes many tokens, so even a short clip is hundreds of images unless you sample.
  4. 4. What does a contrastive loss do in CLIP-style training?

    That training puts related images and captions near each other in one shared embedding space.

Summary

  • Every modality becomes vectors: spectrogram frames for audio, patches for images, sampled frames for video.
  • ASR uses an encoder-decoder over spectrograms; TTS predicts spectrograms and uses a vocoder or audio tokens.
  • Vision-language models combine a vision encoder, a projector and a language model; images consume tokens.
  • Video generation adds time to diffusion and is costly and hard to keep consistent.
  • Contrastive training builds shared spaces that enable cross-modal search and zero-shot classification.

Next: Cost, Latency and Caching.