Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 38 ยท Pipelines and Inference

Pipelines for Vision and Audio

Run image classification, object detection, speech recognition and audio classification pipelines, and see how images become patches and audio becomes frames.

Pipelines are not limited to text. Vision and audio models also need preprocessing, a forward pass and post-processing, but the first step is done by a processor (or image processor / feature extractor) instead of a tokenizer. The idea is the same: convert raw data into the numeric tensors the model expects.

All real calls here are read-only; the stand-ins run in your browser and show the preprocessing ideas with NumPy.

Image classification

Python
from transformers import pipeline

clf = pipeline("image-classification", model="google/vit-base-patch16-224")
clf("photo.jpg")                       # a path, a URL or a PIL image
# [{'label': ..., 'score': ...}, ...]   top few labels

The image processor resizes the picture, normalizes pixel values with the mean and standard deviation the model was trained with, and arranges the channels. Using the wrong normalisation is a classic silent bug; always use the processor that belongs to the model.

How a Vision Transformer sees an image

A ViT cuts the image into square patches (for example 16 by 16 pixels), flattens each patch into a vector and treats the list of patches as a sequence of tokens, with a position embedding for each. A 224 by 224 image with 16-pixel patches gives 14 x 14 = 196 patches.

Python Stand-in: cut an image into patches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a stand-in for the patch step only. A real ViT then projects each patch with a learned linear layer and adds the position embeddings.

Object detection and segmentation

Python
detector = pipeline("object-detection", model="facebook/detr-resnet-50")
detector("street.jpg")
# [{'score': ..., 'label': ..., 'box': {'xmin':..., 'ymin':..., 'xmax':..., 'ymax':...}}, ...]

segmenter = pipeline("image-segmentation", model="nvidia/segformer-b0-finetuned-ade-512-512")

Detection returns bounding boxes with labels; segmentation returns a mask per object or class. Always filter detections by score. A typical filter is score > 0.9 for a precise result and a lower threshold if recall matters.

Python Stand-in: filter detections and compute overlap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second step is called non-maximum suppression. Many detectors do it internally; the stand-in shows the logic.

Speech recognition

Python
asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
asr("meeting.wav")                                     # {'text': '...'}
asr("long.wav", chunk_length_s=30, return_timestamps=True)

Audio is a long list of numbers sampled at a fixed rate, for example 16,000 samples per second. Models expect the sampling rate they were trained with (Whisper: 16 kHz); the pipeline resamples files for you if ffmpeg is installed. Whisper turns the waveform into a log-mel spectrogram: time slices of the signal converted into energy across frequency bands. The encoder reads these frames and the decoder writes the transcript, exactly like a translation model.

Long recordings are cut into chunks (chunk_length_s) with overlap so words at the boundaries are not lost.

Python Stand-in: split audio into frames and chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A frame every 10 ms is the standard speech setting, so Whisper sees 100 frames per second of audio before it compresses them.

Audio classification and other tasks

Python
audio_clf = pipeline("audio-classification", model="superb/wav2vec2-base-superb-ks")

Other tasks follow the same pattern: zero-shot-image-classification (CLIP), image-to-text (captioning), visual-question-answering, document-question-answering, depth-estimation, text-to-speech. You can discover more at the task pages of the Hub.

Practical tips

  • Install the extra packages (pillow for images, librosa or ffmpeg for audio).
  • Resample audio and resize images the way the model card says, or rely on the processor.
  • Use chunk_length_s for long audio and tile large images.
  • Models differ in the labels they know: ImageNet classifiers know 1,000 classes, not "my product".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does a Vision Transformer turn an image into a sequence?

    Patches are flattened, projected and given position embeddings.
  2. 2. How many 16x16 patches does a 224x224 image give?

    224 / 16 = 14 per side, and 14 x 14 = 196.
  3. 3. What sampling rate does Whisper expect?

    Whisper was trained on 16 kHz audio; other rates are resampled.
  4. 4. Why use chunk_length_s for long audio?

    Models have a fixed window; chunks with overlap cover long recordings.

Summary

  • Vision and audio pipelines use image processors and feature extractors instead of tokenizers.
  • ViT treats image patches as tokens; Whisper turns audio into spectrogram frames and decodes text.
  • Filter detections by score and remove duplicates with non-maximum suppression.
  • Match sampling rates, normalisation and chunking to what the model expects.

Next: Text Generation and Decoding Strategies