Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 63 ยท Practice and Production

Speech Recognition and Text-to-Speech

Learn how speech becomes text and text becomes speech: framing, spectrograms and mel features in NumPy, word error rate with edit distance, and the CTC idea.

Most of this course treats language as text that is already typed. Speech is where text meets sound. Automatic speech recognition (ASR), also called speech-to-text, turns an audio recording into words: voice typing, meeting captions, voice assistants and call-center analytics all depend on it. Text-to-speech (TTS), or speech synthesis, goes the other way: reading a message aloud, navigation prompts, audiobooks and accessibility tools.

Both are language-processing tasks with a signal-processing front or back end, and most of the NLP ideas you already know come back: language models rank candidate transcripts, text normalization prepares input for the voice, and evaluation by edit distance scores the output. In this lesson you will turn a sound wave into a spectrogram with NumPy, build mel features, compute the standard ASR metric (word error rate) from scratch, and see the central trick of modern end-to-end ASR, CTC, in a few lines of Python. The browser cannot record or play audio in these examples, so everything runs on synthetic signals and numbers, and the real toolkits are shown read-only.

The big picture

A speech recognizer must bridge three very different things: a waveform sampled thousands of times a second, a sequence of sounds, and a sequence of words. Classical systems used separate parts for each step (an acoustic model, a pronunciation dictionary and a language model). Modern systems learn most of it in one neural network, but the same stages are still visible.

flowchart LR
    A["Audio waveform"] --> B["Frames and spectrogram"]
    B --> C["Log-mel features"]
    C --> D["Neural acoustic model"]
    D --> E["Characters or tokens with CTC"]
    E --> F["Decoder and language model"]
    F --> G["Text transcript"]

A typical ASR pipeline: signal processing, a neural network that outputs symbol probabilities for every time step, then decoding into words.

TTS runs a mirror-image pipeline: text is normalized (Dr. becomes Doctor, 1984 becomes nineteen eighty-four or one thousand nine hundred eighty-four, depending on context), converted to phonemes, and a neural model predicts an acoustic representation such as a mel spectrogram. A vocoder then turns that representation into a waveform.

From waveform to spectrogram

A digital recording is a list of numbers: the air pressure measured, for example, 16,000 times per second (the sample rate). One second of speech is 16,000 numbers, and the raw numbers hardly show which sound was spoken. What identifies a vowel or a consonant is which frequencies are present, and that changes every few milliseconds. So we cut the signal into short overlapping frames, usually 25 ms long and 10 ms apart, multiply each frame by a smooth window (to avoid sharp edges), and compute its Fourier transform, which says how much energy sits at each frequency. Stacking the frames gives a spectrogram: time on one axis, frequency on the other, energy as brightness.

Human hearing is not linear in frequency. We distinguish 200 Hz from 300 Hz easily, but 7,200 Hz from 7,300 Hz hardly at all. The mel scale models this, and a mel filterbank is a set of overlapping triangles, narrow at low frequencies and wide at high ones, that sum the spectrogram into a few dozen bands. Taking the logarithm (loudness is also roughly logarithmic) gives log-mel features, the standard input to speech models. Our example uses 10 bands to keep the output small; real systems use 80.

Python Framing, spectrogram and mel features in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One second of audio, 8,000 numbers, became 61 frames with 129 frequency bins each, and then 61 frames with 10 mel bands. The analysis finds 500 Hz in the early frames and 1,500 Hz from frame 33 on (frame 30 is centered just before the change at 0.5 s and is still dominated by the first note, while frame 33 starts after the change). The bin spacing is 31.25 Hz, which is why both frequencies land exactly on a bin. The band centers show the mel idea: the first bands are about 150 Hz apart, the last ones more than 600 Hz. The text picture at the bottom has time running left to right. The bright rows shift from band 2 to band 6 halfway through. (The darker patches in other rows are the leftover energy that the window leaks into neighboring frequencies.) Real speech has many frequencies at once, but the principle is identical.

Some modern models skip hand-made features and take the raw waveform or learn their own filters (wav2vec 2.0 and its relatives), and recent models such as Whisper still use an 80-band log-mel input. Either way, you should know what the representation holds.

Measuring recognition: word error rate

How do you compare a recognizer's output (the hypothesis) with what was really said (the reference)? The standard metric is the word error rate (WER). It counts the minimum number of word edits needed to turn the hypothesis into the reference: substitutions (S), deletions (D) and insertions (I), divided by the number of words in the reference (N):

WER = (S + D + I) / N

This is the same edit distance you may know from spelling correction, applied to words instead of letters, and it is computed with the same dynamic-programming table plus a backtrace to find which edits were used. For languages without spaces between words, or for names and numbers, the same code on characters gives the character error rate (CER). The program below computes both.

Python Word error rate and character error rate from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The perfect transcript has WER 0%. The second one has two substitutions (off to of, lights to light) and one deletion (the second the), three errors in eight reference words, so 37.5%. The third adds two extra words, and insertions count as errors too: 25%. The last transcript is interesting because it has the same WER as the second, 37.5%, even though burn off the lice in the leaving room would make a voice assistant do something quite different from the other one. WER treats every word as equally important and ignores meaning. The CER on the last line shows the other side: of and light are only a couple of characters away from the reference, 0.105, while the word-level score punished them as full errors.

Two practical notes. WER can exceed 100% when a system inserts many words. And scores depend on normalization: before scoring, teams usually lowercase the text, strip punctuation and spell out numbers consistently, otherwise Dr. versus doctor counts as an error. Toolkits such as jiwer do this, and the same family of scoring appears in translation evaluation with BLEU.

The alignment problem and CTC

There is a basic difficulty in speech recognition: there are many more audio frames than letters. One second of speech has 100 frames but only about 15 characters, and nobody told the model which frame belongs to which letter. Connectionist Temporal Classification (CTC) solves this without alignment data. The model outputs, for every frame, a probability distribution over the characters plus a special blank symbol. A frame-level sequence (a path) is turned into text with two rules: merge consecutive repeats, then remove blanks. So hh_e_ll_lo and hheelllloo do not give the same text: the blank in ll_lo is what separates the double l in hello from a single l.

Training maximizes the total probability of all paths that collapse to the correct transcript, using a dynamic program instead of enumerating them. In a toy example with 8 frames and four symbols we can still enumerate all 65,536 paths by brute force, and the example below does so to make the idea concrete.

Python CTC: collapsing paths and summing alignments

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first five lines show the collapse rule. hh_e_ll_lo, h_ee_l_l_o and _hel_lo__ all become hello, hel_lo does too, but hheelllloo collapses to helo because nothing separates the two ls. The fake model puts the most weight on c in frames 1 and 2, a in frames 4 and 5 and t in frame 6, with blanks around them, and greedy decoding (take the best symbol per frame, then collapse) reads cat. The sum over all alignments gives cat a probability of about 0.35 from 462 different paths, while ct (a path that skips the a) gets 0.011 and at (skipping the c) only 0.007. In a real system the sum is computed by the CTC forward algorithm in time proportional to frames times transcript length, and decoding runs a beam search that can also mix in a language model score (see beam search).

CTC is not the only approach. Attention-based encoder-decoder models (Whisper, for example) read all the audio and write the text token by token, like a translation model, and transducer (RNN-T) models combine a CTC-like streaming alignment with a built-in language model, which is the common choice for live captions on phones.

Running real ASR and TTS

These packages need model downloads, audio files and sometimes a GPU, so the code is read-only here:

Python
# Speech recognition with a pretrained Whisper model
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
print(asr("meeting.wav", return_timestamps=True)["text"])

# Scoring it against a human transcript
import jiwer
print(jiwer.wer("turn off the lights", "turn of the light"))

# Text to speech with a pretrained model, saved as a WAV file
from transformers import pipeline
import scipy.io.wavfile as wavfile

tts = pipeline("text-to-speech", model="suno/bark-small")
speech = tts("Hello! Your order has shipped.")
wavfile.write("hello.wav", speech["sampling_rate"], speech["audio"].squeeze())

Consider these points before you build on speech technology. Accuracy varies a lot with accents, background noise, domain vocabulary and overlapping speakers, so measure WER on your audio, split by speaker group. Recordings contain personal data, so storage and consent matter (see responsible NLP). Cloned or synthetic voices can be misused, so many products label synthetic speech. Finally, put a language-aware step after ASR: punctuation and casing restoration, number formatting and named entities are separate NLP tasks that make a raw transcript readable, and an error from the recognizer will flow into everything downstream (see NLP in production).

Whisper, resampling and where ASR is used

Whisper is an open encoder-decoder Transformer from OpenAI trained on hundreds of thousands of hours of multilingual audio. Unlike the CTC models above, it is sequence-to-sequence: the encoder reads a log-mel spectrogram and the decoder writes the transcript token by token, so the same model can transcribe, translate speech to English and identify the language. In the Hugging Face library it is loaded as WhisperForConditionalGeneration together with a processor that turns audio into features.

Audio must match what the model was trained on. Whisper expects mono audio at a 16 kHz sampling rate, so a recording made at 44.1 kHz is resampled first (libraries such as torchaudio or librosa do this with a proper low-pass filter), and long recordings are cut into 30-second windows. The sketch below shows both ideas with NumPy, using plain linear interpolation instead of a filtered resampler.

Python Resampling to 16 kHz and counting 30-second windows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A one-second tone has 44,100 samples before and 16,000 after, but both last exactly one second, so only the sampling density changed. A 95-second recording needs four 30-second windows of 480,000 samples each. If you forgot to resample, the model would hear the audio about 2.75 times too slow and low-pitched, and the transcript would be garbage.

Python
# pip install transformers torch torchaudio  (read-only here)
from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
print(asr("meeting.wav", chunk_length_s=30)["text"])

Applications

Typical uses are meeting transcription and note taking, video subtitling, voice assistants, call-center analytics, dictation in medical and legal work, and accessibility tools. CTC models suit low-latency streaming and small devices; Whisper-style models give higher accuracy across accents and languages but are heavier. Related uses of the CTC idea include handwriting and gesture recognition, where a sequence of inputs must be aligned to a shorter sequence of labels.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do speech systems cut audio into short overlapping frames before computing the spectrum?

    One transform over the whole recording would blur everything together. Short frames, about 25 ms long, keep time resolution while still resolving frequency.
  2. 2. A reference has 10 words. The hypothesis has 1 substitution, 1 deletion and 2 insertions. What is the WER?

    WER = (S + D + I) / N = (1 + 1 + 2) / 10 = 0.4. Insertions count as errors and the denominator is the reference length.
  3. 3. In CTC, why does the path "ll" collapse to a single l but "l_l" to two?

    First consecutive repeats are merged, then blanks are dropped. In "l_l" the blank stops the two l symbols from being merged.
  4. 4. Two transcripts have the same WER. Which statement is correct?

    WER counts word edits and ignores meaning, so "turn off" versus "burn off" can cost the same as an unimportant function-word error.
  5. 5. What is the job of a vocoder in a TTS system?

    The model predicts a spectrogram-like representation, and the vocoder generates the audible waveform from it.

Summary

  • ASR turns audio into text, TTS turns text into audio; both combine signal processing with language modeling.
  • Audio is cut into short windowed frames, transformed to a spectrogram, and compressed with a mel filterbank and a logarithm into log-mel features.
  • Word error rate is (substitutions + deletions + insertions) divided by the reference length; it is word-level edit distance, and CER is the same idea on characters. Normalize text before scoring, and remember that WER ignores meaning.
  • CTC lets a model emit one symbol (or a blank) per frame without alignment labels: merge repeats, drop blanks, and train on the total probability of all paths that give the right text.
  • Real systems use pretrained models (Whisper-like encoder-decoders or streaming transducers); evaluate them on your own audio, groups of speakers and noise conditions.

Next, see how another sequence-to-sequence task works in Machine Translation in Depth.