Skip to content
Tutorials Center
My learning

Topics

Lesson 66 of 73 ยท Projects

Project: Audio Classification

Turn sound into spectrograms with NumPy, classify five kinds of audio with a CNN, and fix noise problems with augmentation and SpecAugment-style masking.

Can a program tell a siren from a clap, or recognize the word "yes" in a spoken clip? Audio classification powers voice assistants, wake-word detectors, music tagging, bird-call identification and machine-health monitoring. The surprise for newcomers is that the best-known recipe turns an audio problem into an image problem. You convert the sound into a picture of how its frequencies change over time, a spectrogram, and then apply the convolutional networks from the earlier lessons.

In this project you build that pipeline from the ground up in NumPy: synthesize sounds, compute the short-time Fourier transform, map it onto the mel scale, train a CNN, and then make it robust to noise. There is no microphone or audio data set in the browser, so the clips are generated (a steady tone, a siren, a rising chirp, a clap and a low hum), which also lets us know exactly what the model should be hearing.

From sound to picture

Waveform Air pressure over time, thousands of numbers a second.
Short windows Cut it into frames of a few milliseconds.
Spectrum per frame FFT: how much energy at each frequency.
Spectrogram Frames side by side form an image.
1 / 2

Step 1: from waveform to log-mel spectrogram

A digital recording is a long list of numbers: the air pressure sampled thousands of times per second. Our clips are half a second long, sampled at 4,000 Hz, so each has 2,000 numbers. That raw waveform is a poor input for a network. Two sounds with the same pitch can look completely different sample by sample, because they differ in phase, while the thing we care about, which frequencies are present when, is hidden.

The short-time Fourier transform (STFT) exposes it. Cut the waveform into short overlapping frames (here 128 samples, about 32 ms, advancing 120 samples each time), multiply each frame by a smooth Hann window so the edges do not leak, and apply the FFT. Each frame becomes a list of energies at 65 frequencies. Stack the frames side by side and you have a spectrogram: time along one axis, frequency along the other, brightness for energy. The frame length is a trade-off: longer frames separate close frequencies better but smear changes in time.

Two refinements make it better suited to learning. First, the mel scale: human hearing resolves low frequencies in much finer detail than high ones, so we group the 65 bins into 16 triangular mel bands spaced evenly on the mel scale, giving more bands to low frequencies. Second, we take the logarithm of the energy, because loudness is perceived logarithmically, and a raw spectrogram is dominated by a few very loud values. The result is the log-mel spectrogram, the standard input for audio networks.

Python Waveform, STFT and mel bands, drawn as text

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The siren is the clip printed in numbers: the loudest frequency in each frame rises and falls (687, 750, 781, 718, 656, 562, 531, ... Hz), which is exactly the wobble the generator produced. The text pictures show each sound as an image, with frequency going up the side and time going across. The steady tone is a horizontal bar, since its pitch never changes. The siren is a wave that rises and falls. The rising chirp is a diagonal from the lower left to the upper right. The clap is a vertical stripe, because noise contains all frequencies for a short moment. The hum is a stack of horizontal lines, a fundamental and its overtones. A CNN is good at exactly these shapes: lines, diagonals and stripes, wherever they occur.

Step 2: a CNN on spectrograms

Now treat each 16 by 16 log-mel spectrogram as a one-channel image and train the small CNN from Training a CNN From Scratch: a 3 by 3 convolution with 8 filters, ReLU, max pooling and a dense softmax layer. Two practical details:

  • Normalize the spectrograms with the training set's mean and standard deviation.
  • Augment during training. Audio augmentations include adding background noise, shifting the clip in time, changing the pitch or speed, and mixing two clips. For spectrograms, SpecAugment masks a random band of frequencies and a random stretch of time, forcing the model not to rely on any single patch. We use that here.
Python Classify five sounds, then stress-test with noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On clips like those it was trained on, the network scores 97.3%. Four classes are perfect, and the confusion matrix shows that all of the errors are sirens mistaken for steady tones (8 of 61): within a half-second clip a slow wobble can look like a flat line. The robustness test is more revealing. At the training noise level (0.3), accuracy holds at 97.0%. With noise at 0.6, a level it never saw, accuracy falls to 35.0%, and at 1.0 it is 20.0%, which is chance for five classes. This is the central lesson of audio work: a model only knows the conditions it was trained in. A spectrogram contaminated with noise looks like a different image.

The fix is to train on the conditions you expect. When the same network is trained on clips with noise levels between 0.2 and 1.2, accuracy at noise 0.3 is 96.5%, at 0.6 it is 96.0%, and at 1.0 it is 86.5%. The small loss on clean data is a cheap price. In real projects the equivalent is mixing in recorded background sounds (street noise, cafe chatter, room reverb) at random volumes, which is far more convincing than Gaussian noise.

Step 3: the real-audio recipe

With real recordings you load audio files, resample them to a common rate, pad or crop to a fixed length, and compute the same log-mel spectrogram with a library. This code is read-only:

Python
# Read-only: log-mel spectrograms and a CNN on real audio with torchaudio
import torch, torch.nn as nn, torchaudio

wave, sr = torchaudio.load("clip.wav")                                   # (channels, samples)
wave = torchaudio.functional.resample(wave.mean(0, keepdim=True), sr, 16000)
wave = nn.functional.pad(wave, (0, max(0, 16000 - wave.shape[1])))[:, :16000]     # exactly one second

to_mel = torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_fft=400, hop_length=160, n_mels=64)
to_db = torchaudio.transforms.AmplitudeToDB()
spec = to_db(to_mel(wave))                                               # (1, 64 mel bands, 101 frames)

augment = nn.Sequential(torchaudio.transforms.FrequencyMasking(freq_mask_param=8),     # SpecAugment, as above
                        torchaudio.transforms.TimeMasking(time_mask_param=12))

def block(i, o): return nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(), nn.MaxPool2d(2))
model = nn.Sequential(block(1, 16), block(16, 32), block(32, 64), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(64, 10))
logits = model(augment(spec).unsqueeze(0))                               # train as in any image classifier

Useful public data sets: Speech Commands (about 105,000 one-second recordings of 35 spoken words, the classic for keyword spotting), UrbanSound8K (about 8,700 clips of 10 city sounds such as sirens, drills and dog barks) and ESC-50 (2,000 five-second clips in 50 environmental sound classes). For better accuracy than a small CNN, fine-tune a pretrained audio model (for example an audio spectrogram transformer or a model pretrained on AudioSet) with the transfer-learning recipe from Transfer Learning and Fine-Tuning.

Design choices to think about

  1. Clip length. Sounds that unfold over time need a window long enough to contain the pattern. Our siren was ambiguous in half a second.
  2. Time resolution versus frequency resolution. Frame length and hop trade one against the other.
  3. Fixed size. CNNs with dense heads need equal-sized inputs, so pad or crop; global pooling (as in the code above) accepts varying lengths.
  4. Splits. Split by recording session or speaker, not by clip, or the model may memorize voices and rooms (the same leakage problem as in Chest X-ray Pneumonia and Lung Cancer Detection).
  5. Streaming. For a wake-word detector, run the model on a sliding window of audio and require several consecutive detections to cut false alarms.
  6. Sequence models. For longer audio, run the CNN on short windows and feed its features to an LSTM or a transformer, as in Recurrent Networks.

Ideas to extend the project

  • Raise the model's difficulty: add two simultaneous sounds and make it multi-label (a sigmoid output for each sound).
  • Replace SpecAugment by time shifts and compare the robustness table.
  • Try 8 or 32 mel bands, a longer FFT window, or linear frequency bins instead of mel, and see how accuracy changes.
  • Classify a long recording by averaging the predictions of overlapping one-second windows.
  • Add delta features (the change of each band over time) as a second input channel.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a spectrogram show?

    A spectrogram stacks the frequency spectra of short consecutive frames, giving a time-frequency image.
  2. 2. Why take the logarithm of the spectrogram energies?

    The log compresses the huge dynamic range so that quieter details remain visible to the network.
  3. 3. What did the clap look like in the spectrogram?

    Noise spreads energy over all frequencies for a short time, which appears as a vertical stripe.
  4. 4. The network scored 97% at noise level 0.3 but 35% at 0.6. What does this show?

    Distribution shift: at a new noise level the spectrograms differ from those seen in training. Training with a range of noise levels fixed it.
  5. 5. What does SpecAugment do?

    Masking forces the model to use redundant cues instead of relying on any one region, which improves robustness.

Summary

  • Audio classification usually converts the waveform to a log-mel spectrogram, an image of frequency content over time, and applies a CNN.
  • The STFT cuts the signal into windowed frames and takes an FFT of each; mel bands group frequencies the way hearing does, and the log compresses loudness.
  • Tones, sirens, chirps, claps and hums look like lines, waves, diagonals, stripes and stacked lines, patterns that convolutions detect well.
  • The NumPy CNN reached 97.3% on matched clips but collapsed to 35.0% on noisier ones; training on a range of noise levels restored 96.0% at noise 0.6 and 86.5% at noise 1.0.
  • Augment with realistic noise, time shifts and SpecAugment masks, split by speaker or session, and use pretrained audio models for hard tasks.

Next project: Human Activity and Sign Language Recognition.