Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 111 ยท Feature Engineering and Pipelines

Feature Extraction from Signals and Images

Turn raw signals and images into compact features: summary statistics, Fourier and wavelet features, HOG for images, and extraction versus selection.

Many datasets do not arrive as neat tables. A sensor gives you 200 readings per second, a photo gives you thousands of pixels, a recording gives you a waveform. Feeding those raw numbers straight into a model is possible, but it is often wasteful and fragile. Feature extraction converts raw data into a smaller set of numbers that capture what matters for the task: the dominant frequency of a vibration, the direction of the edges in a picture, the average length of a word.

This lesson shows the idea with three kinds of data: time signals (summary statistics, the Fourier transform, wavelets), images (HOG) and a short look at other sources. It builds on creating new features, which works with tabular columns.

Extraction, selection and creation

These three words are easy to mix up:

  • Feature creation builds new columns from existing columns, such as a ratio or an interaction (feature creation).
  • Feature selection keeps a subset of the columns you already have (feature selection).
  • Feature extraction transforms raw data (or many columns) into a new, usually smaller representation. PCA is an extraction method: it replaces your columns with a few new axes. TF-IDF extracts numbers from text.

Good extraction does four things. It simplifies the data (200 readings become 3 numbers), it highlights patterns a model would struggle to find alone, it can improve accuracy and speed, and it makes high-dimensional data manageable. The price is that you must choose the transformation using domain knowledge, and a bad choice throws away exactly the information you needed.

Statistical summaries of a signal

The simplest approach summarizes each recording with a few numbers: mean, standard deviation, minimum, maximum, skewness, number of zero crossings. These are cheap and often useful, but they ignore when things happen and how fast the signal oscillates.

Here is a problem where that fails. Two machines vibrate at different frequencies (5 Hz and 12 Hz), but with the same amplitude and random timing. We compare three feature sets with a random forest.

Python Raw samples, simple statistics and Fourier features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The four summary statistics are almost useless here (accuracy 0.38, worse than guessing, because noise dominates), because both machines have the same amplitude. The 200 raw samples work reasonably (0.96) but depend on the random timing of each recording, so the forest needs many splits to cope. Three Fourier features separate the two classes perfectly (1.00) and are easy to read.

The Fourier transform

The Fourier transform rewrites a signal as a sum of sine waves with different frequencies. For each frequency it tells you how strong that wave is. np.fft.rfft does this for real-valued signals, and np.abs of the result gives the magnitude spectrum. Useful features from it:

  • the peak frequency (where the spectrum is highest),
  • the spectral centroid (the average frequency, weighted by magnitude),
  • the energy in frequency bands (for example 0 to 4 Hz, 4 to 8 Hz, ...),
  • the spectral entropy (how spread out the energy is).

Because it ignores the phase, the magnitude spectrum does not care where in the recording a wave starts. That is why it handled the random timing above so well. Use it when the frequency content identifies the class: vibration monitoring, audio, heartbeats, brain signals.

Wavelets: frequency plus time

A weakness of the Fourier transform is that it says which frequencies are present but not when they occur. A short burst anywhere in the recording spreads a little energy over many frequencies. Wavelets analyze the signal at several scales at once, and each coefficient is tied to a position in time. The library PyWavelets (pywt) splits a signal into an approximation (the slow trend) and several levels of detail (faster and faster changes). The energy in each level makes a compact feature vector.

Python Wavelet energy per scale for a smooth signal and a signal with a burst

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The approximation holds the slow 3 Hz wave in both signals. The fine-scale detail levels (the last three numbers) are close to zero for the smooth wave and clearly larger for the signal with a burst, so even a four-sample event shows up in the features. Wavelet features are popular for ECG and EEG analysis, machine fault detection, and image compression.

Images: histograms of oriented gradients (HOG)

For images, raw pixels have two problems. There are many of them, and small changes (a shifted object, a brighter picture) change many pixel values at once. HOG describes an image by the directions of its edges:

  1. Compute the gradient (change in brightness) at each pixel: its strength and direction.
  2. Split the image into small cells (for example 4 ร— 4 pixels) and build a histogram of gradient directions in each cell, weighted by strength.
  3. Normalize the histograms over overlapping blocks of cells, which makes the result robust to lighting changes.
  4. Concatenate everything into one feature vector.

skimage.feature.hog does all four steps. We use the 8 ร— 8 digit images from scikit-learn, train on the first 1,200 images and test on the rest, both as they are and with every image moved by one pixel.

Python HOG features versus raw pixels on handwritten digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On these tiny images, HOG with fine cells matches raw pixels on the normal test set (0.93 versus 0.92) and, like every method here, loses accuracy when the digit moves. The coarse 4 ร— 4 cells use only 36 numbers per image and drop to 0.83 on the normal test set, but they hold up better when the image is shifted (0.66 versus 0.50 for raw pixels). That is the trade-off in a nutshell: a compact, shift-tolerant representation versus the detail you throw away. On larger photos the effect grows, and HOG was the standard input for pedestrian detectors before deep learning.

Other kinds of extraction

  • Text: bag of words and TF-IDF (text features); n-grams; word counts, average word length, number of exclamation marks.
  • Tabular data: PCA, factor analysis, NMF and others, which replace many columns with a few combinations.
  • Time and dates: hour, weekday, month, "is holiday" flags, cyclic encodings with sine and cosine.
  • Images and audio with deep learning: a pretrained convolutional network turns an image into a vector (an embedding) that you can feed to any classical model. Libraries such as OpenCV, TensorFlow, Keras and PyTorch provide these tools; they are not available in the in-browser runner, so we do not run them here.
  • Domain-specific: the QRS interval of a heartbeat, the pitch of a voice, the number of edges in a shape. Experts often know the best features.

Challenges and good habits

  • Information loss. Every extraction discards something. Check that the features still separate your classes (as the 0.38 accuracy above warns).
  • Computation. Transforms such as wavelets or HOG on millions of items cost time. Do it once and store the result.
  • Leakage. Any step that learns from data (PCA, scalers, vocabulary) must be fitted on the training split only. Put it in a pipeline.
  • Reproducibility. Use the exact same extraction code for training and serving (training/serving skew).
  • Try the simple thing first. Compare against a baseline on raw features before you invest in a complicated extractor.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the magnitude spectrum from np.fft.rfft tell you about a signal?

    The Fourier transform decomposes the signal into sine waves, and the magnitude says how strong each frequency is. It does not say when the frequencies occur.
  2. 2. Why can wavelet features reveal a short burst that Fourier features blur?

    Each wavelet coefficient describes a particular scale at a particular position, so a local event stays local instead of spreading over all frequencies.
  3. 3. What is the main idea behind HOG features?

    HOG captures local edge directions, which survive lighting changes and small shifts better than raw pixel values.
  4. 4. How does feature extraction differ from feature selection?

    PCA, FFT features and HOG create a new representation. SelectKBest or RFE just choose among the columns you already have.

Summary

  • Feature extraction turns raw signals, images or text into a compact set of informative numbers.
  • Summary statistics are cheap but blind to frequency and timing; Fourier features capture frequency content; wavelets add time localization.
  • HOG describes images by local edge directions and is more compact and shift-tolerant than raw pixels.
  • Extraction always loses information, so compare against a baseline and fit learned steps on training data only.

Next: look at how clusters can also be found without choosing the number of groups in advance, in More Clustering Algorithms.