Skip to content
Tutorials Center
My learning

Topics

Lesson 65 of 73 ยท Projects

Project: Face and Emotion Detection

Build a two-stage vision pipeline in NumPy: a sliding-window face detector and a CNN emotion classifier, and learn why the second stage must train on the first stage's crops.

"Detect the faces in this photo and tell me how each person feels." It sounds like one task, but systems that do this are built from two separate models chained together. First a detector answers where are the faces? and produces boxes. Then a classifier looks only at the pixels inside each box and answers which emotion is this? Splitting the job this way is the standard design for many vision applications: license-plate readers, document scanners and medical pipelines all use a detector followed by a recognizer.

In this project you build both stages from scratch and then connect them. As in the other projects, the browser cannot load camera photos or train on large data, so the experiments use generated 24 by 24 scenes that contain one 12 by 12 "face" (an outline, two eyes and a mouth for one of four emotions) among distractors. The pipeline logic, the metrics and the pitfalls are real, and the read-only OpenCV and PyTorch recipe at the end shows how to do it with photographs.

A two-stage pipeline

Detect Slide a window over the image, score each as face or not.
Crop Cut out the best box.
Classify A second CNN names the emotion.
1 / 2

The data: faces, emotions and distractors

Each face is a 12 by 12 picture: a circular outline, two eyes, and a mouth that depends on the emotion. Happy has the corners raised, sad has them lowered, neutral is a flat line and surprised is a small open "o". A scene is a 24 by 24 noisy background with two distractors, objects that look a little like a face but are not (an empty outline with no features, a horizontal bar, a speckled blob), plus one face pasted in at a random position. The distractors matter: a detector trained only on "face versus empty background" would fire on the outline alone.

The first experiment is the second stage in isolation: given a face crop with random noise and up to one pixel of shift, which of the four emotions is it? The CNN is the one from Training a CNN From Scratch. The program also prints each face as text so you can see what the network sees, then a confusion matrix that shows which emotions get mixed up.

Python Stage 2 alone: the emotion classifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classifier reaches 77.5% on 400 unseen noisy crops, which is good against the 25% of guessing but far from perfect. The confusion matrix shows why. Happy faces are recognized almost always (90 of 97), but sad and neutral get confused with each other: 22 sad faces were called neutral and 19 neutral faces were called sad. Their mouths differ by one pixel in each corner, and with noise those pixels are hard to tell apart. That is true for real expressions too: subtle emotions overlap, and a confusion matrix tells you where to collect more data or when to merge categories. Notice also that 15 surprised faces were called happy, a reminder that the errors are not symmetric.

Stage 1: a sliding-window detector

The simplest detector is a binary classifier applied to every window of the image. Train it on 12 by 12 windows labeled "face" or "not face", then slide the window across a new image with a stride of 2 pixels, score each position, and keep the best one. Real detectors are more elaborate (they use anchors, multiple scales and learned box regression, see Object Detection), but this is the idea that the early face detectors used, and it is the right way to learn the mechanics.

Training data matters as much as the model:

  • Positives are windows that are almost exactly on a face (a shift of up to one pixel, so the detector tolerates small misalignment).
  • Negatives are windows at least four pixels away from the face, so they contain background, partial faces and distractors. Including distractors in the negatives is what teaches the detector that an outline alone is not enough.
  • We use twice as many negatives as positives, since in a real image almost every window is a negative.

A box is judged by IoU (intersection over union): the area where the predicted and true boxes overlap, divided by the area they cover together. IoU is 1 for a perfect match and 0 for no overlap; a common success criterion is IoU above 0.5.

The same program then runs the whole pipeline on 150 new scenes: detect, crop the best window, classify. It first uses an emotion classifier trained on clean, centered crops, as in the previous step, and then retrains it on crops taken from scenes, the way the detector will actually hand them over.

Python The full pipeline: detect, crop, classify

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The detector finds the face with IoU above 0.5 in 147 of 150 scenes (98.0%), and the mean IoU is 0.84, so stage 1 works. On average 4.6 windows fire per scene, because a window shifted by a pixel or two still looks like a face. That is why detectors apply non-maximum suppression (NMS): sort the boxes by score, keep the best, discard every box that overlaps it too much, and repeat. With one face per scene we simply keep the strongest window; with several faces you need NMS proper.

Now the interesting result. With the emotion network trained on clean crops, the accuracy on faces that were found falls to 57.8% (56.7% end to end), much lower than the 77.5% measured in isolation. The crops that reach stage 2 are different from the ones it was trained on: they come from a detector that is a pixel or two off, and they carry background clutter and the other half of a distractor sometimes. This is distribution shift between pipeline stages. Retraining the emotion network on crops cut from scenes at slightly jittered positions brings the accuracy on detected faces to 78.9% and the end-to-end accuracy to 77.3%. A lesson that applies to every cascaded system: train each stage on the outputs of the stage before it, and always evaluate the pipeline end to end, not just the parts.

The last line is a reminder about false alarms: on 100 scenes with no face at all, the detector still fires 19 times. A real application sets the confidence threshold with that rate in mind, the same recall/precision trade-off as in Chest X-ray Pneumonia and Lung Cancer Detection.

The real-photo recipe

With photographs you would not train the detector yourself at first. Pretrained face detectors are accurate and fast. The emotion classifier is trained on a labeled face data set. This code is read-only:

Python
# Read-only: detection with OpenCV's Haar cascade, then a CNN on each face crop
import cv2, torch, torch.nn as nn

cascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
gray = cv2.cvtColor(cv2.imread("group.jpg"), cv2.COLOR_BGR2GRAY)
boxes = cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5, minSize=(48, 48))   # minNeighbors does NMS-like merging

class EmotionCNN(nn.Module):                          # for 48 x 48 grayscale crops
    def __init__(self, n_classes=7):
        super().__init__()
        def block(i, o): return nn.Sequential(nn.Conv2d(i, o, 3, padding=1), nn.BatchNorm2d(o), nn.ReLU(), nn.MaxPool2d(2))
        self.net = nn.Sequential(block(1, 32), block(32, 64), block(64, 128), nn.Flatten(),
                                 nn.Dropout(0.4), nn.Linear(128 * 6 * 6, 256), nn.ReLU(), nn.Linear(256, n_classes))
    def forward(self, x):
        return self.net(x)

model = EmotionCNN()      # train it with cross-entropy on FER2013-style data, as in the earlier CNN projects
for (x, y, w, h) in boxes:
    crop = cv2.resize(gray[y:y + h, x:x + w], (48, 48)).astype("float32") / 255.0
    logits = model(torch.tensor(crop)[None, None])        # shape (1, 7)

The standard public data set for this is FER2013: roughly 36,000 grayscale 48 by 48 face images with seven labels (angry, disgust, fear, happy, sad, surprise, neutral). It is noisy, with imbalanced classes (very few "disgust" images) and labels that humans themselves often disagree on, so published accuracies are modest, typically somewhere in the range of 65 to 75%. For a stronger system, start from a pretrained backbone as in Transfer Learning and Fine-Tuning, use mild augmentation (small shifts, brightness changes, horizontal flips are fine for faces), and consider modern detectors such as MTCNN, RetinaFace or YOLO-based face models, which return boxes, scores and landmarks.

Responsible use

Face and emotion technology is sensitive, and this project is a good place to think about it.

  • Detection is not recognition. Finding that a face exists is different from identifying whose face it is. Identification raises much larger privacy and consent questions.
  • Emotion labels are shaky. The link between a facial expression and an inner feeling is disputed by researchers. A model predicts how a face is labeled, usually by crowd workers, not how a person feels. Do not use such output for decisions about people (hiring, grading, policing).
  • Bias. Detectors and classifiers can be less accurate for some skin tones, ages and genders if the data does not cover them. Measure accuracy by subgroup. See Ethics, Bias and Safety.
  • Consent and privacy. Be careful where images come from and where they go. Process on the device where possible, and follow the law of your region.

Ideas to extend the project

  • Detect several faces per scene and implement non-maximum suppression with an IoU threshold.
  • Search multiple scales by scaling the scene and rescanning, so faces of different sizes are found.
  • Replace the sliding window with a fully convolutional detector that scores all positions in one pass, as in Image Segmentation.
  • Add the four-class emotion head to the detector so that one network does both jobs, and compare it with the cascade.
  • Measure accuracy under heavier noise and see which emotions break first.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why are face detection and emotion classification usually separate models?

    A detector localizes objects, and a classifier then labels each crop. The cascade is easy to build and easy to debug stage by stage.
  2. 2. What does IoU measure?

    IoU is 1 for identical boxes and 0 for disjoint ones. A prediction with IoU above 0.5 is usually counted as a correct detection.
  3. 3. Why did the emotion accuracy drop from 77.5% to 57.8% when the classifier was plugged into the pipeline?

    This is distribution shift between stages. Training the classifier on crops like the detector's output recovered most of the accuracy.
  4. 4. What problem does non-maximum suppression solve?

    Because windows a pixel or two apart all look like the same face, several boxes score high. NMS keeps the highest-scoring box and discards overlapping ones.
  5. 5. Why is it risky to use predicted emotion labels for decisions about people?

    The models predict how faces are labeled, and the mapping from expression to feeling is contested and varies by person and culture. Such systems can also be biased.

Summary

  • A face-and-emotion system is a cascade: a detector finds boxes, a classifier labels each crop.
  • A sliding-window detector classifies every window as face or not-face; negatives should include distractors and partial faces. Several windows fire around one face, so non-maximum suppression keeps the best.
  • Judge boxes with IoU, and judge the pipeline end to end: the detector reached 98.0% and the classifier 77.5% alone, but the combination suffered until the classifier was trained on the detector's crops.
  • Subtle classes such as sad and neutral get confused; a confusion matrix shows where.
  • On photographs, use a pretrained face detector and train a CNN on a data set such as FER2013, remembering that labels are noisy and the task has ethical limits.

Next project: Audio Classification.