Skip to content
Tutorials Center
My learning

Topics

Lesson 67 of 73 ยท Projects

Project: Human Activity and Sign Language Recognition

Recognize hand signs and body activities from pose keypoints in NumPy: landmark normalization for static signs, and an LSTM against MLPs for motion sequences.

Cameras produce millions of pixels per frame, but to recognize what a person is doing, you rarely need most of them. A pose-estimation model can reduce each frame to a short list of keypoints: the positions of the wrist, fingertips, elbows, knees and so on. A hand becomes 21 points and a body becomes 33. Recognizing a hand sign or an activity from keypoints is then a small, fast, cheap machine-learning problem, and it ignores clothing, lighting and background, which makes it more robust than working from raw video.

This project covers the two settings that cover most of the field. Static recognition classifies a single frame, such as a hand shape in a fingerspelled alphabet. Dynamic recognition classifies a motion over many frames, such as walking, running or waving, and needs a model that understands order. You will write both in NumPy, on generated keypoints (the browser has no camera or pose model), and learn the preprocessing step that decides whether such systems work at all.

Keypoints instead of pixels

Hand 21 landmarks per hand.
Body 33 landmarks per person.
Small input Dozens of numbers instead of millions of pixels.
1 / 2

Part 1: recognizing hand signs

Our simulated hand has 21 landmarks: a wrist and four points on each of five fingers. A finger is either extended (a long line of points) or curled (a short stub, as it appears in a flat 2-D view). Six signs are defined by which fingers are extended: fist, open hand, point, peace, thumbs up and call me. Every image places the hand at a random position, scale (0.5 to 2 times) and tilt (up to 20 degrees), as a real camera would. The classifier is a small two-layer network (42 inputs, 64 hidden units, 6 outputs) trained with Adam.

The point of the experiment is normalization. Raw camera coordinates mix what the hand looks like with where it is and how big it is. A normalization step removes the nuisance: subtract the wrist position so the wrist sits at the origin (translation invariance), and divide by the distance from the wrist to the base of the middle finger, the hand's own size (scale invariance). Rotation can be handled by also rotating the hand so that this line points up, or, as here, by letting the network absorb a small tilt.

Python Hand signs: raw coordinates versus normalized landmarks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table makes the case for preprocessing. With only 150 training images, a network on raw coordinates reaches 45.0% test accuracy, because it has to learn every combination of position and size from a handful of examples. The same network on normalized landmarks gets 99.8%. With 600 training images the raw version catches up to 95.8%, but normalization still reaches 100.0%. Good preprocessing is worth hundreds of labeled examples. The principle is general: put the invariances you know about into the input rather than asking the network to discover them. Other useful keypoint preprocessing includes flipping left hands to look like right hands, rotating to a canonical orientation, and using distances or angles between joints (such as the angle at each finger joint) rather than raw coordinates.

Part 2: recognizing activities from motion

A hand sign is a pose. An activity is a pose that changes. Walking and running use the same limbs and nearly the same amplitude, and differ mainly in speed: how many swings per second. A single frame cannot tell them apart, so the model must see several frames in order.

Our generator produces 24-frame clips of six joints (head, both hands, hip and both feet), each with an x and a y coordinate: 288 numbers per clip. There are five activities. Walking and running swing feet and hands in opposition, at about 2 versus 4 cycles per clip. Waving raises the right hand above the head and oscillates it quickly. Jumping bounces the whole body with the arms up. Sitting is a static low pose. Each clip gets a random starting phase, speed and amplitude, plus noise, so no two look the same.

We compare three models:

  1. An MLP on summary statistics of each coordinate (its mean and standard deviation over the clip, 24 numbers). This discards the order of frames and the frequency of the motion.
  2. An MLP on the flattened sequence (all 288 numbers at once).
  3. An LSTM (from LSTM and GRU) that reads the 24 frames one at a time and classifies from its final state.
Python Activity recognition: summary statistics, MLP and LSTM

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The summary-statistics model scores 83.7% overall, and the breakdown shows exactly what it lacks: waving, jumping and sitting are 100%, but running is only 33% and walking 70%, because they have similar means and spreads and differ only in frequency, which the summary threw away. Models that see the whole sequence solve it: the flattened MLP and the LSTM both reach 100.0%.

That the flattened MLP does as well as the LSTM is itself a fair lesson. On short, clean, fixed-length clips a plain MLP is a strong baseline, and you should always try one. The LSTM's advantages show up elsewhere, and the program prints two of them. First, it has far fewer parameters (3,677 versus 18,821), because the same weights are reused at every frame. Second, it accepts clips of any length without retraining: given only the first 16 of the 24 frames it still scores 98.0%, whereas the flattened MLP cannot even accept the input. That matters for real-time use, where you want to classify as frames arrive. Longer, messier recordings widen the gap further, and modern systems often use temporal convolutions or transformers over the keypoint sequence for the same reasons.

The real-data recipe

On real video you would obtain keypoints from a pose-estimation library and feed the sequences to your model. This code is read-only:

Python
# Read-only: MediaPipe landmarks + a PyTorch LSTM (needs mediapipe, opencv-python, torch)
import cv2, numpy as np, torch, torch.nn as nn
import mediapipe as mp

hands = mp.solutions.hands.Hands(static_image_mode=False, max_num_hands=1)

def landmarks_from_frame(frame_bgr):
    result = hands.process(cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB))
    if not result.multi_hand_landmarks:
        return np.zeros(42, dtype=np.float32)                            # no hand found: a real pipeline must handle this
    pts = np.array([[p.x, p.y] for p in result.multi_hand_landmarks[0].landmark], dtype=np.float32)
    pts -= pts[0]                                                        # wrist at the origin
    return (pts / np.linalg.norm(pts[9])).reshape(-1)                    # divide by the wrist-to-middle-finger-base distance

class SignLSTM(nn.Module):
    def __init__(self, n_classes, hidden=64):
        super().__init__()
        self.lstm = nn.LSTM(42, hidden, num_layers=2, batch_first=True, dropout=0.2)
        self.head = nn.Linear(hidden, n_classes)
    def forward(self, x):                                                 # x: (batch, frames, 42)
        out, _ = self.lstm(x)
        return self.head(out[:, -1])                                      # classify from the last frame's state

# Collect 30 frames per sign, train with cross-entropy and Adam exactly as in the NumPy experiment.

For the static case drop the LSTM and use the normalized 42 numbers in an MLP or gradient-boosted trees. For full-body activities, MediaPipe Pose gives 33 landmarks per frame (so 66 numbers per frame), and the same sequence model works. Useful data sets: for hand signs, the ASL Alphabet image set on Kaggle (tens of thousands of images for the fingerspelled letters) and for whole signs WLASL (a word-level American Sign Language video collection); for activities from video Kinetics and NTU RGB+D (skeleton data), and for activities from phone and wearable sensors UCI HAR (30 volunteers, six activities, accelerometer and gyroscope windows) and WISDM. The sensor version is the same project with 3 or 6 channels per time step instead of keypoints, and a 1D convolutional or LSTM model.

Design notes and pitfalls

  1. Missing keypoints. Hands leave the frame and joints are occluded. Handle missing detections explicitly (zeros plus a mask, or interpolation) and train with random dropout of keypoints so the model learns to cope.
  2. Split by person. Different people move and sign differently. If the same person appears in train and test, you measure memorization. Hold out whole subjects (see the leakage warning in Chest X-ray Pneumonia and Lung Cancer Detection).
  3. Augment geometrically. Small rotations, scalings, mirroring (where it does not change the meaning) and time-stretching of sequences make models more robust.
  4. Windowing. For streaming recognition, classify overlapping windows (for example 1 second, advancing 0.25 seconds) and smooth the predictions.
  5. Sign languages are languages. Fingerspelling is only a small part of a sign language. Real signing uses two hands, facial expression, body posture, location and movement, and has its own grammar that differs from spoken languages. A hand-shape classifier is a learning exercise, not a translator. Projects in this area should involve deaf and hard-of-hearing users and sign language experts from the start.
  6. Privacy. Keypoints are less revealing than video, but they can still identify people by their movement. Process on-device where you can.

Ideas to extend the project

  • Make walking and running harder to separate (similar speeds) and see which model degrades first.
  • Add the angle at each finger joint as features and compare with raw coordinates.
  • Train the LSTM on clips of different lengths by padding and masking.
  • Replace the LSTM with a 1D convolution over time followed by global pooling.
  • Combine both parts: a two-hand signing model whose input is 84 numbers per frame.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does normalizing hand landmarks (wrist at origin, divide by hand size) help so much?

    Position and size are nuisance variables for the sign. Removing them in preprocessing means the network does not have to learn that invariance from data.
  2. 2. Why could a model on summary statistics (mean and standard deviation over the clip) not separate walking from running?

    Walking and running have similar average positions and spreads. Their difference is speed, which only appears in the frame-by-frame sequence.
  3. 3. Which is an advantage of the LSTM over the flattened-sequence MLP shown in the lesson?

    Both reached 100% here. The LSTM reuses the same weights at each frame, so it had 3,677 parameters against 18,821, and it still scored 98.0% on 16-frame clips.
  4. 4. Why should you split an activity data set by subject (person) rather than by clip?

    People have individual movement and signing styles. If they appear in both sets, the model can exploit them and the test score is inflated.
  5. 5. What is the right way to treat a hand-shape classifier for the fingerspelled alphabet?

    Fingerspelling is a small part of sign language. Real recognition needs motion, both hands, face and language-level modeling, ideally developed with the signing community.

Summary

  • Pose estimation reduces video to keypoints: 21 per hand and 33 per body, a small, robust input.
  • Static signs are classified from one frame. Normalizing landmarks (wrist at the origin, scaled by hand size) took accuracy from 45.0% to 99.8% when only 150 training images were available.
  • Activities need sequences. A model on summary statistics could not tell running from walking (33% and 70%), while a flattened MLP and an LSTM both reached 100.0%.
  • The LSTM used 3,677 parameters instead of 18,821 and still scored 98.0% on clips shortened to 16 frames; it suits streaming and variable-length input.
  • Handle missing keypoints, split by subject, augment geometrically and respect that sign languages are full languages.

Next project: Anomaly Detection with Autoencoders.