Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 46 ยท Images as Data

What Is Computer Vision?

Learn what computer vision is, the main tasks (classification, detection, segmentation, tracking), classical versus deep learning pipelines, and how this course works.

Computer vision is the field that teaches machines to get useful information out of images and video. A phone unlocking when it sees your face, a car noticing a pedestrian, a factory camera rejecting a cracked bottle, a scanner turning a crumpled receipt into text: all of them start with the same raw material, a grid of numbers, and end with a decision or a measurement.

For a person, seeing feels effortless. For a program, a photo is just millions of brightness values with no built-in meaning. The whole field is about closing that gap: from numbers to "there is a dog in the lower left, 40 pixels wide". In this lesson you will meet the main tasks, see the two big families of methods, and learn how the rest of this course is organized.

The main tasks

Most vision problems fall into a handful of families. They differ in what the answer looks like.

Task Question it answers Output
Classification What is in this image? One label (or a list of probabilities)
Detection What is where? A list of boxes, each with a label and a score
Segmentation Which pixels belong to what? A label for every pixel (a mask)
Tracking Where did this object go? The same object's position across video frames
Measurement How big, how far, how many? Numbers: sizes, counts, distances
Matching and geometry Is this the same scene? How did the camera move? Correspondences, transforms

The tasks build on one another. A detector usually contains a classifier, a tracker needs a detector (or at least a way to find the object in each frame), and segmentation is classification done for every pixel. Measurement tasks, such as counting coins or checking whether a part is the right size, often use plain image processing and no learning at all.

flowchart LR
    A["Image"] --> B["Classification<br/>'cat'"]
    A --> C["Detection<br/>box + 'cat'"]
    A --> D["Segmentation<br/>mask of cat pixels"]
    C --> E["Tracking<br/>same cat, next frames"]

The same input can be answered at different levels of detail, from one label to a label per pixel.

Why vision is hard

Humans handle changes without noticing. A computer sees every change as different numbers. The same mug photographed in bright sun, in shade, from above, half hidden behind a book, or with a slightly different camera gives completely different arrays. A good vision system has to ignore the changes that do not matter (lighting, position, scale, small amounts of noise) and keep the ones that do (the shape and appearance of the object).

Typical sources of trouble:

  • Illumination: shadows and highlights change pixel values dramatically.
  • Viewpoint and scale: an object can be close or far, rotated or tilted.
  • Occlusion: parts of an object may be hidden.
  • Clutter and variation: backgrounds are messy, and two chairs can look nothing alike.
  • Noise and blur: sensors are imperfect, and cameras move.

Every technique in this course is a different answer to one of these problems.

Classical versus deep learning pipelines

For decades, vision systems were built by hand. An engineer designed a sequence of steps, each with a clear mathematical meaning: smooth the image, find edges, group pixels, measure shapes, and apply rules. In the last decade, deep learning replaced many hand-made steps with neural networks that learn their own features from large sets of labeled examples.

flowchart TB
    subgraph Classical
      direction LR
      C1["Pixels"] --> C2["Filtering<br/>and cleanup"] --> C3["Hand-made features<br/>edges, corners, HOG"] --> C4["Rules or a<br/>simple classifier"]
    end
    subgraph Deep
      direction LR
      D1["Pixels"] --> D2["Convolutional or<br/>transformer network<br/>(learned features)"] --> D3["Prediction"]
    end

Classical pipelines hand-design each stage; deep networks learn the middle stages from data.

Neither family is "old" or "new" in a way that makes the other useless:

  • Classical methods need no training data, run fast on tiny devices, are easy to debug, and give exact, explainable measurements. They are still the right answer for controlled settings such as a fixed camera over a conveyor belt, and they prepare data for learned models.
  • Deep learning wins when the world is messy and the rules are hard to write down: recognizing thousands of object types in arbitrary photos, faces, medical scans, road scenes.

Even if you plan to use deep learning every day, the classical ideas explain what networks are doing. A convolutional layer is a learned version of the filters you will build by hand in this course, and the metrics for judging models are the same ones used for old-school pipelines.

The steps of a vision pipeline

Whatever the technique, almost every vision system moves through the same stages:

  1. Image acquisition. A camera, scanner, microscope, depth sensor, satellite or video file produces pixel data. Lighting, lens, exposure and resolution decide how much the later steps have to fix, so many projects are won or lost here.
  2. Pre-processing. Resize, normalize brightness, correct colour, reduce noise, correct lens distortion.
  3. Feature extraction or learning. Edges, corners and descriptors (classical), or learned features from a network.
  4. Analysis and decision. Classify, detect, segment, measure or track.
  5. Post-processing and aggregation. Suppress duplicate detections, smooth results over video frames, combine results from several images or tiles.
  6. Visualization and interpretation. Draw boxes and masks, produce reports or trigger an action, and check the output against what a human would expect.

Every later chapter of this course sits at one of these steps. Keep in mind the common limits as well: vision systems need representative data and good light, can fail on situations they never saw (a snowy road, a rare skin type), cost compute, and are hard to interpret.

The software landscape

You do not have to write everything from scratch. The usual Python tools (read-only here, since the in-browser runner offers NumPy and SciPy only):

Library What it is good for
OpenCV (cv2) The workhorse: reading and writing images and video, filters, edges, features, camera calibration, classical detectors, a small deep learning module
Pillow (PIL) Simple loading, saving, resizing, cropping and format conversion
scikit-image Clean, well-documented classical algorithms with a NumPy interface
NumPy and SciPy The array and filtering foundation that all the others use
PyTorch and torchvision Research favourite for training networks; pretrained classification, detection and segmentation models
TensorFlow and Keras Production-oriented training and deployment, mobile and web exports
Albumentations and Kornia Data augmentation and differentiable image operations

A typical project loads images with OpenCV or Pillow, does the numeric work in NumPy, and trains or runs a model in PyTorch. Older or smaller frameworks (SimpleCV, MXNet) appear in legacy code but are rarely chosen for new work.

How this course works

You do not need a camera, image files or a GPU. Every example in this course runs in your browser, with Python and NumPy, on tiny synthetic images that we generate in code: gradients, shapes, noisy blobs, and the small handwritten-digit pictures that ship with scikit-learn. Small images have two advantages. You can print them and see every number, and the results run instantly.

The tools we use:

  • NumPy holds images as arrays. If arrays are new to you, read Your First Arrays and Indexing and Slicing first.
  • matplotlib draws arrays as pictures (see Arrays as Images and Plots).
  • SciPy supplies filters, labeling and geometric transforms in scipy.ndimage.
  • scikit-learn supplies classical machine learning for the later chapters.

Everything else we write ourselves, from scratch, so nothing is a black box. Production libraries such as OpenCV and scikit-image do the same jobs faster; once you understand the from-scratch versions, switching to them is easy. If your Python is rusty, review the Python tutorial alongside.

The course follows the order in which the ideas build:

  1. Images as data: arrays, color, resolution and noise.
  2. Pixel operations: brightness, histograms, thresholds and geometry.
  3. Filtering and edges: convolution, blur, gradients and morphology.
  4. Features and matching: scale, corners, descriptors, Hough lines.
  5. Segmentation and shapes: labeling, measuring, clustering, templates.
  6. Classical machine learning: HOG features, classifiers and PCA.
  7. Neural networks for vision: convolution layers, training, pretrained models.
  8. Detection, segmentation and video.
  9. Evaluating and shipping vision systems.
  10. Projects and practice.

A first taste: seeing with arithmetic

Here is a complete, if tiny, vision program. We create an 8 by 8 "image" that contains a bright square on a dark background, add a little noise, and answer a question about it: where is the object, and how big is it?

Python Find the bright object

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what happened. A threshold turned numbers into a yes/no decision for each pixel, and a few lines of NumPy turned that mask into a size and a position. No learning was involved. That is the classical way of seeing: a chain of small, understandable steps. The threshold of 110 is a hand-picked number, which is also the weakness of this style; Thresholding later shows how to choose it automatically.

A first taste of learning

Now compare with the second family. Instead of writing a rule, we show the computer examples with labels and let it find the rule. Here we use scikit-learn's tiny 8 by 8 digit images and a simple classifier.

Python Learning a rule from examples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The classifier never received a rule such as "a zero has a hole in the middle". It saw about 1,350 labeled pictures and found a pattern that works on pictures it has never seen. We will return to digits in Classifying Handwritten Digits, and later chapters show how neural networks learn the features too.

Note: Computer vision involves real choices about who is in the data and how the results are used. Face recognition, surveillance and automated decisions deserve care. The chapter Pitfalls, Ethics and Deployment comes back to this.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A system must output a label for every single pixel of an image (road, car, sky, and so on). Which task is this?

    Segmentation assigns a label to each pixel, producing a mask. Classification gives one label per image, detection gives boxes, and tracking follows objects over time.
  2. 2. Which is an advantage of classical, hand-designed vision methods?

    Classical pipelines are built from understandable steps and need no training set, but they often have hand-chosen parameters and struggle with messy real-world variety.
  3. 3. Why does this course use tiny synthetic images?

    Real images are also arrays, but tiny generated ones let you inspect each value and keep every example fast and reproducible.
  4. 4. In the bright-object example, what does img > 110 produce?

    Comparing an array with a number gives an array of booleans, the same shape as the image. That mask marks the pixels we decided are part of the object.

Summary

  • Computer vision turns pixel grids into decisions and measurements: classification, detection, segmentation, tracking, matching and measurement.
  • Vision is hard because lighting, viewpoint, scale, occlusion, clutter and noise change the numbers without changing the meaning.
  • Classical pipelines chain hand-designed steps (filtering, features, rules); deep learning learns features from labeled data. Both are valuable, and classical ideas explain what networks do.
  • In this course every example is NumPy-based and runs in your browser on tiny generated images, with all key algorithms written from scratch.
  • A threshold plus array operations is already a small vision program: mask, count, locate.

Next, see how pictures are stored in memory: Images as Arrays.