Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 73 ยท Convolutional Networks

Images as Tensors

See how images become arrays: pixels, channels, the (N, C, H, W) layout, normalization, the scikit-learn digits dataset, and flattening versus structure.

A neural network only understands numbers, so before it can look at a photograph, the photograph has to become an array. Luckily that is exactly what a digital image already is: a grid of numbers. The skill to learn here is not the conversion itself but the bookkeeping: which axis is height, which is width, which is color, which is the batch, and how to scale the values so a network can learn from them. Most shape errors in vision code come from getting this wrong.

This lesson introduces the vocabulary and the conventions used by almost every deep-learning library, with the small handwritten-digit images that ship with scikit-learn. The following lessons build on them: The Convolution Operation, Convolution, Pooling and Feature Maps and Training a CNN From Scratch. If NumPy shapes feel unfamiliar, revisit reshaping first.

An image is a stack of grids

Pixel One number (or a few) for one tiny square.
Channel One grid per color component, such as red, green, blue.
Height and width The size of every grid.
Batch Many images stacked along one more axis.

A batch of color images is a 4-dimensional array.

1 / 2

Pixels and grayscale images

A grayscale image is a 2-D array of shape (height, width). Each entry is a pixel intensity: how bright that spot is. In files, intensities are usually 8-bit integers from 0 (black) to 255 (white). The digits dataset used here comes from scikit-learn's load_digits: 1,797 images, each only 8 by 8 pixels, with intensities from 0 to 16, and a label from 0 to 9. They are tiny enough to run in your browser and still real enough to learn from.

The first example loads the data, inspects its shapes and types, prints one image as the raw numbers a network would see, and draws a grid of examples.

Python Digits: shapes, values and a picture grid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

digits.images has shape (1797, 8, 8): the first axis counts examples, the other two are height and width. digits.data holds the same numbers with each image flattened into a row of 64 values. Both views are useful, and moving between them with reshape costs nothing because the numbers are not copied.

Channels: color images

A color image needs three numbers per pixel: how much red, green and blue. So it has a third axis, the channel axis. Two conventions exist:

  • Channels last, (H, W, C): how image files are read, how matplotlib draws images, and the default in TensorFlow and Keras.
  • Channels first, (C, H, W): the default in PyTorch, and the one we use for the rest of this tutorial because it makes "one grid per channel" literal.

Converting between them is a transpose: img.transpose(2, 0, 1) goes from HWC to CHW, and transpose(1, 2, 0) goes back. Mixing the two up is a classic source of shape errors and strangely scrambled pictures.

There are no photographs to load here, so the next example synthesizes a small color image from arrays: a red-to-blue gradient, a green disk, and then it shows each channel on its own, converts the layout, and computes a grayscale version as a weighted sum of the channels (the weights reflect how bright each color looks to the human eye).

Python Building a color image and splitting its channels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what each channel shows by itself: the red grid is a gradient, the green one is only the disk, the blue one is the reverse gradient. A convolutional network treats these grids as separate input "feature maps" and learns how to combine them.

Batches and the (N, C, H, W) shape

Networks are trained on many images at once, so there is one more axis in front: the batch. The standard shape is (N, C, H, W):

Letter Meaning Digits example
N number of images in the batch 1,797 (or 64 per mini-batch)
C channels 1 (grayscale)
H height in pixels 8
W width in pixels 8

Grayscale images have one channel, but the axis must still exist: a convolution layer expects (N, C, H, W), not (N, H, W). Adding it is a reshape or np.newaxis. Pay attention to ranks: (8, 8) is one image, (1, 8, 8) is one image with a channel axis, and (1, 1, 8, 8) is a batch containing that one image.

Normalizing pixel values

Raw intensities (0 to 255, or 0 to 16) are poor inputs for a network. Large inputs push neurons into saturation and make gradients erratic (see Weight Initialization). Two standard steps:

  1. Scale to the range 0 to 1 by dividing by the maximum (255 for typical image files, 16 for these digits).
  2. Standardize: subtract the mean and divide by the standard deviation, so the data is centered with unit spread. For color images, compute the mean and standard deviation per channel, since red, green and blue are distributed differently.

The statistics must come from the training set only and then be reused unchanged for validation and test data. Computing them on all the data lets information from the test set leak into training.

Python Scaling, standardizing and the data-leak rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The standardized training set has mean 0 and standard deviation 1 by construction, while the test set lands close to those values but not exactly on them, because it is measured with the training statistics. That small difference is the honest picture of how new data behaves.

Flattening versus keeping the structure

The simplest way to feed an image to the networks you have built so far is to flatten it: unroll the 8 by 8 grid into 64 numbers and treat them as 64 unrelated features, like the columns of a spreadsheet. This works for tiny images, but it throws information away: a dense layer has no idea which pixels were neighbors. Two facts make this concrete.

First, order does not matter to a dense layer. If you permute the 64 columns the same way for every image, a fully connected model learns just as well, because it never used the positions. A human can no longer recognize the picture, but the model cannot tell the difference. That is a sign the model is ignoring spatial structure.

Second, images have strong local patterns: neighboring pixels are strongly correlated, and a shifted digit is still the same digit. A flattened vector changes completely when the digit moves one pixel. Convolutional networks keep the grid and share one small filter across all positions, which is the subject of the next lessons.

Python What flattening throws away

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A logistic-regression model scores the same 0.907 on the shuffled pixels as on the original, which shows it never used the layout. Horizontal neighbors correlate at about 0.39, while unrelated pixels sit near 0.00: the structure is real, and flattening ignores it. And a one-pixel shift moves the flat vector (distance about 51) almost as far as swapping in a completely different digit (about 60).

Scale makes this worse as images grow. A dense layer connecting a 224 by 224 color image (150,528 inputs) to just 1,000 hidden units needs about 150 million weights. A convolutional layer with 64 filters of size 3 by 3 on the same image needs only 1,792 parameters (64 filters times 27 weights, plus 64 biases), because the same small filter slides over every position. Fewer parameters, built-in respect for neighborhoods and robustness to shifts are the reasons convolutional networks dominate vision.

When you debug a vision model, print the shape after every step and plot one image from the batch at the point where it enters the network. Wrong axis order, a missing channel axis and a missing normalization show up immediately.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the shape (32, 3, 224, 224) describe in PyTorch convention?

    The layout is (N, C, H, W): batch size 32, 3 channels (RGB), height 224, width 224.
  2. 2. An image has shape (64, 64, 3) as read from a file. How do you convert it to channels-first?

    transpose(2, 0, 1) moves the channel axis to the front without scrambling pixels. A plain reshape would mix pixel values across channels.
  3. 3. Where should the mean and standard deviation used for standardizing images be computed?

    Using only training statistics prevents information about the test data from leaking into training. The same numbers are then applied to validation and test data.
  4. 4. Why does a fully connected network on flattened images ignore spatial structure?

    A dense layer has a separate weight for every input position and no notion of neighbors, so a fixed permutation of the pixels does not affect its ability to fit.

Summary

  • An image is an array: (H, W) for grayscale, with a channel axis for color; a batch adds a leading N.
  • Libraries differ: channels last (H, W, C) for files, matplotlib and Keras, channels first (C, H, W) for PyTorch; convert with transpose.
  • Grayscale images still need a channel axis of size 1 before a convolution layer.
  • Scale pixel values to a small range and standardize with per-channel statistics from the training set only.
  • Flattening discards neighborhood information and explodes the parameter count; convolutions keep the grid and share small filters.

Next: The Convolution Operation.