Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 46 ยท Detection, Segmentation and Video

Super-Resolution and Image Colorization

Learn how to enlarge and restore images: interpolation, PSNR, learned super-resolution from SRCNN to ESRGAN, pixel shuffle, and colorization in Lab and YCbCr, with NumPy code.

Two classic restoration tasks look like magic the first time you see them. Super-resolution (SR) takes a small, blurry picture and produces a larger, sharper one. Colorization takes a black-and-white photograph and adds plausible colour. Both are ill-posed: many different high-quality images shrink to the same low-quality input, and many colourings produce the same grayscale image. Classical methods can only interpolate or copy what is there. Deep learning can imagine the missing detail from experience, which is powerful and, as you will see, risky.

This lesson explains the problem and the main model families, measuring quality with PSNR on real arrays, and showing the two tricks that make the networks work: sub-pixel convolution for upsampling and luminance/chrominance splitting for colour. The colour-space basics are in Color Channels and Color Spaces; sampling theory is in Sampling and Resolution.

The super-resolution problem

Shrinking an image to a quarter of its size throws information away. Running the process backward means inventing the missing pixels. The simplest approach, interpolation, fills them from neighbours (nearest, bilinear or bicubic). It never adds detail, so edges look soft or blocky.

Traditional (non-deep) alternatives have limits too: example-based methods match patches against a dictionary of high/low resolution pairs, and multi-frame methods combine many shifted shots of the same scene. Single-image SR with a trained network is far better at textures and edges.

We measure how close a result is to the true image with PSNR (peak signal-to-noise ratio), computed from the mean squared error: PSNR = 10 * log10(MAX^2 / MSE) decibels. Higher is better; 30 dB is acceptable and 40 dB is very good. PSNR rewards pixel-wise closeness, not good looks, so SSIM (structural similarity) and perceptual metrics are often reported alongside.

Python Interpolation baselines and PSNR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three methods land at about 22 dB (bilinear happens to score best on this image, a reminder that PSNR is a rough guide), and none of them recovers the fine stripes or the sharp edge. That missing detail is what learning adds.

Learned super-resolution

The idea (SRCNN, 2014): train a convolutional network on pairs of (low-resolution, high-resolution) images made by downsampling real photographs. The network learns the typical relationship between blurry and sharp patches. Main model families:

Model Idea
SRCNN First CNN for SR: upsample with bicubic first (pre-upsampling), then 3 convolution layers sharpen it
FSRCNN Works on the small image and upsamples at the end with a learned deconvolution; much faster
VDSR Very deep (20 layers) with a residual output: it learns only the missing detail, high = bicubic(low) + network(low)
ESPCN, EDSR, MDSR Post-upsampling with sub-pixel convolution; EDSR removes batch-norm and uses deep residual blocks (MDSR handles several scale factors in one model)
SRGAN, ESRGAN Add a generative adversarial loss: a discriminator pushes results to look like real photos, producing sharper textures
flowchart LR
    A["Low-res image"] --> B["Pre-upsampling:<br/>bicubic first,<br/>CNN at high resolution"]
    A --> C["Post-upsampling:<br/>CNN at low resolution,<br/>upsample at the end"]
    B --> D["High-res output"]
    C --> D
    A --> E["Generative: network + adversarial loss"]
    E --> D

Pre-upsampling is simple but slow because all convolutions run on the large image; post-upsampling runs most of the network on the small image.

Two loss ideas explain the difference in look. A pixel-wise loss (MSE) gives the highest PSNR but averages all plausible sharp images, which looks smooth. A perceptual loss compares features from a pretrained classification network, and an adversarial loss rewards realism, giving crisp but partly invented texture. A model trained for PSNR and one trained for looks are really two different tools.

Sub-pixel convolution (pixel shuffle)

How does a network upsample cheaply? Sub-pixel convolution (ESPCN) makes a convolution output r^2 channels at low resolution, then rearranges them into an r times larger image: the r^2 values at one low-res position become an r by r block of high-res pixels. No interpolation, no checkerboard artefacts, and the learned filters decide each sub-pixel.

Python Pixel shuffle: channels become pixels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four feature maps of 2 by 3 become one 4 by 6 image, and the 2 by 2 block at the top left holds, in order, the four channel values taken at position (0, 0). It is only a reshape, so it is free to compute; the learning happens in the convolution before it.

Training data and evaluation

SR models are trained on synthetic pairs: take HR images, downsample (usually bicubic) to get LR. Real-world degradations (camera noise, JPEG compression, unknown blur) differ, so "real" SR models use randomised degradation pipelines. Standard benchmarks include Set5, Set14, BSD100 and DIV2K, reporting PSNR and SSIM at scales 2x, 3x, 4x. Practical caveats:

  • Hallucination: a GAN can add convincing but false detail (a wrong letter on a licence plate, a different tooth). Never use SR output as evidence of what was really there.
  • Artefacts: ringing, repeated textures, over-smoothed faces.
  • Cost: large models are slow; mobile uses small post-upsampling networks.

Colorization

A grayscale photograph has the luminance of the scene but no colour. In the Lab colour space the image splits cleanly into L (lightness) and two colour channels a and b. Colorization is therefore a prediction problem: input L, predict a and b, then combine and convert back to RGB. Keeping the original L guarantees the structure of the photo is untouched. The related YCbCr space (Y luminance, Cb and Cr chroma differences) works the same way.

Colour is multimodal: a shirt can be red, blue or green and still have the same brightness. A network trained with plain MSE tends to predict the average colour, which is dull and brownish. Better models predict a distribution over quantised colour bins (a classification problem) and use class rebalancing to favour vivid colours, or use GANs.

The toy colorizer below learns one thing from a colour training image: the average chroma for every brightness level. It then colours a different grayscale image. Because it has no concept of objects, it can only guess from brightness, which makes its mistakes easy to read.

Python A toy colorizer in YCbCr

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result keeps the test image's own brightness, so the structure is exactly right, and it recovers approximately the right hues: error drops well below the plain gray version. The surprise shows the weakness. The green sign is far closer in brightness to the sky than to the grass, so the model paints it bluish: with no concept of objects, brightness is all it can use. A deep colorizer sees context (this is a sign, that is sky), which is why real systems rely on large networks and datasets, and even they must guess when the colour is truly ambiguous.

Where these techniques are used

  • Restoring old photographs and film, upscaling video, medical and satellite imagery (more detail from the same sensor), video call streams (send small, enlarge at the receiver), phone cameras (multi-frame SR for zoom).
  • Colorization for historical archives, as an assistive tool: remember that the colours are plausible, not true.
  • Both tasks appear as stages in photo-enhancement pipelines together with denoising and deblurring (see Blur and Denoise).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is super-resolution called ill-posed?

    Downsampling loses information, so recovering the original is ambiguous and the network has to choose a plausible solution.
  2. 2. What does a residual SR network such as VDSR learn?

    Predicting the difference between the interpolated image and the target is easier than predicting the whole image.
  3. 3. What does sub-pixel convolution (pixel shuffle) do?

    The convolution predicts r^2 values per low-res position, and the shuffle turns them into an r by r block of pixels.
  4. 4. Why do colorization models predict Lab's a and b channels while keeping L?

    The grayscale image is the luminance, so only the colour channels need to be predicted, and the original detail is preserved.
  5. 5. What is a key risk of GAN-based super-resolution?

    Adversarial training rewards realism, so the output may contain invented textures that look right but are wrong.

Summary

  • Super-resolution and colorization are ill-posed restoration tasks that deep networks solve by using experience to fill in missing information.
  • Interpolation (nearest, bilinear, bicubic) is the baseline; PSNR and SSIM measure closeness, though PSNR does not capture visual quality.
  • SRCNN pre-upsamples, FSRCNN and ESPCN upsample at the end with learned layers, VDSR and EDSR use residual learning, SRGAN and ESRGAN add adversarial and perceptual losses.
  • Pixel shuffle turns r^2 channels into an r-times larger image without interpolation.
  • Colorization predicts the chroma channels of Lab or YCbCr from the luminance; colour is multimodal, so naive MSE gives dull results.
  • Learned restoration can hallucinate: never treat enhanced detail as evidence.

Next: Satellite and Remote Sensing Imagery.