Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 73 ยท Frameworks and Production

Model Compression

Shrink neural networks with structured pruning, per-channel int8 quantization and knowledge distillation, each demonstrated in runnable NumPy code.

State-of-the-art networks keep growing, but the places they have to run do not: a phone has a few gigabytes of memory and a battery, a browser tab has a time budget, and a server pays for every millisecond. Model compression is the family of techniques that make a trained network smaller and cheaper while keeping most of its accuracy. Three ideas carry most of the weight: pruning (remove parameters that matter little), quantization (store each number with fewer bits) and knowledge distillation (train a small model to imitate a big one).

Deploying and Serving Models introduced pruning and quantization on a digit classifier, including file sizes and the sparsity-versus-accuracy curve. This lesson goes deeper on what that lesson only touched: why structured pruning is the kind that really speeds things up, why per-channel quantization beats a single scale, and how distillation uses a teacher's soft probabilities. All three experiments use one small classifier trained from scratch in NumPy, so you can run everything here.

Three ways to shrink a model

Pruning Delete weights or whole neurons that matter little.
Quantization Use 8 bits (or fewer) per number instead of 32.
Distillation A small student learns from a big teacher.

They combine well: distill, then prune, then quantize.

1 / 2

The setup: a small trained network

The experiments share one task: classify points in 10 dimensions into 3 overlapping classes. The first block builds the data and trains a classifier with one hidden layer of 64 units, using the same forward and backward passes as Backpropagation in NumPy. Later blocks repeat this setup (each TryIt runs on its own), so the helper functions are repeated.

Python A baseline network to compress

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The network has 899 parameters (10 x 64 + 64 + 64 x 3 + 3) and reaches 91% test accuracy, a little below its 94% training accuracy. That 91% is the number every compressed variant below is compared with. (Overlapping classes cap what any model can reach on this data.)

Pruning: unstructured versus structured

Unstructured pruning sets individual small-magnitude weights to zero, wherever they are. It can remove 50 to 90 percent of the weights with little damage, as the previous lesson showed. But the matrix keeps its shape: it is a normal (10, 64) array full of zeros, and ordinary matrix multiplication does exactly the same work. Speed-ups need sparse formats and sparse kernels, and hardware supports only a few patterns (for example, "2 out of every 4 weights are zero" on recent GPUs).

Structured pruning removes whole units: a hidden neuron, a convolution filter or an attention head. Neuron j of a layer owns column j of W1 (its incoming weights), entry j of b1, and row j of W2 (its outgoing weights). Delete all three and you get a genuinely smaller dense network: smaller matrices, fewer multiplications and less memory, on any hardware. The price is that structure is a coarser tool, so accuracy drops sooner for the same number of removed parameters. A common ranking score for a neuron is the L2 norm of its incoming weights (or incoming times outgoing).

Python Unstructured and structured pruning compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without any fine-tuning, zeroing half of the weights in each layer costs about three points (0.910 to 0.878), 80% sparsity drops accuracy to 0.736, and 95% sparsity collapses it to 0.348, roughly chance. Structured pruning is harsher on this tiny network: keeping 32 of 64 neurons halves the parameters (899 to 451) and accuracy falls to 0.823, and keeping 8 neurons leaves 0.463. Accuracy below chance at 4 neurons (0.230) is not a bug: the remaining neurons were never trained to work alone, so the output is distorted. Accuracy is not the whole comparison, though. The unstructured models still occupy 899 slots and need sparse kernels to run faster, while the structured ones are small dense networks that run faster everywhere.

Both numbers improve a lot with fine-tuning after pruning, which these demos skip to keep the effect visible. Over-parameterized networks, such as the large CNNs and transformers used in practice, tolerate far more pruning than this 64-unit toy.

In practice you prune gradually (prune a little, fine-tune, repeat) rather than in one shot, because the remaining weights can compensate during retraining. The lottery ticket hypothesis observes that large networks contain small subnetworks which, trained alone from their original initial weights, can match the full network, which is one reason pruning works so well.

Quantization: per-tensor versus per-channel

Quantization replaces each 32-bit float with a small integer and a shared scale. For symmetric 8-bit quantization of a tensor w:

TEXT
scale = max(|w|) / 127
q = round(w / scale)        (an integer between -127 and 127)
w_approx = q * scale        (dequantized value used in the arithmetic)

The rounding error of each weight is at most half a step, scale / 2. A single scale for the whole tensor (per-tensor) is simple, but one large weight stretches the scale and wastes resolution on every small weight. Per-channel quantization computes one scale per output channel (per column of W1 here, per filter in a CNN), so each channel uses its own range. It costs only a handful of extra numbers and is the default in most toolkits for weights.

Quantization is also the reason lower precision is attractive in hardware: integer arithmetic is cheaper than floating point, and 4 times fewer bytes means 4 times less memory traffic, which is often the real bottleneck.

Python Per-tensor versus per-channel quantization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first table shows the effect of the outlier. With one column scaled up twelve times, the per-tensor scale is set by that column, so the average weight error is 0.0124 at 8 bits against 0.0016 per channel, about 8 times worse, and the gap persists at every bit width (0.2253 versus 0.0306 at 4 bits). The second table uses the real network without an outlier: 8-bit quantization of both layers leaves accuracy unchanged (0.910 and 0.911), and even 4 bits holds (0.912 and 0.909, differences that are noise). The two methods only separate at very low precision: at 2 bits per-tensor falls to 0.693 while per-channel keeps 0.792. So a well-behaved small network needs no cleverness at 8 bits; the tricks matter for outliers (common in large language models) and for aggressive bit widths.

Two refinements matter in practice. Post-training quantization (PTQ) quantizes a finished model, needing only a small calibration set to choose the scales for the activations (the numbers flowing between layers, whose range is not known in advance). Quantization-aware training (QAT) instead inserts "fake quantization" into the forward pass during training: each weight is rounded to its quantized value, but the backward pass pretends the rounding was the identity (the straight-through estimator), so the network learns weights that survive rounding. QAT recovers most of the accuracy lost at very low bit widths such as 4. For large language models, weight-only schemes at 4 bits are routine.

Knowledge distillation

A big model, trained on the same data, tends to be more accurate than a small one, but you often cannot afford to run it. Distillation (Hinton and colleagues, 2015) trains the small student to imitate the big teacher. The surprise is that imitating works better than learning from the original labels alone, because the teacher's output probabilities carry more information than a hard label does. A teacher that outputs [0.80, 0.19, 0.01] for a picture is saying more than "class 0": it says the example looks a bit like class 1 and nothing like class 2. Those "dark knowledge" ratios reveal how classes relate.

Teacher probabilities are often very peaked, so the small ones carry little signal. The fix is a temperature T: divide the logits by T before the softmax. At T = 1 you get the usual probabilities, and at larger T the distribution becomes softer and the relative sizes of the small probabilities become visible. The student is trained to match the teacher's softened distribution with a KL-divergence (or cross-entropy) loss, often mixed with the normal loss on the true labels:

TEXT
loss = alpha * T^2 * KL( softmax(teacher_logits / T) || softmax(student_logits / T) )
     + (1 - alpha) * CrossEntropy( softmax(student_logits), true_label )

The factor T^2 keeps the gradient scale comparable when T changes. Importantly, the teacher can label unlabeled data, so the student is not limited to the few examples that have human labels.

Python Temperature, soft targets and a distilled student

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The temperature table shows the softening for logits [6, 3, 0]: at T = 1 the top class has 0.95 and the third class 0.002, while at T = 4 the three classes get 0.59, 0.28 and 0.13, so the second and third are now visible to a student. The student here has 59 parameters, 15 times fewer than the teacher's 899. Trained on only 40 labeled examples it reaches 0.843 on average over three seeds. Trained on the teacher's probabilities for the 1,960 other inputs (without using their labels) it reaches 0.915 at T = 1 and 0.914 at T = 4, as good as the teacher itself within noise.

Be honest about what this experiment shows. Much of the gain comes from the teacher turning 1,960 unlabeled inputs into training data, which is a real and common advantage of distillation, not from the temperature. On this easy problem T = 1 and T = 4 perform the same, so temperature made no measurable difference. Temperature pays off when the teacher is very confident (probabilities near 0 and 1 carry almost no information beyond the label) and the classes have a rich similarity structure, as in image classification with hundreds of classes. Treat T as a hyperparameter to tune (values between 2 and 5 are typical).

Distillation shows up everywhere: DistilBERT is a smaller BERT trained to imitate the larger one, many on-device speech and vision models are distilled from server-side giants, and LLM "small models" are often trained on text generated by larger ones. Variants distill intermediate features and attention maps as well as outputs.

Combining techniques and a framework note

The techniques stack, and the usual order is: pick or distill a smaller architecture, prune it (structured first if you want real speed), fine-tune, then quantize and export. Always measure on the target hardware: parameter counts and FLOPs are only proxies for latency, and an int8 model on a CPU without fast integer instructions can even run slower. Keep a held-out test set and track accuracy at every step, since each technique has its own failure mode.

The frameworks provide these tools, shown here read-only because PyTorch is not available in the in-browser runner:

Python
import torch
import torch.nn as nn
import torch.nn.utils.prune as prune
import torch.nn.functional as F

# 1. Pruning: zero 30% of weights by magnitude, or remove 25% of neurons by L2 norm
prune.l1_unstructured(model.fc1, name="weight", amount=0.3)
prune.ln_structured(model.fc2, name="weight", amount=0.25, n=2, dim=0)
prune.remove(model.fc1, "weight")            # make the pruning permanent

# 2. Dynamic int8 quantization of the linear layers (post-training, CPU)
quantized = torch.ao.quantization.quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)

# 3. Distillation loss: soft teacher targets plus the normal label loss
def distill_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.7):
    soft = F.kl_div(F.log_softmax(student_logits / T, dim=1),
                    F.softmax(teacher_logits / T, dim=1), reduction="batchmean") * T * T
    return alpha * soft + (1 - alpha) * F.cross_entropy(student_logits, labels)

Keras and TensorFlow offer the same ideas through the TensorFlow Model Optimization toolkit and TFLite converters, and ONNX Runtime has its own quantization tools (see Deploying and Serving Models for export).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does structured pruning usually give a real speed-up while unstructured pruning often does not?

    Zeros scattered in a dense matrix still get multiplied unless sparse kernels exist. Deleting entire units yields genuinely smaller dense matrices that any hardware runs faster.
  2. 2. Why is per-channel weight quantization usually more accurate than per-tensor quantization?

    With one scale for the whole tensor, a single large weight sets the step size for everyone. Per-channel scales keep small channels precise.
  3. 3. In knowledge distillation, what does raising the temperature T do to the teacher's output?

    Dividing logits by T before the softmax flattens the distribution without changing which class is largest. The student can then learn from how the teacher ranks the wrong classes.
  4. 4. What is the straight-through estimator used for in quantization-aware training?

    Rounding has zero gradient almost everywhere. Passing the gradient straight through lets the weights adapt to the quantized forward pass.

Summary

  • Compression makes models smaller and cheaper; the main tools are pruning, quantization and distillation, and they combine.
  • Unstructured pruning zeroes individual weights and needs sparse support to speed anything up; structured pruning removes whole neurons, filters or heads and yields genuinely smaller dense models.
  • Quantization stores weights as low-bit integers plus scales; per-channel scales beat a single per-tensor scale, especially with outlier channels, and quantization-aware training recovers accuracy at very low bit widths.
  • Distillation trains a small student on a teacher's softened probabilities (temperature T, loss scaled by T squared), and can use unlabeled inputs.
  • Always fine-tune after pruning, calibrate activations for quantization, and measure latency on the real target hardware.

Next: Responsible Deep Learning.