Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 73 ยท First Steps

What Is Deep Learning?

Learn what deep learning is, how it differs from machine learning, what a neural network learns, and when it wins, with runnable NumPy examples.

Deep learning is a way of building programs that learn their behavior from examples instead of from hand-written rules. You show a model thousands of pictures labelled "cat" or "dog", and it adjusts millions of internal numbers until it labels new pictures correctly. The "deep" part refers to the shape of the model: many simple layers stacked on top of each other, each one transforming the output of the one before.

This tutorial builds everything from the ground up. You will write neurons, layers, backpropagation, optimizers, convolutions and even a tiny transformer in plain NumPy, running live in your browser. Once you have built the machinery yourself, frameworks like PyTorch and TensorFlow stop being magic: they are the same ideas, faster and with more conveniences.

AI, machine learning and deep learning

The three terms are nested, like Russian dolls.

Three nested ideas

Artificial intelligence Any technique that makes machines act smart.
Machine learning Learn the rules from data instead of writing them.
Deep learning Machine learning with many-layered neural networks.

Every deep learning system is machine learning; not all machine learning is deep.

1 / 2

In classic machine learning (for example the models in scikit-learn), a person decides which measurements matter: the area of a house, the number of bedrooms, the average color of an image. The algorithm only finds how to combine them. In deep learning, the first layers of the network learn useful features from the raw input, the next layers combine those into bigger features, and so on. An image network might discover edges first, then corners and textures, then eyes and wheels, then whole objects.

What a neural network actually is

A neural network is a function with adjustable numbers inside it. You give it an input (a list of numbers) and it returns an output (more numbers). The adjustable numbers are called parameters: weights and biases. At the start they are random, so the output is nonsense. Training means nudging the parameters, again and again, so the outputs get closer to the right answers.

That is the whole story, and it fits in one picture.

flowchart LR
    X["Input batch"] --> F["Forward pass:<br/>predict"]
    F --> L["Loss:<br/>how wrong?"]
    L --> B["Backward pass:<br/>gradients"]
    B --> U["Update<br/>parameters"]
    U --> F

The training loop: predict, measure the error, compute which way to change each parameter, change it, repeat.

Each step has its own lesson later:

  1. Forward pass: push inputs through the layers to get predictions (Layers and the Forward Pass).
  2. Loss: one number that says how wrong the predictions are (Loss Functions).
  3. Backward pass: use calculus to find how much each parameter contributed to the error (Backpropagation).
  4. Update: move every parameter a small step in the direction that lowers the loss (Gradient Descent).

Here is that loop at its smallest: a model with exactly two parameters, w and b, learning the rule y = 2x + 1 from examples. Run it and watch the loss shrink and the parameters approach 2 and 1.

Python The training loop with two parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real networks have millions or billions of parameters instead of two, and the loop is the same one. The rest of this tutorial is about doing the four steps for bigger, cleverer functions.

Why depth helps

A single layer can only draw straight lines through data. Many real problems are not separable by a straight line, and stacking layers with a non-linear step between them lets the model bend its decision boundaries. You can see this immediately on a "two moons" dataset, which no straight line can split.

The next example trains a tiny network (one hidden layer of 16 units) next to a logistic regression, which is a single straight-line model. Do not worry about every line yet; each piece gets its own lesson. Focus on the result.

Python A straight line versus a small network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The straight-line model scores 0.84 and the network 0.98 (these are training accuracies, enough for a first look). The straight line cuts the moons in half and misclassifies the points near the tips. The network curves its boundary around them. Nobody told it what a moon looks like; the loop above found the parameters on its own.

When deep learning wins, and when it does not

Deep learning is not always the right tool. A good rule of thumb:

  • It wins when you have a lot of data and the input is raw and unstructured: images, audio, video, free text, protein sequences. It also wins when the right features are hard to write by hand.
  • It loses or ties on small tabular datasets (a few thousand rows of numbers and categories). There, gradient-boosted trees or even plain linear models are usually as accurate, faster to train and easier to explain.
  • It costs more: more data, more compute, more tuning, and results that are harder to interpret.

Start with the simplest model that could work, such as a linear model or a decision tree, and measure it. Reach for a neural network when that baseline is clearly not good enough, or when your input is images, sound or text.

The tools of the trade

Most production deep learning is written in PyTorch (very popular in research and increasingly in industry) or TensorFlow with Keras (strong tooling for deployment). Both give you three things this site cannot run in your browser: tensors that live on a GPU, automatic differentiation, and a large library of ready-made layers. They look like this, and you will see them again in PyTorch Essentials:

Python
# Read-only: PyTorch is not available in the browser runner
import torch
from torch import nn

model = nn.Sequential(nn.Linear(2, 16), nn.Tanh(), nn.Linear(16, 1))
optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
loss_fn = nn.BCEWithLogitsLoss()

for step in range(2000):
    loss = loss_fn(model(X), y)
    optimizer.zero_grad()
    loss.backward()        # the backward pass, automatic
    optimizer.step()       # the update

This is the same loop you just ran by hand. We build the networks from scratch in NumPy first, because understanding what loss.backward() does is what separates people who can debug training from people who copy snippets.

What you will build

By the end you will have written, in NumPy:

  • single neurons, layers and a full network with backpropagation;
  • optimizers (momentum, Adam), regularization, dropout and batch normalization;
  • a convolutional network that reads handwritten digits;
  • recurrent networks, attention and a tiny transformer language model;
  • autoencoders and small generative models;
  • a mini deep-learning framework with automatic differentiation.

The only prerequisite is comfortable Python and a little NumPy. If arrays still feel new, skim NumPy first arrays and vectorization first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes a neural network 'deep'?

    Depth refers to the number of stacked layers. Training time, data size and hardware are separate matters.
  2. 2. What are the four steps of the basic training loop?

    Predict, measure the error, compute gradients, then update the parameters. Then repeat.
  3. 3. Which task is the weakest fit for deep learning?

    On small tabular data, simpler models such as gradient-boosted trees are usually as accurate and much cheaper. Deep learning shines on large, raw, unstructured inputs.
  4. 4. In the two-parameter example, what does the learning rate control?

    Each parameter moves by learning rate times its gradient. Too small is slow; too large can overshoot.

Summary

  • Deep learning is machine learning with many-layered neural networks that learn their own features from raw data.
  • A network is a function with adjustable parameters; training adjusts them to reduce a loss.
  • The training loop is: forward pass, loss, backward pass, update, repeat.
  • Stacked layers with non-linearities can bend decision boundaries that a straight line cannot.
  • Deep learning shines on large unstructured data and is often unnecessary for small tabular problems.
  • PyTorch and TensorFlow automate the same loop you will build by hand in NumPy.

Next: The Math You Need.