Skip to content
Tutorials Center
My learning

Topics

Lesson 115 of 149 ยท Data and the Web

Neural Networks: TensorFlow and PyTorch at a Glance

Understand neural networks in Python. Build and train a tiny network with NumPy, see how PyTorch and Keras code looks, and learn when deep learning is the right tool.

Image recognition, speech-to-text, translation and chatbots all run on neural networks, the models behind "deep learning". They sound mysterious, but the core idea fits in a few lines of NumPy: multiply inputs by weights, add them up, squash the result, repeat in layers, and nudge the weights a little after every mistake. Do that millions of times on lots of data and the network learns surprisingly complex patterns.

In this lesson you'll build that idea from scratch: a single artificial neuron, then a tiny two-layer network that learns a problem no straight line can solve, trained by hand-written gradient descent. Then you'll see what the same network looks like in PyTorch and Keras (TensorFlow), the two frameworks professionals use, and learn when deep learning is worth it and when the scikit-learn models from the previous lesson are the better choice.

Note: PyTorch and TensorFlow are large native packages that don't run in the browser editors here, so their code on this page is read-only. Everything you can run is plain NumPy (plus one scikit-learn example), and it shows exactly the same mechanics, just at a smaller scale.

A single neuron

An artificial neuron takes several numbers as input and produces one number:

  1. Multiply each input by its own weight and add the results.
  2. Add a bias, a number that shifts the result up or down.
  3. Pass the sum through an activation function, which squashes or bends it.

With NumPy, steps 1 and 2 are a single dot product. The weights and the bias are the neuron's parameters: the numbers that training will adjust. Here's a hand-made neuron that estimates whether a student passes from hours studied and hours slept:

Python One neuron, by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The sigmoid function turns any number into a value between 0 and 1, so the output reads like a probability. A single neuron with a sigmoid is exactly logistic regression. The difference with a neural network is what happens next: you stack many neurons into layers.

Activation functions

Without activation functions, stacking layers would be pointless: a linear function of a linear function is still linear, so ten layers would be no more powerful than one. The activation adds the bend that lets a network model curves. Three are worth knowing:

  • ReLU, max(0, z): the default for hidden layers in modern networks. It's cheap to compute and trains well.
  • tanh: an S-curve from -1 to 1, common in small networks.
  • sigmoid: an S-curve from 0 to 1, used on the output when you want a probability for a yes/no answer. (For several classes, the softmax function turns a list of scores into probabilities that add up to 1.)
Python ReLU, tanh and sigmoid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Layers: from one neuron to a network

A layer is a group of neurons that all read the same inputs. If a layer has 4 neurons and receives 2 inputs, its weights form a 2ร—4 matrix, and one matrix multiplication computes all 4 neurons for all samples at once. That's why NumPy (and GPUs, which are very good at matrix math) fit neural networks so well.

A typical network has:

  • an input layer, which is just your features;
  • one or more hidden layers, which learn useful in-between representations;
  • an output layer, sized for the answer: one neuron for a yes/no probability or a number, one per class for classification.

"Deep" learning simply means many hidden layers. Running inputs through the layers to get a prediction is called the forward pass.

How a network learns

Training repeats four steps, over and over:

  1. Forward pass: compute the predictions with the current weights.
  2. Loss: measure how wrong they are with a loss function, such as mean squared error for numbers or cross-entropy for classes.
  3. Backward pass (backpropagation): use the chain rule from calculus to compute the gradient: for every weight, which direction makes the loss smaller, and how strongly.
  4. Update: move every weight a small step in that direction. The step size is the learning rate. This is gradient descent.

One pass over the whole training set is an epoch. Real training splits the data into mini-batches of, say, 32 samples and updates after each batch.

A tiny network learns XOR

XOR ("exclusive or") is 1 when exactly one of two inputs is 1. Plot the four cases and you'll see that no single straight line separates the 1s from the 0s, so a single neuron (or logistic regression) can't learn it. A network with one small hidden layer can. Here's the whole thing, forward pass, backpropagation and gradient descent, in NumPy:

Python A two-layer network learns XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls from 0.2400 to 0.0002, and the final predictions (about 0.007, 0.984, 0.984 and 0.019) round to the XOR answers 0, 1, 1, 0. The network has just 17 parameters. Each gradient line is one application of the chain rule: output * (1 - output) is the slope of the sigmoid, and 1 - hidden ** 2 is the slope of tanh.

Try a few experiments:

  • Set learning_rate = 0.1: training still works but crawls, ending at a loss of about 0.0036 instead of 0.0002. Set it to 20: the steps are so big that the network overshoots and gets stuck at a loss of 0.25, predicting 1 for three of the four cases.
  • Change the seed to 0: the network starts from different random weights and gets stuck in a poor solution (a loss around 0.125, with two predictions near 0.5). Small networks are sensitive to their starting point; more hidden neurons make this less likely.
  • Change the hidden layer to 2 neurons (size=(2, 2), zeros(2) and size=(2, 1)): with this seed it gets stuck too, while other seeds succeed.

Writing backpropagation by hand is a great way to understand it, and a terrible way to build real models. Frameworks compute every gradient for you automatically. That feature, automatic differentiation, is the main reason they exist.

A ready-made network in scikit-learn

For small tabular problems you don't need a deep-learning framework at all. scikit-learn includes a basic neural network, MLPClassifier (MLP stands for multi-layer perceptron), with the same fit/predict interface as every other model. On the curved "two moons" data, it beats a straight-line model:

Python MLPClassifier vs logistic regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

hidden_layer_sizes=(16, 16) means two hidden layers of 16 neurons. The weight shapes show the layers: 2 inputs โ†’ 16 โ†’ 16 โ†’ 1 output. MLPClassifier uses ReLU and the Adam optimizer by default. It runs on the CPU only, which is fine for thousands of rows but not for images or text at scale.

The same network in PyTorch

PyTorch (from Meta, now run by the PyTorch Foundation) is the most widely used framework in research and increasingly in industry. Its core type is the tensor, a NumPy-like array that can live on a GPU and remembers how it was computed, so loss.backward() can compute every gradient automatically. Here's the XOR network again, read-only:

Python
import torch
from torch import nn

X = torch.tensor([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=torch.float32)
y = torch.tensor([[0], [1], [1], [0]], dtype=torch.float32)

model = nn.Sequential(
    nn.Linear(2, 4),      # weights and bias of the hidden layer
    nn.Tanh(),
    nn.Linear(4, 1),
)
loss_fn = nn.BCEWithLogitsLoss()               # sigmoid + cross-entropy in one, numerically stable
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)

for epoch in range(1000):
    optimizer.zero_grad()            # clear old gradients
    loss = loss_fn(model(X), y)      # forward pass + loss
    loss.backward()                  # backward pass: autograd fills .grad
    optimizer.step()                 # update every weight

with torch.no_grad():
    print(torch.sigmoid(model(X)).round(decimals=3))

Compare it with the NumPy version: nn.Linear holds a weight matrix and a bias, loss.backward() replaces all the hand-written gradient lines, and optimizer.step() replaces the four update lines. Bigger models follow exactly the same loop; you write a class that subclasses nn.Module, feed data in mini-batches from a DataLoader, and move model and data to the GPU with .to("cuda").

The same network in Keras (TensorFlow)

Keras is a high-level API that focuses on short, readable model definitions. Keras 3 runs on top of TensorFlow (Google), JAX or PyTorch; with TensorFlow you import it as from tensorflow import keras. Training is one fit() call, a lot like scikit-learn:

Python
import numpy as np
import keras
from keras import layers

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype="float32")
y = np.array([0, 1, 1, 0], dtype="float32")

model = keras.Sequential([
    keras.Input(shape=(2,)),
    layers.Dense(4, activation="tanh"),
    layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer=keras.optimizers.Adam(0.05), loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X, y, epochs=500, verbose=0)

print(model.predict(X, verbose=0).round(3))
model.summary()        # layers, output shapes and parameter counts

A Dense layer is the fully connected layer you built by hand. compile() picks the optimizer and loss, and fit() runs the training loop for you, with options for batch size, validation data and early stopping. model.summary() would report 17 trainable parameters, the same count as your NumPy network.

PyTorch or TensorFlow/Keras?

PyTorch TensorFlow / Keras
Style Explicit Python training loop Compact compile() + fit()
Strengths Research, flexibility, most new models and papers Quick prototypes, mobile and browser deployment (LiteRT, TensorFlow.js)
Ecosystem Hugging Face, PyTorch Lightning, TorchVision Keras applications, TFX pipelines
Install pip install torch pip install tensorflow (includes Keras)

Both are excellent, the concepts transfer directly, and switching later is easy. If you have no other reason to pick one, PyTorch is the more common choice today.

When to use deep learning

Deep learning isn't automatically better. It shines on some problems and is overkill on others:

Reach for deep learning when:

  • The data is unstructured: images, audio, video, free text. Networks learn features from raw pixels or words that would be impossible to design by hand.
  • You have a lot of data: tens of thousands of examples or more, or you can start from a pretrained model (transfer learning) and fine-tune it on your smaller dataset.
  • You have the compute for it, usually a GPU for anything beyond small experiments.

Prefer classic machine learning when:

  • The data is a table of features (sales, customers, sensor readings). Gradient-boosted trees and random forests often match or beat networks here, train in seconds and need far less tuning.
  • You have little data, where a network tends to overfit.
  • You need to explain each prediction, for example in lending or medicine.

Tip: many modern applications don't train a network at all. They call a pretrained model, such as an image classifier from TorchVision or a language model through an API, and only write the code around it. Knowing how networks work, which you now do, helps you use those models well.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a single artificial neuron compute?

    Each input is multiplied by a weight, the products and a bias are added, and an activation such as sigmoid or ReLU shapes the result.
  2. 2. Why does a neural network need non-linear activation functions between layers?

    A linear function of a linear function is still linear; the activation adds the bends that let deeper networks fit complex shapes like XOR.
  3. 3. In the training loop, what is the job of backpropagation?

    Backpropagation applies the chain rule to find the gradient of the loss for every weight; gradient descent then uses those gradients to update the weights.
  4. 4. In PyTorch, which call replaces all the hand-written gradient lines of the NumPy example?

    Autograd records the forward computation, and loss.backward() fills every parameter's .grad. optimizer.step() then applies the update.
  5. 5. You have 2,000 rows of customer data in 15 numeric columns and want to predict churn. What is the most sensible first step?

    Small tabular datasets are where classic machine learning shines; start there and only move to deep learning if it clearly helps.

Summary

  • A neuron computes a weighted sum plus a bias and passes it through an activation; layers of neurons are matrix multiplications.
  • Activations such as ReLU, tanh and sigmoid add the non-linearity that makes extra layers useful.
  • Training repeats forward pass โ†’ loss โ†’ backpropagation โ†’ gradient-descent update; the learning rate sets the step size, and an epoch is one pass over the data.
  • A two-layer NumPy network with 17 parameters can learn XOR, which no single straight line can separate.
  • scikit-learn's MLPClassifier is enough for small tabular problems; PyTorch and Keras add automatic differentiation, GPUs and building blocks for large models.
  • Use deep learning for images, audio and text, large datasets or pretrained models; prefer classic machine learning for small tables and when predictions must be explained.

Next: Project: Rock, Paper, Scissors, where you'll start putting everything together in complete programs.