Skip to content
Tutorials Center
My learning

Topics

Lesson 49 of 73 ยท Frameworks and Production

Deep Learning Frameworks Compared

Compare deep learning frameworks and their history: Theano, Caffe, TensorFlow, Keras, PyTorch, MXNet and JAX, with eager vs graph execution demonstrated in code.

By now you have built the pieces of a deep learning library yourself: arrays, layers, automatic differentiation, optimizers and a training loop. A deep learning framework packages those pieces so that you can write a model in a few lines and run it quickly on a GPU. There are several, they have different philosophies, and they come and go. Knowing why they differ helps you pick one for a project and, just as important, read code written in another one.

This lesson tells the short history, compares the main frameworks, and explains the two ideas that explain most of the differences: how the computation is recorded (eager execution versus a static graph) and how gradients are exposed (a method on a tensor versus a function transformation). Both ideas are demonstrated with small runnable programs. The framework code itself is read-only because none of these libraries runs in the browser runner.

What every framework gives you

Tensors N-dimensional arrays on CPU, GPU or TPU.
Automatic gradients Backpropagation without writing it.
Layers and optimizers Ready-made building blocks.
Deployment tools Export, quantize and serve.
1 / 2

A short history

Frameworks arrived in waves, each fixing a pain point of the one before.

Year Framework What it contributed
2007 Theano (University of Montreal) Symbolic graphs and automatic differentiation compiled to fast code for CPU and GPU. It taught the field what a framework should do. Its development ended in 2017.
2013 Caffe (Berkeley) Models defined in configuration files, fast convolution on GPU, a popular "model zoo". Great for image models, awkward for anything dynamic. Its successor Caffe2 was merged into PyTorch.
2015 TensorFlow (Google) Static graphs, distributed training and a path from research to servers and phones. Version 1 was powerful but famously hard to debug.
2015 Keras A friendly layer-and-model API on top of Theano and then TensorFlow. Became TensorFlow's official high-level API in version 2.
2015 MXNet (Apache) Mixed symbolic and imperative styles, efficient multi-GPU scaling, and the Gluon API. It was used by Amazon, but its community shrank and development has largely stopped.
2015 Chainer (Preferred Networks) Pioneered define-by-run: build the graph as the code executes. Its team later moved to PyTorch.
2016 PyTorch (Facebook, now Meta) Define-by-run with a NumPy-like API, easy debugging with normal Python tools. Took over research within a few years.
2018 JAX (Google) NumPy-style code plus composable function transformations (grad, jit, vmap), compiled with XLA.
2019 TensorFlow 2 Eager execution by default, with tf.function to compile when you want speed.
2023 Keras 3 Multi-backend again: the same Keras code can run on JAX, TensorFlow or PyTorch.

There are others: PaddlePaddle (Baidu), MindSpore (Huawei), MLX (Apple silicon), and runtime-only tools such as ONNX Runtime, TensorFlow Lite and Core ML, which run a model that was trained elsewhere. They matter in specific ecosystems but the first three in the table above (PyTorch, TensorFlow/Keras and JAX) cover most of what you will see in jobs, papers and tutorials today.

Two ways to record a computation

Backpropagation needs to know which operations led to the loss. Frameworks differ in when that record is created.

Define-by-run (eager). Operations execute the moment you write them, and each one leaves a note about how to differentiate it. The graph exists only as a side effect of running the code. Because it is just Python, you can print a tensor in the middle of the model, use an if or a for loop whose length depends on the data, and step through with a debugger. PyTorch works this way, and so does TensorFlow 2 by default.

Define-then-run (static graph). You first build a description of the computation, with placeholders for the inputs, and nothing is calculated. Then you hand the graph to the framework, which can optimize it (fuse operations, plan memory, split it across devices), and you feed data through. Theano, Caffe and TensorFlow 1 worked this way. The graph can be saved and run without Python, which is attractive for production. Debugging is harder, since the Python line you wrote and the code that runs are separated by a compile step.

The program below implements both styles in a few lines for the tiny expression y = w * x + w * w and checks that they agree. The eager version also computes dy/dw by recording each operation.

Python Eager execution versus a static graph

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two styles give identical answers (10.00 for x = 3, w = 2, and -0.25 for x = -1, w = 0.5). The eager version also produced the gradient 7.00, which matches the analytic x + 2w. The static version built a five-node graph first (the placeholders x and w, two multiplications and an addition, with w shared) and only calculated when run was called. That separation is what lets a graph compiler optimize the whole computation, and it is also why today's frameworks blend the two: PyTorch has torch.compile and TorchScript, TensorFlow 2 has tf.function, and JAX has jit. You write code eagerly while developing, then ask for a compiled, graph-like version when you need speed.

Gradients as a method, or as a function

PyTorch and TensorFlow give you a tensor that remembers its history. You compute a loss, call loss.backward() (PyTorch) or use a tf.GradientTape, and gradients appear on the parameters. The parameters are objects with hidden state.

JAX takes the functional route. A model is a pure function of its parameters, so there is no hidden state, and grad is a function that takes a function and returns a new function that computes its derivative. Because the results are ordinary functions, transformations compose: jit(grad(loss)) is a compiled gradient, vmap(grad(loss)) gives per-example gradients, and grad(grad(f)) is a second derivative. Real JAX uses reverse-mode differentiation for grad; the demonstration below uses the simpler forward-mode idea (numbers that carry their derivative along, "dual numbers"), but the composability is the same.

Python grad, vmap and a purely functional update

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For f(x) = x^3 + 2x, grad(f)(2) gives 14, grad(grad(f))(2) gives 12 and a third application gives 6, each time the exact answer from calculus. vmap takes code written for one number and applies it to a whole batch. And the last line trains a one-parameter model with an update that has no hidden state: it takes the parameters in and returns new ones. That style is why JAX code is easy to parallelize and compile, and also why it can feel unfamiliar: layers do not own their weights, so libraries such as Flax or Equinox add a convenient structure on top.

The same model in three frameworks

Here is one small classifier written three ways. The snippets are read-only, but you have met each ingredient before: a stack of layers, a loss, an optimizer and a training step.

Python
# PyTorch (read-only): explicit loop, tensors remember their history
import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 3))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for xb, yb in loader:
    loss = nn.functional.cross_entropy(model(xb), yb)
    opt.zero_grad(); loss.backward(); opt.step()
Python
# Keras (read-only): declare, compile, fit. The framework owns the loop.
from tensorflow import keras

model = keras.Sequential([keras.layers.Dense(64, activation="relu"), keras.layers.Dense(3)])
model.compile(optimizer="adam", loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True))
model.fit(x_train, y_train, epochs=10, batch_size=32)
Python
# JAX (read-only): parameters are plain arrays, the step is a pure compiled function
import jax, jax.numpy as jnp

def loss(params, x, y):
    w1, b1, w2, b2 = params
    logits = jax.nn.relu(x @ w1 + b1) @ w2 + b2
    return -jnp.mean(jax.nn.log_softmax(logits)[jnp.arange(len(y)), y])

@jax.jit
def step(params, x, y, lr=1e-3):
    grads = jax.grad(loss)(params, x, y)
    return [p - lr * g for p, g in zip(params, grads)]

The PyTorch Essentials and TensorFlow and Keras Essentials lessons go through the first two in detail.

More tools you will meet

Beyond the big three there is a layer of libraries for specific jobs. You do not need to learn them now, but you should know what they are for.

Tool What it does
JAX NumPy-style arrays with automatic differentiation, jit compilation through XLA, and easy vectorization (vmap) and parallelism; popular for research and large-scale training
DeepSpeed Microsoft library on top of PyTorch for training very large models: the ZeRO optimizer partitions optimizer state, gradients and parameters across GPUs, plus offloading and mixed precision
OpenVINO Intel toolkit that optimizes trained models and runs them fast on Intel CPUs, integrated GPUs and NPUs
ONNX / ONNX Runtime An open file format for models, and a runtime that executes it on many hardware targets; the usual bridge between frameworks
TensorRT NVIDIA inference optimizer: fuses layers, picks fast kernels and lowers precision for NVIDIA GPUs
Hugging Face Transformers Pretrained language, vision and audio models behind a uniform API (see Hugging Face)
PyTorch Lightning / Keras Structure for training loops: logging, checkpoints, multi-GPU, so you write less boilerplate
TFLite / Core ML / ExecuTorch Runtimes for phones and embedded devices (see Deploying Models)

A common workflow is to train in PyTorch (with DeepSpeed for the biggest models), export to ONNX, and serve with ONNX Runtime, OpenVINO or TensorRT, depending on the hardware you deploy to.

Choosing a framework

Question PyTorch TensorFlow / Keras JAX
Style eager, object-oriented eager with tf.function, high-level Keras functional, compiled
Strongest in research, most open-source models mobile and browser deployment (TF Lite, TF.js), established pipelines large-scale research on TPU and GPU, numerical computing
Learning curve gentle for Python users gentle with Keras, steeper below it steeper (pure functions, random keys)
Ecosystem Hugging Face, Lightning, torchvision TF Hub, TFX, Keras applications Flax, Optax, Equinox, many Google models

Practical advice:

  • If you are not sure, learn PyTorch first. Most recent papers, tutorials and pretrained models (including most on Hugging Face) use it, and the concepts transfer.
  • Pick by constraint, not fashion. A company that deploys to phones with TensorFlow Lite, or a team that already owns TPUs, has a good reason to choose differently.
  • Interchange with ONNX. The ONNX format describes a trained model in a framework-neutral way, so you can train in PyTorch and run it with ONNX Runtime or convert it for another target (see Deploying Models).
  • Do not worry about lock-in. The ideas (tensors, layers, losses, optimizers, batches) are identical, and moving between frameworks mostly means learning new names. Keras 3 can even run one model on three backends.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What distinguishes define-by-run (eager) frameworks from static-graph frameworks?

    In eager mode the graph is created while the Python code runs, which makes debugging natural. A static graph is described first and executed afterwards, which allows whole-graph optimization.
  2. 2. Which framework is built around composable function transformations such as grad, jit and vmap?

    JAX transformations take a function and return a new function, so grad(jit(f)) and vmap(grad(f)) all work.
  3. 3. Why do modern frameworks offer compilation such as torch.compile, tf.function or jax.jit?

    Compilation turns recorded operations into an optimized graph, giving most of the speed of static graphs without giving up easy development.
  4. 4. Which pairing of framework and idea is correct?

    Theano compiled symbolic graphs. Chainer pioneered define-by-run, Caffe used configuration files, and Keras 3 supports JAX, TensorFlow and PyTorch backends.
  5. 5. What is ONNX used for?

    ONNX lets you export a model from one framework and run it with another runtime or on another device.

Summary

  • A framework supplies tensors on accelerators, automatic differentiation, layers, optimizers and deployment tools. The main ones are PyTorch, TensorFlow with Keras, and JAX.
  • Theano, Caffe, MXNet and Chainer shaped the field; each is now historical or niche, and their ideas live on in the current frameworks.
  • Eager (define-by-run) execution is easy to debug; static graphs are easy to optimize and ship; modern frameworks combine them with torch.compile, tf.function and jax.jit.
  • PyTorch and TensorFlow attach gradients to tensors, while JAX exposes grad as a function transformation on pure functions that composes with jit and vmap.
  • If you are unsure, start with PyTorch; use ONNX or Keras 3 when you need portability.

Next: PyTorch Essentials.