Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 40 ยท Projects and Practice

Project: Fit a Curve with Gradient Descent

A complete JAX project: define a model and loss, take gradients with jax.grad, jit a train step, fit noisy data with a pytree of parameters, and check gradients numerically.

Time to combine what you have learned in one small, complete program. The task: fit a damped oscillation y = a * exp(-b * x) * cos(c * x) to noisy data. It has three parameters, the model is non-linear, and the gradient is awkward to derive by hand. That is exactly where jax.grad earns its place.

You will write the project in its real JAX form, then run the same algorithm in NumPy with a numerical gradient standing in for jax.grad, so you can watch the optimization and also learn how to test any gradient you compute.

JAX is not available in this site's live editors. The JAX program is read-only; run it with pip install jax on your machine. The runnable blocks are NumPy, with finite-difference gradients instead of automatic differentiation. Their printed numbers come from NumPy.

The plan

flowchart LR
    D["synthetic data<br/>(x, y)"] --> M["model(params, x)"]
    M --> L["loss(params, x, y)<br/>mean squared error"]
    L --> G["jax.value_and_grad"]
    G --> U["update: p - lr * g<br/>(tree_map)"]
    U --> M

Parameters are a pytree (a dict), so the update is one tree_map, and grad returns a dict of the same shape.

The JAX program

Python
import jax, jax.numpy as jnp

# 1. data
key = jax.random.key(0)
x = jnp.linspace(0.0, 6.0, 120)
true = {"a": 2.0, "b": 0.5, "c": 3.0}
noise = 0.05 * jax.random.normal(key, x.shape)
y = true["a"] * jnp.exp(-true["b"] * x) * jnp.cos(true["c"] * x) + noise

# 2. model and loss: params is a dict pytree
def model(params, x):
    return params["a"] * jnp.exp(-params["b"] * x) * jnp.cos(params["c"] * x)

def loss(params, x, y):
    return jnp.mean((model(params, x) - y) ** 2)

# 3. one jitted training step
@jax.jit
def step(params, x, y, lr):
    value, grads = jax.value_and_grad(loss)(params, x, y)
    params = jax.tree_util.tree_map(lambda p, g: p - lr * g, params, grads)
    return params, value

# 4. loop
params = {"a": 1.0, "b": 0.2, "c": 2.7}
for i in range(2001):
    params, value = step(params, x, y, 0.02)
    if i % 400 == 0:
        print(i, float(value), {k: round(float(v), 3) for k, v in params.items()})

Check how little code is not about the problem. There is no gradient formula anywhere. Change the model to a different function and nothing else changes: that is the practical payoff of automatic differentiation. The step function follows the pattern from value_and_grad, and the dict of parameters works through every transformation because it is a pytree.

Running the same algorithm

Below, numeric_grad plays the role of jax.grad. It perturbs each parameter slightly and measures the change in the loss (a central difference). It is slow and approximate, and it needs two loss evaluations per parameter, but it makes the idea concrete and is the standard way to test a gradient.

Python Curve fitting with a numerical gradient (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss falls steadily and the parameters approach a = 2, b = 0.5, c = 3 (they stop a little off because of noise). If you start with c = 1.0 instead of 2.7 you will probably land in a different local minimum: oscillation fits are non-convex, and gradient descent only finds a nearby valley. A decent starting guess matters; plotting the data first usually gives one.

Testing a gradient

Never trust a hand-written gradient until it agrees with a numerical one. In JAX the same check is available as jax.test_util.check_grads. Here is the manual version for an analytic gradient of a simpler loss.

Python Check an analytic gradient against finite differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Improvements to try

  1. Better optimizer. Replace plain descent with Optax Adam (optax.adam(0.05)), as in Optax and TrainState; convergence is faster and less sensitive to the learning rate.
  2. Robust loss. Replace squared error with a Huber loss so a few outliers do not dominate.
  3. Multiple starts. Use vmap over several initial guesses and keep the best final loss: jax.vmap(lambda p0: run(p0))(starts) (see vmap basics).
  4. Confidence in the fit. The Hessian of the loss at the optimum (jax.hessian(loss)(params, x, y), see Jacobians and Hessians) estimates how well each parameter is determined.
  5. Replace the loop. For a fixed number of steps, lax.scan over a step function compiles the entire optimization into one program (scan lesson).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the project store parameters in a dict?

    Pytrees pass through grad, jit and tree_map unchanged in structure.
  2. 2. What does a central finite difference approximate?

    It estimates the slope by evaluating the function slightly above and below the point.
  3. 3. Why might fitting the damped oscillation with a poor starting guess fail?

    Oscillating models have many valleys; gradient descent follows the one it starts in.
  4. 4. How does the numerical gradient compare to jax.grad in cost for n parameters?

    Reverse-mode autodiff costs a small constant times one loss evaluation regardless of n.

Summary

  • A JAX project is: data, a pure model, a scalar loss, value_and_grad, a jitted update, a loop.
  • Parameters as a pytree keep updates to one tree_map and match the gradient's structure.
  • Gradients should be tested against finite differences; jax.test_util.check_grads automates it.
  • Non-convex problems need sensible starting points or multiple starts (vmap).
  • Optimizers, robust losses and scan make the program better without changing its shape.

Next: Project: Simulate a Pendulum with scan and grad