Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 40 ยท Gradient Boosting Basics

Loss, Gradients and Hessians

See how boosting libraries turn any loss function into per-row gradients and Hessians, how leaf values come from them, and write a custom objective that matches the built-in one.

In the last lesson you built a boosting loop where each tree fits the residuals. That works for squared error, but real projects need log loss, absolute error, quantiles and more. The trick that makes one algorithm handle all of them is to describe a loss by just two numbers per row: its gradient and its Hessian. This lesson shows what those numbers are, how a leaf value is computed from them, and how to hand your own versions to the library.

One loss, two numbers

Pick a loss L(y, f), where y is the true target and f is the model's current raw score for a row. Boosting asks: if I nudge f a little, how does the loss change?

  • The gradient g = dL/df is the slope. It says which way to move f and how urgently.
  • The Hessian h = dยฒL/dfยฒ is the curvature. It says how much to trust the slope: a high curvature means the loss bends quickly, so take smaller steps.

XGBoost and LightGBM both use a second-order (Newton) step, so they need both. For a group of rows that end up in the same leaf, the best leaf value is:

TEXT
leaf value = - (sum of g) / (sum of h + lambda)

Here lambda is the L2 penalty reg_lambda. The minus sign points the score downhill. Divide by the summed Hessian and you get a Newton step. The learning rate then shrinks that step.

Loss Gradient g Hessian h
Squared error (f - y)ยฒ / 2 f - y 1
Logistic loss (binary), p = sigmoid(f) p - y p (1 - p)
Absolute error sign(f - y) 0 (so libraries use a fixed substitute)

Notice the first row: for squared error the gradient is just the negative residual, which is why the from-scratch loop in Gradient Boosting from Scratch was the same thing in disguise.

Checking the formula against XGBoost

Let us fit one tiny tree with no regularization and compare the leaf values the library stores to the formula. With squared error and lambda = 0, a leaf value should be the mean residual of the rows in it.

Python Leaf value = -sum(g) / (sum(h) + lambda)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The dump shows two leaves whose leaf= values equal the two numbers printed below it (the starting score is the mean, 7.5, and the left and right averages are 4 and 11). The library did nothing more mysterious than add up gradients and Hessians per group.

Why the Hessian matters for classification

For log loss the Hessian p (1 - p) is largest when the model is unsure (p near 0.5) and tiny when it is confident. The consequence is that a confidently wrong row has a big gradient but a small Hessian, so a leaf containing mostly such rows gets a large value, while a leaf of rows the model is already sure about barely moves. That is also why min_child_weight (a minimum on the sum of Hessians, not a row count) behaves differently for classification than for regression. You will use it in Tree Complexity Parameters.

flowchart LR
  A["Current score f for each row"] --> B["Loss gives g and h per row"]
  B --> C["Group rows by candidate split"]
  C --> D["Sum g and sum h in each group"]
  D --> E["Leaf value = -G / (H + lambda)"]
  E --> F["Add learning_rate * leaf to f"]
  F --> A

Each boosting round recomputes g and h from the current scores, builds one tree from their sums, and adds a shrunken version of it.

Writing your own objective

Because the library only needs g and h, you can supply any differentiable loss. The native API accepts a function obj(preds, dtrain) that returns the two arrays. Note that preds are raw scores (log-odds), not probabilities, so apply the sigmoid yourself.

The next example re-implements logistic loss by hand and compares the result to the built-in binary:logistic objective trained with the same settings.

Python A custom objective that matches the built-in one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two models agree to within floating-point noise because they compute the same g and h. Two details to remember. First, with a custom objective the model returns raw scores, so you convert them yourself. Second, a custom loss also changes what the library reports as its default evaluation metric, so pass eval_metric explicitly (more in Objectives and Evaluation Metrics).

What if the Hessian is zero or negative?

The Newton step divides by sum(h) + lambda. A zero Hessian is only safe because of that lambda, and a negative one can flip the step the wrong way. Absolute error has Hessian zero almost everywhere, so libraries either use a fixed Hessian of 1 or replace the leaf values after the tree is grown with the median residual. If you write a custom loss, make sure h is positive; when the true curvature is not, return a small constant such as 1.0 and accept that you are doing gradient descent for that loss.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. For squared error, what are the gradient and Hessian of a row with target y and score f?

    Squared error L = (f - y)^2 / 2 has slope f - y and constant curvature 1, so the gradient is the negative residual.
  2. 2. In the Newton step for a leaf, what does reg_lambda do?

    The leaf value is -G / (H + lambda), so a larger lambda makes every leaf value smaller in magnitude.
  3. 3. A custom objective passed to xgb.train receives preds that are...

    The booster works in raw-score space. For logistic loss you apply the sigmoid inside the objective to get probabilities.
  4. 4. Why is min_child_weight a sum of Hessians rather than a row count?

    With log loss, confident rows have tiny Hessians, so a leaf of many confident rows can still be too light to trust.

Summary

  • A loss is summarized per row by its gradient (slope) and Hessian (curvature).
  • A leaf value is -sum(g) / (sum(h) + lambda), then scaled by the learning rate.
  • Squared error has g = f - y and h = 1; logistic loss has g = p - y and h = p(1 - p).
  • Any loss with a gradient and a positive Hessian can be plugged in through a custom objective.
  • Custom objectives work in raw-score space, so convert outputs yourself.

Next: Regularization and Subsampling