Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 40 ยท Stationarity and Transformations

Transformations: Log and Box-Cox

Stabilize growing variance with log, square-root and Box-Cox transformations, pick lambda, back-transform forecasts correctly, and know when a transformation hurts.

Differencing fixes a changing mean. A transformation fixes a changing variance: the situation where seasonal swings or random fluctuations get bigger as the level rises. Typical for sales, traffic, populations and prices. After the transformation the swings are about the same size everywhere, which makes additive models and constant-variance assumptions valid.

The problem: variance that grows with level

If a series doubles in size over the years and its wiggles double too, an additive model would fit the early years too loosely and the later years too tightly. Taking logarithms turns constant percentage change into constant absolute change.

Python Log stabilises growing variance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the raw series the spread grows block after block. In the log series it is much steadier (the trend still adds some spread inside each block, which differencing would remove).

The family of power transformations

Common choices, from mild to strong, are:

Transform Formula Typical use
None y already constant variance
Square root sqrt(y) counts, mild growth
Cube root y ** (1/3) works with zeros and negatives
Logarithm log(y) percentage growth
log(1 + y) np.log1p(y) data with zeros
Reciprocal 1 / y very strong variance growth

The Box-Cox transformation unifies them with one parameter lambda:

  • lambda = 1: no change (just shifted)
  • lambda = 0.5: square-root-like
  • lambda = 0: natural log
  • lambda = -1: reciprocal-like

Formally w = (y**lambda - 1) / lambda for lambda != 0 and w = log(y) for lambda = 0. It requires y > 0. The Yeo-Johnson version also accepts zeros and negatives.

Pick a transformation

Grows with level Try log or Box-Cox.
Has zeros Use log1p or Yeo-Johnson.
Count data Square root is gentle.
Always invert Forecast, then back-transform.

Transform, model, forecast, then transform back.

1 / 2

Choosing lambda

You can choose lambda by eye (try values and look for constant swings) or let SciPy pick the value that makes the data most nearly normal by maximum likelihood:

Python Estimate lambda with SciPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ratio of the later spread to the earlier spread is near 1 when variance is stable; the raw series has a large ratio. The lambda is estimated for the whole series including its trend, so treat it as a guide and prefer a simple round value (0, 0.5) that you can explain.

Back-transforming forecasts

Models forecast in the transformed world. To report real numbers, apply the inverse function:

  • log: np.exp(w)
  • Box-Cox: scipy.special.inv_boxcox(w, lam)

There is a subtlety. A forecast in log space is a mean of a symmetric distribution; exponentiating it gives the median of the original distribution, which is lower than the mean for a right-skewed variable. If you need the mean, apply a bias adjustment: exp(w + sigma**2 / 2), where sigma**2 is the forecast variance in the log scale.

Python Back-transform with and without bias adjustment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain back-transform underestimates the mean; the adjusted one matches it. Prediction intervals can simply be back-transformed endpoint by endpoint, because the transformation is monotonic.

Transformations and decomposition

If seasonal swings scale with the level, a log transform makes the multiplicative structure additive: log(y) = log(trend) + log(season) + log(noise). You can then use STL on the logged data. See STL decomposition and classical decomposition.

Percentage changes and log returns

For positive series such as prices, np.log(y).diff() gives the log return, approximately the percentage change (0.05 means about 5 percent) and additive over time. It is the standard way to make a growing series stationary in both mean and variance.

Python Log returns compared with percent change

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When not to transform

  • If variance is already roughly constant, a transform just complicates interpretation.
  • Data containing zeros need log1p, a shift, or Yeo-Johnson; take care that the shift is reasonable.
  • If the business measures errors in original units, make sure you evaluate them there, after back-transformation.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which problem does a log transformation mainly address?

    Logs turn constant percentage fluctuations into constant absolute fluctuations.
  2. 2. Which lambda makes the Box-Cox transformation equal to the natural log?

    By definition Box-Cox uses log(y) when lambda is 0.
  3. 3. What does exp() of a forecast made in log space give you?

    The back-transformed point forecast is the median; adding sigma squared over two gives the mean.
  4. 4. Your series contains zeros. Which option works?

    log of zero is undefined, and Box-Cox needs strictly positive data; log1p and Yeo-Johnson handle zeros.

Summary

  • Transformations stabilise the variance; differencing stabilises the mean.
  • Box-Cox covers square root (0.5), log (0) and others with one parameter; Yeo-Johnson handles zeros and negatives.
  • Pick lambda by eye or maximum likelihood, preferring simple values.
  • Always back-transform forecasts; consider a bias adjustment for means.
  • Log differences are log returns, a standard stationary form for prices.

Next: Autocorrelation: ACF and PACF