Skip to content
Tutorials Center
My learning

Topics

Lesson 61 of 73 ยท Projects

Project: Time-Series Forecaster

Forecast a noisy seasonal series with an MLP, an RNN and an LSTM written in NumPy, and measure every model against naive baselines and a growing horizon.

Forecasting asks a question that looks simple: given what happened so far, what comes next? Sales, energy demand, web traffic and weather all work this way. It is also one of the easiest places to fool yourself, because a model that looks wonderful in a plot can be no better than "tomorrow equals today". So this project is built around honest measurement. You will construct a noisy seasonal series, turn it into supervised-learning examples, train three kinds of network from scratch (an MLP, a recurrent network with backpropagation through time, and an LSTM), and judge all of them against naive baselines, first one step ahead and then over a growing horizon.

The data is synthetic so that we know the truth: the series is a smooth seasonal curve plus random noise, which means there is a hard floor under any model's error. A forecaster cannot predict noise, so the best possible score is the noise level itself. Knowing that floor tells you when to stop tuning.

From a series to examples

The series Period-12 seasonality plus noise.
Windowing 12 past values in, the next value out.
Time split Train on the past, test on the future. Never shuffle first.
1 / 2

The series, windows and baselines

The series has 720 points with a seasonal cycle of 12 (think of months in a year): a main wave with period 12, a smaller wave with period 6, a constant level of 10, and Gaussian noise with a standard deviation of 0.6. The first 480 points are the training period and the last 240 the test period. We split by time, not at random. Shuffling would let the model train on points that come after the ones it is tested on, which is cheating in a forecasting setting.

To use a standard network we need examples with inputs and a target. Windowing slides a window of 12 values along the series: the window is the input and the value right after it is the target. We also standardize with the mean and standard deviation of the training period only.

Before any learning, measure two baselines on the test windows:

  • Last value: predict that the next value equals the most recent one.
  • Seasonal naive: predict the value from one full season (12 steps) ago. For seasonal data this is a surprisingly tough opponent.
Python Windows, the noise floor and naive baselines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three numbers frame the project. A model that knew the clean curve exactly would still be off by the noise, about 0.57 on the test period: that is the floor. Seasonal naive scores about 0.81, because it inherits the noise of the old value as well as the noise of the new one (its error is close to the noise times the square root of two). Repeating the last value scores 1.61, since the series moves a lot between neighbouring points. Any learned model that lands between 0.57 and 0.81 is learning something real.

A feed-forward baseline: the MLP

The simplest neural forecaster ignores the time structure and treats the 12 window values as 12 independent features: 12 -> 32 (ReLU) -> 1. It is trained with mean squared error and Adam, exactly like the regression network. Because it sees the whole window at once, it can learn the seasonal shape as a weighted combination of the inputs. Its weakness is that the window length is baked into its shape.

Besides the one-step score, we measure multi-step forecasts. To predict h steps ahead we forecast one step, append the prediction to the window as if it were real, and repeat. This is called a recursive (or iterated) forecast, and errors compound because each step starts from a guess. We start the forecast from many origins in the test period and average the squared error at each horizon.

Python MLP forecaster and the horizon curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The MLP scores about 0.66 one step ahead, close to the floor and clearly better than seasonal naive. The horizon curves add detail. The last-value forecast is poor, and swings to an error of 4.3 six steps ahead, where the cycle sits at the opposite extreme; at 12 and 24 steps it coincides with seasonal naive, because it then lands on the same phase of the cycle. Seasonal naive has a flat error of 0.77 to 0.86, since it just repeats a season. The MLP stays between 0.61 and 0.67 at every horizon: its forecasts are smooth, so feeding them back in does little harm. That is not guaranteed for every model, as the next section shows. (The curves average 108 forecast origins, so expect wiggles of a few hundredths.)

Reading the window in order: RNN and LSTM

A recurrent network treats the window as a sequence. At each time step it reads one value, updates a hidden state, and only the last hidden state feeds the output layer. The same weights are used at every step, so the model does not care how long the window is. Training uses backpropagation through time, as covered in Backpropagation Through Time: the backward loop walks the 12 steps in reverse and accumulates the weight gradients at each one. We also clip the gradient norm at 5 (inside the Adam.step method), the standard protection against exploding gradients.

An LSTM replaces the plain tanh cell with gates that decide what to forget, what to write and what to expose, with a separate cell state that carries information with less decay. The code is longer, so the first thing the example does is check both backward passes against finite differences on a tiny problem. A correct backward pass has a relative error far below 1e-3. The gate-bias trick used here, starting the forget gate near "remember" (bias 1), is a common choice.

Python RNN and LSTM with BPTT, compared with the MLP

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both gradient checks pass, so the hand-written BPTT is correct. All three networks end up within a few hundredths of one another, around 0.63 to 0.66 RMSE one step ahead, which is within about 0.05 to 0.1 of the noise floor and clearly ahead of seasonal naive. That is a sobering and typical result: on a clean, short, regular series, a recurrent network has no big advantage over a window-based MLP. The LSTM earns its keep on series with long and irregular memory, such as text, rather than on a fixed 12-step pattern.

The horizon plot is where the models differ in character. All three start near 0.63 to 0.67 at one step. As the guesses are fed back in, the MLP stays at about 0.67 after 24 steps, the LSTM grows to 0.76, and the RNN to 0.99, which is worse than seasonal naive (0.83). Small biases in a recurrent output are re-read as input at every step and compound. Over a long horizon a strong naive baseline can be hard to beat, which is why practical systems often forecast several steps directly (one output per future step) rather than recursively.

When the series drifts

Real series rarely stay level. If the series climbs over time, the test period contains values the training period never showed, and networks, which interpolate well and extrapolate poorly, struggle. A common remedy is to make each window relative to its own level: subtract the window mean from the inputs and from the target, let the network predict the deviation, and add the mean back. The window mean carries the drifting level, and the network only has to learn the seasonal shape around it. (Subtracting a single old value, such as the one a season ago, is tempting but also copies that value's noise into the target.)

Python A trending series: predict relative to the recent level

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The series now rises by 0.05 per step, so the test period (30 to 48) lies entirely above the training range (6 to 36). The MLP that predicts the raw value must extrapolate: its error rises from 0.66 (no trend) to 0.82, and the seasonal-naive baseline falls to 1.02 because a season-old value is always one season too low. The level-relative MLP only sees deviations from the recent level, which look the same in training and in the future, and it scores 0.705, almost as good as without the trend. Transforming the problem so that the target looks the same in training and in the future is often worth more than a bigger model.

Never shuffle a time series before splitting it, and never compute scaling statistics on the whole series. Both quietly let information from the future into training, producing results that look wonderful and fail in production.

Real data: weather and stock forecasting

The same recipe is applied to two very different real series, and they teach opposite lessons. Weather has structure that a model can learn, and several related measurements that help. Stock prices look just as forecastable on a chart but are not, and the mistakes people make with them are among the most expensive in machine learning. Both experiments below use generated data, so that the truth is known: there is no weather feed or market data in the browser.

Weather: several features, one target

Real weather data comes with many measurements per day: temperature, humidity, pressure, wind. The input window is therefore not a list of numbers but a table of shape (days, features), and a network input is (batch, 7, 3): 7 days of 3 features. The target is still one number, tomorrow's temperature. Every feature is standardized with training-year statistics, and the error is converted back to degrees at the end, because RMSE in standardized units is meaningless to anyone else.

Our eight years of simulated daily data have a yearly temperature cycle plus day-to-day anomalies. A high pressure reading tends to be followed by a warmer day, humidity is the mirror of temperature, and a day's anomaly depends partly on yesterday's. We train on six years and test on the last two, and compare five forecasters: persistence (tomorrow equals today), climatology (the training-period average for that calendar day), ridge regression on 7 days of temperature only, ridge on all three features, and an LSTM on all three features. Then walk-forward evaluation repeats the experiment with the training period growing one year at a time.

Python Weather: baselines, multivariate windows, an LSTM and walk-forward testing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baselines set the bar. Persistence is off by 1.79 degrees, because the anomaly changes from one day to the next. Climatology, which knows the season but not the weather, is far worse at 4.25 degrees: the recent past matters much more than the calendar. Ridge on temperature alone gets 1.77, barely better than persistence, which tells you that more days of the same variable add almost nothing. The gain comes from other variables: with humidity and pressure, ridge reaches 1.22 degrees, close to the 1.2 of unpredictable noise built into the simulation (the floor). The LSTM, given the same three features, gets 1.34, good but not better than the linear model: when the true relationship is linear, a linear model is hard to beat, and the network needs more data and tuning to catch up. That is a typical outcome on small tabular-style series, and why you always include a linear baseline.

The walk-forward table is how forecasts should be validated. Instead of one split, you pretend to be at the end of year 4, train on years 1 to 4 and forecast year 5, then move on, each time refitting on everything before the test year. The ridge model scores 1.33, 1.29, 1.24 and 1.21 degrees on years 5 to 8, against 1.76 to 1.81 for persistence, and its error shrinks as the training period grows. Consistency across folds is evidence; a single lucky split is not. In real weather data, expect seasons and unusual years to change the fold scores considerably.

Stocks: when the chart lies

A price series wanders, so a model that predicts tomorrow's price appears to work almost perfectly: the price tomorrow is nearly the price today. The next experiment simulates 3,000 days of a stock whose daily returns are unpredictable in direction, but whose volatility clusters: calm periods and stormy periods, as in real markets. It then makes four measurements.

  1. Predict the price. Both "tomorrow equals today" and ridge regression on 10 past prices reach an Rยฒ near 0.99. The score is high because the price level is easy, not because the model is clever.
  2. Predict the return (tomorrow's percentage change), the quantity a trader cares about. The same kind of model now has an Rยฒ of -0.005 and calls the direction right 49.1% of the time, which is chance.
  3. Repeat the return model with an off-by-one bug in the window (the window slides one step too far and contains tomorrow's return). The Rยฒ becomes 1.0000 and the direction is right 100% of the time. This is what "too good to be true" looks like, and it is the most common way a promising backtest is born.
  4. Predict the size of tomorrow's move from the sizes of the last 10 moves. This has a weak but real signal.
Python Stocks: the price illusion, the leak, and what is predictable

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading the output: the price-level Rยฒ values of 0.9892 and 0.9891 are the same whether you use a model or the naive rule, so they prove nothing. The return forecast has no skill, with Rยฒ of -0.0046 and direction accuracy of 49.1% against a market that rose on 48.9% of days. The buggy window gives a perfect score, which a real project would catch by asking what information was available at prediction time? for every input column. And the volatility forecast is the one thing that works: it correlates 0.12 with the real size of the next move, and the days it flags as stormy had an average absolute move of 0.0188 against 0.0148 on the calmest days. That is small, but it is real, and risk management and option pricing use exactly this. In a real market, trading costs, changing regimes and competition erase most apparent edges, so treat this as an educational exercise, not investment advice.

Practical rules for real forecasting data

  1. Know the target and the baseline. Predict a stationary quantity (returns, differences, deviations from a seasonal average) rather than a wandering level, and compare with persistence and seasonal naive.
  2. Split by time and walk forward. Never shuffle. Report the score on several consecutive test periods.
  3. Scale with past data only, including any rolling statistics, and standardize each feature separately.
  4. Audit every feature for look-ahead. Ask, for each column and row, whether it would have been known at the moment of the forecast. Data that is revised later (economic statistics) and timestamps in the wrong time zone are classic traps.
  5. Add useful features, not only more lags: other series (pressure for temperature), calendar features (month, weekday, holiday), exogenous information (forecasts of the weather, promotions).
  6. Try a linear model first. When ridge regression matches a neural network, you have learned something about the problem.
  7. Report uncertainty. A forecast with an error band is more honest and more useful than a single line, for example by training several models and showing their spread.

A real project plugs actual files into the same shapes. This code is read-only:

Python
# Read-only: a multivariate LSTM forecaster on real data with pandas and PyTorch
import pandas as pd, numpy as np, torch, torch.nn as nn

df = pd.read_csv("weather.csv", parse_dates=["date"]).set_index("date")[["temp", "humidity", "pressure"]]
train = df[:"2022-12-31"]; mu, sd = train.mean(), train.std()           # statistics from the past only
z = ((df - mu) / sd).to_numpy(dtype="float32")
X = np.stack([z[i:i + 7] for i in range(len(z) - 7)]); y = z[7:, 0]       # 7 days in, tomorrow's temperature out
n_train = int((df.index[7:] <= "2022-12-31").sum())

class Forecaster(nn.Module):
    def __init__(self, n_features=3, hidden=32):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)
    def forward(self, x):
        out, _ = self.lstm(x)
        return self.head(out[:, -1]).squeeze(-1)                          # forecast from the last time step

Data sets for practice include public daily weather records from national weather services and the Jena climate data set (14 weather variables every 10 minutes), and any daily price history for the stock exercise, where the sensible goal is to reproduce the lessons above (return Rยฒ near zero, volatility clustering) rather than to find a money machine.

Where to go from here

  • Change the window. Try 6 and 24 and look at the error: a window shorter than the season cannot see the cycle.
  • Forecast directly. Give the MLP 24 outputs, one per future step, and compare its horizon curve with the recursive one.
  • Add features. For real data you would add calendar features (month, weekday) or other series as extra input channels.
  • Quantify uncertainty. Train several networks with different seeds and look at the spread of their forecasts.
  • Read the theory again. LSTM and GRU explains the gates used here in more detail.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is a time series split into training and test sets by time instead of at random?

    Forecasting must predict the future from the past. The test set has to lie after the training data, as it will in real use.
  2. 2. What does the noise floor of 0.6 tell you in this project?

    Random noise cannot be predicted. When a model reaches about the noise level, further tuning mostly overfits.
  3. 3. What is a recursive multi-step forecast?

    Each forecast becomes an input for the next one, so errors can compound with the horizon.
  4. 4. In backpropagation through time for the simple RNN, the weight gradients are:

    A shared weight affects every step, so by the chain rule its gradient is the sum of the contributions from all steps.
  5. 5. Why did predicting values relative to the window mean help on the drifting series?

    Networks interpolate far better than they extrapolate. Subtracting the window mean removes the drifting level from what the network must predict.

Summary

  • Turn a series into supervised examples with a sliding window, split by time, and scale with training statistics only.
  • Measure naive baselines (last value, seasonal naive) and the noise floor before judging any model.
  • An MLP, an RNN trained with backpropagation through time, and an LSTM all come close to the noise floor on a clean seasonal series.
  • Recursive multi-step forecasts compound errors; evaluate several horizons, not only one step.
  • Networks extrapolate poorly: for trending data, work relative to the recent level.
  • A longer horizon, drift and shifting patterns are where forecasting gets hard in practice.
  • With several features per day (weather), extra variables helped far more than extra lags of the same one: ridge on temperature, humidity and pressure reached the noise floor (1.22 degrees against 1.79 for persistence), and walk-forward testing confirmed it across years.
  • For stock-like series the price level is deceptively easy (R-squared 0.99 for "tomorrow equals today"), returns are essentially unpredictable (49.1% direction accuracy), an off-by-one window gives a perfect but worthless score, and only volatility has a weak, real signal.

Next project: Cat vs Dog and CIFAR-10 Classifier.