Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 73 ยท Projects

Project: Predicting House Prices With a Network

Build a regression neural network in NumPy to predict house prices from tabular data, scale the features, stop early, and compare it with linear and boosted trees.

So far, most of your networks have classified things: which moon, which digit, which class. Plenty of real problems ask for a number instead: the price of a house, the delivery time of a parcel, the temperature tomorrow. That is regression, and a neural network handles it with only three changes: the last layer has no activation, the loss is mean squared error, and the metric you report is an error in the units of the target.

In this project you build a complete regression pipeline on tabular data. You will generate a house-price dataset, split it properly, scale the features, train a network with Adam and early stopping, and then do the most important step of any practical project: compare the network with simpler models from scikit-learn, linear regression and gradient-boosted trees. Tabular data is the one place where classical models are very hard to beat, so an honest comparison is part of the job.

The project plan

Data 1,500 synthetic houses with five features.
Scale Standardize inputs and the target.
Train Two hidden layers, Adam, early stopping.
Compare Linear model and gradient boosting as baselines.
1 / 2

The data

Real housing data brings licensing, cleaning and download steps that would distract from the lesson, so we generate a dataset whose rules we know. Each house has five features: floor area in square meters, number of rooms, age in years, distance to the city center in kilometers, and whether it has a garage. The price grows with area, but the value of each square meter falls with distance and age, and very central houses get a bonus. Those interactions are what make the problem interesting: a straight-line model cannot express "area matters more when you are close to the center".

We keep 1,000 houses for training, 250 for validation (used to decide when to stop) and 250 for the final test. Before building a network, measure three baselines: always guess the average price, fit a linear regression, and fit a gradient-boosted tree ensemble. Any network we train has to be judged against them.

Python Generate the data and measure the baselines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "always the mean" model is off by about 94,000 on a typical house, which is the number to beat. Linear regression brings the error down to about 27,500 (an R-squared of 0.91), and gradient boosting to about 16,000 (0.97). The scatter plots show why a line struggles: price rises with area, but the cloud fans out, and the colour shows that distance changes the slope. Near the center, prices spike.

Scaling the inputs and the target

Area is in the hundreds, the garage flag is 0 or 1, and prices are in the hundreds of thousands. Gradient descent treats every weight alike, so a feature with a huge range produces huge gradients along one axis and tiny ones along another, the ravine problem from Momentum, RMSProp and Adam. The remedy is standardization: subtract the mean and divide by the standard deviation, so that every column is centred on 0 with a spread of about 1.

Two rules keep this honest:

  • Compute the mean and standard deviation on the training set only, then apply them to the validation and test sets. Using statistics from all the data leaks information from the future test set into training.
  • Scale the target too. A network whose outputs start near 0 has to travel a long way to predict 200,000. Train on standardized prices and convert the predictions back with pred * std + mean.

The next example trains the same network three times for the same short budget: raw features and raw prices, raw features and scaled prices, and everything scaled.

Python What happens without scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the same architecture, optimizer and number of steps, the difference is dramatic. With raw prices the network is still far from the right scale (an error of about 67,700 against 89,000 for always guessing the mean), scaling only the prices gets it to about 31,500, and scaling everything reaches about 14,500. The raw-feature network also has to cope with area values hundreds of times larger than the garage flag. Standardization costs two lines and is rarely optional.

The regression network

The architecture is 5 -> 32 -> 32 -> 1 with ReLU in the hidden layers. Compared with the classification networks you wrote earlier:

  1. The output layer is linear. A sigmoid would squeeze predictions into 0 to 1, and a softmax makes no sense for one number.
  2. The loss is mean squared error. Its gradient with respect to the output is 2 * (prediction - target) / N, which is the first line of the backward pass in the code below.
  3. Hidden layers use He initialization, as in Weight Initialization.

Training runs in mini-batches of 64 with Adam, and after every epoch we measure the loss on the validation set. Early stopping keeps a copy of the weights from the best validation epoch and stops after 15 epochs without improvement. The validation set is how we notice the moment the network starts memorizing the training houses (overfitting).

Python Train the regression network with early stopping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The curves tell the story. The validation loss follows the training loss down, then flattens, and training stops 15 epochs after the best point. On this run the best validation epoch is epoch 28 and training stops at epoch 43. On the test set the network reaches an RMSE of about 14,400 and an R-squared of about 0.98, and the predicted-versus-true plot hugs the diagonal. Because the test set took no part in training or in the choice of the stopping epoch, this number is a fair estimate of how the model would do on new houses.

Never use the test set to decide when to stop, which features to keep or which architecture to prefer. Each decision made by looking at it makes the reported score a little more optimistic. Use the validation set for decisions and the test set once, at the end.

The comparison

Now put the three models side by side on the same untouched test houses. We also split the error by distance from the center, because an average can hide where a model fails. The stable seeds mean you get identical numbers every time you press Run.

Python Network versus linear regression versus boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On this dataset the network wins narrowly overall, with an RMSE near 14,400 against about 16,100 for boosting and 27,500 for the linear model. Look at the error bands, though: the linear model is off by 19,000 to 34,000 in every band because it cannot bend, and it is worst for far-away houses, while the two non-linear models stay within a narrow range. The network is ahead in all three bands, but the gap to boosting is only a couple of thousand. The residual plots show the same thing as a pattern. A residual cloud that fans out or curves with a feature means the model is missing structure in that feature.

Do not read too much into the winner. The data is smooth and was generated from a formula, which favours a network. On real tabular data with many categories, missing values and sudden jumps, gradient boosting is usually the stronger or equal choice and needs far less tuning. The sensible workflow is to start with the simple baselines, and reach for a network when the data is large, has image, text or sequence parts, or when you want one model that learns jointly from several kinds of inputs.

Where to go from here

  • Try other widths and depths. Change [5, 32, 32, 1] to [5, 8, 1] and watch the model underfit, or to [5, 128, 128, 1] and watch early stopping kick in sooner.
  • Add weight decay or dropout from Overfitting and Regularization and Dropout and Data Augmentation.
  • Predict the log of the price. Prices are skewed, so many practitioners model log(price) and exponentiate the result. Errors then become relative instead of absolute.
  • Use a more robust metric. RMSE punishes big mistakes heavily. Mean absolute error treats all errors equally. See Evaluating Models for the regression metrics.
  • Average several networks trained from different seeds. Ensembles of small networks are noticeably steadier than any single one.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What are the three standard changes when turning a classification network into a regression network?

    A regression output is an unbounded number, so the last layer has no squashing activation. MSE measures the distance to the target, and RMSE or MAE reports it in dollars (or whatever the target's unit is).
  2. 2. Where should the mean and standard deviation used for standardization be computed?

    Statistics from the test set leak information about it into training. Fit the scaler on the training data and apply the same numbers everywhere else.
  3. 3. Early stopping with patience 15 means:

    Validation loss is the signal for memorization. The weights from the best epoch are kept and used for the final predictions.
  4. 4. On real tabular data, which statement is the most reliable lesson of this comparison?

    Tabular problems are where tree ensembles are strongest. A network is justified only when it beats solid baselines on held-out data.
  5. 5. Why does the project scale the target prices as well as the features?

    A freshly initialized network outputs numbers near zero. Predicting values in the hundreds of thousands would take many steps and produce gigantic gradients, so we train on standardized prices and convert back.

Summary

  • Regression networks end with a linear layer and use mean squared error; report RMSE or MAE in the target's own units.
  • Standardize inputs and targets with statistics from the training set only.
  • Split into training, validation and test sets: validation guides decisions such as early stopping, the test set is used once.
  • Early stopping restores the weights from the best validation epoch.
  • Always compare with a trivial baseline, linear regression and gradient boosting; on tabular data, a network has to earn its place.
  • Residual plots by feature show where a model is systematically wrong.

Next project: Shape Recognition With a CNN.