Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 76 ยท Modelling

Models, Learning and the Modelling Loop

Understand supervised vs unsupervised learning, regression vs classification, signal and noise, the bias-variance trade-off and baselines, with live scikit-learn examples.

A model is a simplified, mathematical description of how some outcome depends on other things. "Price rises about 3,000 per extra square meter" is a model. So is a spam filter or a recommendation list. A model is always wrong in detail, and useful when it is wrong in ways that do not matter for your decision.

This lesson opens the modelling chapter. It gives you the map: the kinds of learning problems, what it means for a model to "learn" from data, why more complexity is not always better, and why every project starts with a humble baseline. The following lessons go deep on specific models; here you meet the ideas they all share. The cleaning and exploration steps from earlier chapters feed directly into this stage, and the data science workflow shows where it sits.

Kinds of learning problems

Almost every modelling task fits one of a few shapes.

Supervised learning uses examples with a known answer. Each row has features (the inputs, X) and a target (the answer you want to predict, y). The model learns the pattern linking them and then predicts the target for new rows.

  • Regression predicts a number: a house price, tomorrow's demand, a delivery time.
  • Classification predicts a category: churn or stay, fraud or legitimate, which of five product types.

Unsupervised learning has no target. You ask the algorithm to find structure: groups of similar customers (clustering), or a simpler description of many correlated columns (dimensionality reduction). Later lessons cover clustering and dimensionality reduction.

There are other families (reinforcement learning, semi-supervised learning), but supervised learning is the workhorse of everyday data science and the focus here.

flowchart TD
    A["What do you want?"] --> B{"Is there a target<br/>column to predict?"}
    B -->|"Yes"| C{"Is the target<br/>a number?"}
    B -->|"No"| D["Unsupervised:<br/>clustering, dimensionality reduction"]
    C -->|"Yes"| E["Regression"]
    C -->|"No, a category"| F["Classification"]

Choosing the family of problem starts with one question: is there a target, and what type is it?

What "learning" means: signal and noise

Real data combine two parts. The signal is the repeatable pattern (larger homes cost more). The noise is everything else: quirks of individual sales, measurement error, factors you did not record. A model should capture the signal and ignore the noise.

Learning means choosing the model's parameters so it fits the training data well, usually by minimizing an error measure (a loss function), such as the average squared difference between predictions and true values. The real test, however, is performance on new data the model has not seen. A model that fits the training set perfectly by memorizing its noise will fail on tomorrow's data. This is called overfitting, and it is the central danger in modelling.

Because of noise, no model can be perfect. If house prices scatter randomly by about 25 thousand around any pattern, even the true pattern has an unavoidable error of that size. This irreducible error sets the floor for what is achievable, and knowing it stops you from chasing impossible accuracy.

Always start with a baseline

A baseline is the simplest sensible prediction: always predict the average, always predict the most common class, repeat last week's value. It is your reference point. A model that cannot beat a baseline is not worth deploying, and a model that barely beats it may not justify its complexity. Baselines also reveal how hard the problem is.

Python A naive baseline versus a linear model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline misses by about 97 thousand on average; the linear model by about 20.5 thousand. Because we built the data ourselves we also know the best any model could do is roughly 19.9 thousand (the mean absolute value of normal noise with standard deviation 25 is 0.798 times 25), so the model is already close to the ceiling. With real data you never know the ceiling, which is why baselines and sensible comparisons matter. Note the pattern in the code: fit on the training rows, evaluate with the metric on the held-out test rows. The learned coefficients (about 3.1 and -0.84) land close to the true 3.0 and -0.8.

Bias, variance and the complexity trade-off

Models differ in flexibility. A straight line is rigid; a high-degree polynomial can bend almost anywhere. Choosing flexibility means balancing two kinds of error:

  • Bias is error from a model that is too simple to capture the signal. A line through a curved pattern is wrong however much data you give it. High bias leads to underfitting.
  • Variance is error from a model so flexible that it follows the noise in the particular training sample. Give it a different sample and it changes wildly. High variance leads to overfitting.

As complexity grows, bias falls and variance rises. Training error only ever falls (the model can always fit the training set better), but error on unseen data first falls and then rises. The best model sits at the bottom of that U shape. The experiment below fits polynomials of degree 1 to 12 to 30 noisy points sampled from a sine wave.

Python Training error versus test error as complexity grows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Degree 1 and 2 underfit (both errors are high). Degree 3 gets the test error close to the noise floor of 0.09 (the noise standard deviation is 0.3, so its variance is 0.09). Beyond that the training error creeps lower while the test error explodes: by degree 12 the curve fits the 30 training points almost perfectly and is wildly wrong between them. The training error alone would have told you to pick the worst model. That is why you must always judge a model on data it has not seen, the topic of train/test splits and cross-validation.

Measuring bias and variance, and reading learning curves

The trade-off above is a picture. You can also measure it, and you can diagnose a real model with two standard plots: the learning curve and the validation curve.

Bias and variance by simulation

For a fixed test point, imagine drawing many different training sets from the same process and fitting the same model to each. The predictions at that point vary from fit to fit. Then:

  • Bias squared is how far the average prediction is from the truth.
  • Variance is how much the predictions spread around their own average.
  • Noise is the irreducible randomness in the target.

Expected squared error is, on average, bias^2 + variance + noise. Simple models have large bias and small variance; flexible models are the opposite.

Python Estimating bias and variance of a rigid and a flexible model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The straight line (degree 1) has large bias and little variance. The degree 9 curve has almost no bias but a large variance. The middle model balances them, which is why it wins on new data. In real projects you cannot redraw the data hundreds of times, so you use the two curves below.

Learning curves: does more data help?

A learning curve plots training and validation score against the number of training examples. sklearn.model_selection.learning_curve computes it with cross-validation. How to read it:

  • Both curves are low and close together: high bias (underfitting). More data will not help. Use a richer model or better features.
  • Training score is high and validation score is much lower, with a gap that narrows slowly: high variance (overfitting). More data, simpler models or regularisation will help.
  • Curves have converged at a good level: you are done, and more data will not change much.
Python Learning curves for an underfit and an overfit model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The tree memorises its training data (training accuracy near 1.0) while its validation score stays lower: a large gap, the signature of variance. The logistic regression has a small gap and a lower ceiling, closer to bias. Notice that the flexible tree still scores higher on validation (about 0.80 against 0.73): a large gap is a problem to manage with more data, pruning or an ensemble, not a reason to prefer the rigid model. Neither diagnosis is a verdict, only a hint about what to try next.

Validation curves: choosing a complexity setting

A validation curve plots training and validation score against one hyperparameter that controls flexibility, such as a tree's max_depth or a ridge model's alpha. validation_curve runs the cross-validation for each value. The best setting is where the validation score peaks; training and validation scores that diverge to the right indicate overfitting.

Python A validation curve for tree depth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pick the setting from the validation curve, then confirm it once on a held-out test set, as explained in Train, Test and Validation. Tuning many settings automatically is the subject of Pipelines and Tuning.

When accuracy lies: baselines for classification

The same baseline habit protects you in classification, where a naive-looking metric can be very misleading. If only 10% of customers churn, a "model" that predicts that nobody churns is right 90% of the time.

Python A majority-class baseline and accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The do-nothing baseline scores 90% accuracy and finds none of the churners. The logistic regression with its default 0.5 cut-off is barely better in accuracy and finds only 10% of churners. Lowering the threshold to 0.15 finds more than half of them at the cost of accuracy, which is a trade you should make on purpose, based on what a missed churner costs compared with a wasted retention offer. Choosing metrics and thresholds is the subject of evaluating models; the lesson here is that a number means little until you know the baseline it must beat.

Prediction: preparing the data and comparing methods

In everyday language "prediction" means guessing a value we do not know yet. In data mining the word usually means predicting a number (regression: next month's sales), while classification predicts a category (will this customer churn?). Both learn from rows whose answer is known and apply what they learned to new rows. Because the preparation and the comparison criteria are the same, treat them together.

Preparing the data for prediction.

  • Cleaning: fix or remove noise, and decide how to treat missing values (Missing Values: Strategies).
  • Relevance analysis: drop attributes that carry no information or only repeat others, which speeds up learning and often improves accuracy (Feature Selection).
  • Transformation: normalize numeric attributes (for distance-based methods such as k-nearest neighbours) and generalize detailed values to higher-level concepts, for example street to city, or exact age to an age band (Encoding and Scaling Features).

Criteria for comparing prediction methods. A single accuracy number is not enough:

Criterion Question
Accuracy How close are predictions to the truth on unseen rows?
Speed How long do fitting and predicting take?
Robustness Does it cope with noise, outliers and missing values?
Scalability Does it still work when rows and columns grow a hundredfold?
Interpretability Can a person understand why it predicted what it did?
Python Comparing prediction methods on several criteria

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table with the criteria in mind. The plain linear model is the most accurate here (the data really is linear plus noise), the fastest, the easiest to explain, and only slightly hurt by the corrupted rows. The random forest is much slower to fit, ignores the corruption entirely (trees split on thresholds, so an extreme value just lands in the last leaf), but cannot match the linear model on this kind of data. Nearest neighbours does worst because most of the 15 features are irrelevant noise to a distance measure, which is the relevance-analysis point from the list above. Exact times differ per computer; the ordering does not. Choose the method that satisfies your actual constraints, not the one with the best single number.

The modelling loop

Modelling is iterative, not a single run:

  1. Define the target and the metric that reflects the decision.
  2. Split the data so some rows stay unseen until the end.
  3. Build a baseline and record its score.
  4. Try a simple model, then richer ones, changing one thing at a time.
  5. Evaluate on held-out data; inspect the errors, not just the average.
  6. Improve features or data, which usually pays off more than a fancier algorithm.
  7. Stop when the gain no longer justifies the complexity, and document the result.

Tip: Prefer the simplest model that meets the need. It is easier to explain, test, maintain and trust. Reach for complexity only when a baseline and a simple model leave a gap that matters.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You want to predict the number of minutes a delivery will take. Which kind of problem is this?

    The target is a number, so this is supervised regression. Classification predicts categories, and clustering has no target.
  2. 2. Training error keeps falling as a model gets more complex, but test error starts rising. What is happening?

    Rising test error with falling training error is the signature of overfitting, where variance outweighs the reduction in bias.
  3. 3. Why build a baseline before a sophisticated model?

    Without a baseline you cannot tell whether a model's score is good. A 90% accuracy may be no better than predicting the majority class.
  4. 4. Only 2% of transactions are fraud. A model that labels everything 'legitimate' has what accuracy?

    It is right on every legitimate transaction, about 98%, yet catches no fraud. Accuracy alone is a poor metric for rare events.
  5. 5. What does 'irreducible error' mean?

    Noise and unrecorded factors cap the achievable accuracy; no model, however flexible, can predict below that floor on new data.

Summary

  • Supervised learning predicts a target from features (regression for numbers, classification for categories); unsupervised learning finds structure without a target.
  • Data contain signal and noise; a good model captures the signal, and noise sets a floor on error.
  • Overfitting means memorizing noise: training error falls while error on unseen data rises, so always evaluate on held-out data.
  • Bias (too simple) and variance (too flexible) trade off; the best model sits in between.
  • A baseline gives the score to beat, and accuracy can mislead when classes are imbalanced.
  • Work in a loop: metric, split, baseline, simple model, evaluate, improve, stop.

Next: Linear Regression for Inference and Prediction.