Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 111 ยท Support Vector Machines and Other Models

Gaussian Processes, Neural Nets and Choosing a Model

Fit a Gaussian process that reports its own uncertainty, train a small neural network with MLPClassifier, and learn a practical way to choose a model for your data.

You now know a wide menu of models: linear models, trees, forests, boosting, SVMs and discriminant analysis. This lesson adds two more flexible families and then steps back to the question every project faces: which model should I try first?

A Gaussian process (GP) is a regression model that does not just predict a number, it predicts a number and how unsure it is. Far from the training data, its uncertainty grows, which is exactly the behavior you want when extrapolating is risky. A neural network (in scikit-learn, the multi-layer perceptron) stacks layers of weighted sums and nonlinear functions, and can learn very flexible shapes. Neither is a default choice for tabular data, but both are worth knowing, and understanding when not to use them is part of choosing well.

Two flexible models

Gaussian process Predicts a value plus an uncertainty band.
Neural network Layers of weights and nonlinear units.
1 / 2

Gaussian process regression

A GP treats the unknown function as a random thing and asks which functions are consistent with the training points. You describe your beliefs about the function with a kernel, a similarity measure: nearby inputs should have similar outputs. The most common choice is the RBF kernel, whose length scale says how quickly the function can wiggle. Add a WhiteKernel for noise in the measurements, and fit tunes the kernel's parameters by maximizing the likelihood of the data.

The prediction method accepts return_std=True and gives you a standard deviation for each point. The shaded band mean +/- 2 * std is roughly a 95% interval for the function value.

Python A prediction with an uncertainty band

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the band. Between the training points it is narrow. Outside the data (left of 0 and right of 6) it widens quickly, and the mean falls back towards the average. At x = 3 the printed standard deviation is small, and at x = 8.5 it is several times larger. A random forest or a polynomial would give you a confident-looking line out there; the GP honestly says "I do not know."

Where Gaussian processes fit

GPs shine with small datasets (tens to a few thousand rows) and smooth relationships, when you care about uncertainty: tuning expensive experiments, scientific measurements, Bayesian optimization of hyperparameters. They struggle with big data, because training costs roughly the cube of the row count and memory grows with its square. Past a few thousand rows, choose another model. GPs also accept classification (GaussianProcessClassifier), but regression is where they are most useful.

Neural networks with MLPClassifier

A multi-layer perceptron is a chain of layers. Each layer multiplies its input by a weight matrix, adds a bias and applies a nonlinear activation (such as ReLU). Training adjusts the weights by gradient descent, the same idea from Gradient Descent from Scratch, only with many more parameters. hidden_layer_sizes=(32, 16) means two hidden layers of 32 and 16 units.

Neural networks are sensitive to feature scale, so put a StandardScaler in front, ideally in a pipeline. They also start from random weights, so fix random_state for repeatable results.

Python A small network on two concentric rings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One class forms a ring around the other, so no straight line can separate them. The linear model scores at or below chance on the test set (about 0.39, because its line cuts through both rings), while the network bends its boundary into a closed curve and scores 1.0 here. Real data is rarely this clean, but the picture shows what nonlinearity buys you. coefs_ shows the weight matrices: 2 inputs to 32 units, 32 to 16, and 16 to 1 output.

Useful MLPClassifier options: alpha (an L2 penalty, the main regularizer), early_stopping=True (hold out part of the training data and stop when the score stops improving), activation, and solver (adam is the default and works well; lbfgs can be faster on small data). A ConvergenceWarning means training hit max_iter before settling: raise max_iter, scale the features, or lower the model size.

Scikit-learn's networks run on CPUs, with no convolutional or recurrent layers. They are fine for small tabular or toy problems. For images, audio or text at scale, use a deep-learning library.

Choosing a model

No model wins everywhere. The no-free-lunch idea says that averaged over all possible problems, every algorithm performs equally well; a model is good only to the extent that its assumptions match your data. So you do not pick the best algorithm in theory, you narrow the candidates by the shape of your problem and then measure.

A practical order of play:

  1. Start with a baseline (a dummy model), then a simple linear or logistic model with scaling.
  2. Try a strong general model for tabular data: a random forest or histogram gradient boosting.
  3. Add special-purpose models only when the data calls for them.
Situation Good first candidates
Small data (hundreds of rows), smooth signal Regularized linear models, GP, SVM with RBF kernel
Medium tabular data (thousands to 100k rows) Random forest, gradient boosting
Large tabular data (100k+ rows) Histogram gradient boosting, SGD linear models
Many sparse features (text) Logistic regression, linear SVM, Naive Bayes
Need interpretability Linear models, shallow trees
Need uncertainty on small data Gaussian process, ensembles
Images, audio, long text Deep learning libraries, not scikit-learn

The table is a starting point, not a rule. Let us run a fair comparison on a synthetic problem, using the same cross-validation folds for each candidate.

Python A fair model shoot-out

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table by comparing means relative to their spread. When two models differ by less than a standard deviation, treat them as tied and prefer the simpler, faster or more interpretable one. The dummy line is your floor: any candidate must clear it comfortably. On this problem, which has several clusters per class, all the nonlinear models (about 0.71 to 0.74) clearly beat the logistic regression (about 0.64), and the dummy sits near 0.51. But the nonlinear models are within each other's spread, so the winner could change with a different seed or dataset. That is the no-free-lunch lesson in practice: you cannot know in advance, so you measure with proper cross-validation.

Beyond accuracy, weigh training time, prediction speed, memory, ease of explaining the model and how well it behaves when the data drifts. A model that is 0.5% better but ten times slower or impossible to explain is often the wrong choice.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does GaussianProcessRegressor.predict(X, return_std=True) give you beyond a plain prediction?

    A GP predicts a distribution at each input. return_std=True returns the standard deviation, which grows where the model has seen little data.
  2. 2. Why are Gaussian processes rarely used on datasets with hundreds of thousands of rows?

    Fitting a GP involves a matrix as large as rows by rows, so time and memory blow up as the data grows.
  3. 3. You train an MLPClassifier and get a ConvergenceWarning. Which action is sensible?

    The warning means the optimizer stopped at max_iter before converging. Unscaled features slow training, so scaling and allowing more iterations are the standard fixes.
  4. 4. What does the no-free-lunch idea suggest for model selection?

    A model works well only when its assumptions fit the problem, so you narrow candidates by data type and size and then measure with cross-validation.
  5. 5. Two models score 0.912 +/- 0.020 and 0.905 +/- 0.018 in cross-validation. How should you treat them?

    The difference is smaller than the fold-to-fold spread, so it is within noise. Other factors, like simplicity and speed, should decide.

Summary

  • A Gaussian process regressor returns a mean and a standard deviation; uncertainty widens away from the data. It is best on small, smooth problems and too slow for large ones.
  • MLPClassifier stacks layers of weights and nonlinear activations; scale the inputs, fix random_state, and watch for convergence warnings.
  • Scikit-learn's networks are fine for small tabular problems; use deep-learning libraries for images, audio and large text.
  • There is no universally best model: start with a baseline and a simple model, try a forest or boosting, then add special models if needed.
  • Compare candidates with the same cross-validation folds, and treat differences within the spread as ties.

Next up: Overfitting, Underfitting and Bias-Variance.