Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 38 · Projects and Practice

Project: A Classifier from Scratch

Build a logistic-regression classifier on synthetic data, train it with gradient descent, Nesterov, SGD and Newton, then add an L1 penalty with proximal gradient for a sparse model.

This final project ties the track together. You will build a classifier from nothing but NumPy, train it with several optimizers from this course, compare them fairly, and finish with an L1-regularized version that selects features. Everything you need was covered in the lessons on logistic regression, stochastic gradient descent, momentum and proximal gradient.

The data

We generate a synthetic dataset so we know the truth: 2,000 samples with 20 features, standardized. Only 4 features actually matter. Labels are drawn at random with probability σ(Xw_true), so the classes overlap and no model reaches 100% accuracy. We split 1,500 samples for training and 500 for testing, and add an L2 penalty λ = 0.001 to keep every problem strongly convex.

Part 1: four optimizers, one problem

The objective is the average logistic loss plus the ridge term. For a fair comparison, we count passes over the data (each full-gradient step is one pass; SGD with batch 50 does one pass in 30 steps; Newton needs one pass for the gradient plus the cost of a 20 × 20 Hessian, which we also count as one pass).

Python Comparing four optimizers

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table with care. All four methods have identical accuracy (0.842 on the training set, 0.856 on the test set), because accuracy is a coarse measure. The loss gap shows how close each is to the optimum after a budget of 30 passes. Newton is the reference, so its gap is 0 by construction. Nesterov and SGD end near 5 × 10⁻⁶, and plain gradient descent is about 25 times further away (1.3 × 10⁻⁴). On this easy, well-conditioned problem even SGD with a decaying step does well; on harder problems the differences grow. The test accuracy of the true weights (0.862) is a useful yardstick: because the labels are random, even the true model cannot be perfect.

Part 2: a sparse classifier with L1

Now we want a model that uses only the features that matter. We replace the L2 term by μ‖w‖₁ and use proximal gradient (ISTA with the logistic loss as the smooth part). Sweeping μ shows the trade-off between sparsity and accuracy.

Python L1-regularized logistic regression with ISTA

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For large μ (0.2) the model keeps a single feature (the strongest one) and accuracy drops to about 0.69. At μ = 0.05 and 0.02 it selects exactly the four informative features and its test accuracy (0.850 and 0.856) matches the dense model. At μ = 0.005 it admits 9 noise features, and at μ = 0 there is no penalty and all 20 features are used.

Part 3: write your report

A good optimization project ends with a short written summary. Write answers to these questions in your own words, using the numbers your runs produced:

  1. Which optimizer reached the smallest loss gap after 30 passes, and why do you think so?
  2. Why is test accuracy nearly the same across methods while the loss gap is very different?
  3. At which μ did the L1 model select the true four features? What happened for larger and smaller μ?
  4. Which method would you choose for a dataset of 100 million samples and 20 features? What about 2,000 samples and 20 features?

A reasonable answer to the last question: with 100 million rows, a pass over the data is expensive, so mini-batch SGD with a good schedule (or a few passes of a quasi-Newton method) is the practical choice. With 2,000 rows and 20 features, Newton's method converges in a few cheap iterations and is hard to beat.

Extensions

  • Add the Adam update from the momentum lesson as a fifth optimizer.
  • Add a bias term (an extra constant feature) and see whether accuracy changes.
  • Use the dual view: fit an SVM on the same data using the code from the SVM lesson and compare accuracy.
  • Choose μ by splitting the training set into train and validation parts instead of using the test set.

Never use the test set to choose hyperparameters such as μ or λ. Doing so leaks information and makes the test accuracy too optimistic. Use a separate validation set or cross-validation, and touch the test set only once at the end.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. Why does the project add a small L2 penalty to the logistic loss in Part 1?

    Without a penalty, separable or nearly separable data can push the weights toward infinity. The penalty guarantees a unique minimizer.
  2. 2. Which optimizer should reach the smallest loss gap after 15 iterations on a small, well-conditioned problem?

    Newton's method converges quadratically near the solution, so a handful of iterations are enough.
  3. 3. What is the role of the soft-threshold in Part 2?

    Each ISTA step takes a gradient step on the logistic loss and then soft-thresholds the weights by μ/L.
  4. 4. Why should hyperparameters such as μ not be chosen on the test set?

    The test set must stay untouched until the end to give an honest estimate of performance on new data.

Summary

  • You trained one convex model with four optimizers and compared them by loss gap after the same number of data passes.
  • Newton's method is the most accurate on a small problem; Nesterov beats plain gradient descent; SGD trades accuracy for cheap steps.
  • An L1 penalty plus proximal gradient yields a sparse classifier that can recover the informative features.
  • Always use validation data, not the test set, to choose regularization strength.

Congratulations: you have now covered the full path from convex sets to duality, KKT, LPs, QPs, proximal and stochastic methods, and machine-learning applications. Return to the Convex Optimization track page to review any chapter.