Skip to content
Tutorials Center
My learning

Topics

Lesson 97 of 111 ยท Projects

Project: Click-Through Rate Prediction

Predict ad clicks from high-cardinality categories: one-hot, the hashing trick, target encoding, log loss versus AUC, and calibration of predicted click rates.

Every time a web page loads, an advertising system decides which ad to show. Its main ingredient is a prediction: what is the chance that this person clicks on this ad in this place? This is the click-through rate (CTR) prediction problem, and it has a few properties that make it a great project:

  • the data is almost entirely categorical (ad id, site, device, hour, banner position), and some columns have hundreds or millions of values (high cardinality),
  • clicks are rare (a few percent of impressions),
  • the model's output is used as a probability, because it is multiplied by a price to rank ads. So log loss and calibration matter more than accuracy.

The data here is a synthetic table of 60,000 impressions, so the lesson runs anywhere. The ad quality, site quality, device, hour of day and banner position all change the click chance, and good ads on mobile get an extra boost (an interaction). Real CTR datasets (for example the large public Criteo or Avazu logs) have the same shape with many more columns.

Related lessons: categorical encoding, calibration, logistic regression and loss functions.

Step 1: Look at the data and set a baseline

We first inspect the click rate, how unevenly impressions are spread over ads (a long tail: one ad gets almost half of the traffic), and the click rate by banner position and device. Then we set a baseline, "always predict the average CTR", and fit a logistic regression on one-hot encoded columns. Since the output is a probability, we judge it with log loss (lower is better) and with AUC (the chance that a random clicked impression is ranked above a random non-clicked one).

Python Step 1: explore and fit a first model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The overall CTR is 6.3%, but it ranges from 3.6% for banners at the bottom to 9.6% at the top, and from 4.4% on desktop to 7.5% on mobile. These effects are exactly what a model can use. One-hot encoding the five columns creates 550 sparse columns, and the logistic regression lowers the log loss from 0.2355 (the baseline that always predicts the average) to 0.2080 and reaches an AUC of 0.756. Notice that a CTR model has a modest AUC even when it is good: most of the outcome is luck (nobody can predict exactly who clicks), so the goal is to rank and to estimate the probabilities well.

Step 2: Three ways to encode high-cardinality categories

One-hot encoding makes one column per value. That is fine for 550 columns, but with millions of ad or user ids it explodes. Two alternatives dominate industry practice:

  • The hashing trick: convert each column=value pair into a text token, and hash the token into one of a fixed number of columns (here 2ยนโด = 16,384). Memory use is fixed, new values at prediction time cause no problem, and feature crosses such as ad ร— device are just more tokens. The price is occasional collisions, where two tokens share a column.
  • Target encoding: replace each category by the average target observed for it (here, its average CTR), with shrinkage toward the global average for rare values. It gives one numeric column per feature, which suits tree-based models such as gradient boosting. scikit-learn's TargetEncoder fits the encoding with internal cross-fitting so that the training rows do not see their own labels (leakage).

We compare three models and then check calibration: if the model predicts a 4% CTR for a group of impressions, do about 4% of them actually click?

Python Step 2: one-hot, hashing with crosses and target encoding, plus calibration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three approaches end up close together (log loss 0.208, 0.210 and 0.211), and the plain one-hot logistic regression wins. That is a good lesson in humility: crosses and boosting are not automatically better, and their extra freedom costs data (each crossed token appears in only a handful of rows here). With millions of rows, crosses and non-linear models usually pull ahead, and the hashing trick becomes the only practical way to store the weights. The calibration table shows the strength of a log-loss-trained logistic regression: in all five groups, from the lowest to the highest scores, the average predicted CTR (1.30%, 2.54%, 4.06%, 6.91%, 16.74%) is very close to the real click rate (1.33%, 2.57%, 3.90%, 6.83%, 16.93%). When the predicted probabilities are multiplied by what an advertiser pays per click, this kind of accuracy is what makes the result trustworthy.

What to do on real data

  • Split by time. Click behavior drifts, so train on earlier days and test on later days (cross-validation for special data).
  • Count features need care. "This ad's CTR so far" is a strong feature, but it must be computed using only past impressions, or it leaks the label.
  • Handle position bias. Ads at the top get clicked more because they are at the top. Include position as a feature during training and fix it to a standard value when you rank ads for a future slot.
  • Watch the long tail. New ads and rare sites have little data; back off to group averages (target encoding with shrinkage does this) and keep exploring them occasionally (bandits).
  • Mind the cost of errors. Over-predicting CTR makes the system overpay; under-predicting hides good ads. Re-calibrate regularly (isotonic regression or Platt scaling).
  • Serve quickly. Ads are chosen in milliseconds, which favors linear models, small trees and precomputed encodings (serving).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is log loss preferred over accuracy for a CTR model?

    Predicting 'no click' for everything is about 94% accurate here, but useless. Log loss rewards probabilities that match the real click rates.
  2. 2. What does the hashing trick do?

    Each column=value token goes to one of a fixed number of buckets. New values are handled automatically; occasional collisions are the price.
  3. 3. A calibrated CTR model predicts 5% for a set of impressions. What should be true?

    Calibration means predicted probabilities match observed frequencies on average, like in the five-group table.
  4. 4. Why should the training and test sets be split by time for real ad data?

    A random split lets the model learn from the future of the test impressions. A time split mimics production.

Summary

  • CTR prediction is a rare-event classification problem on high-cardinality categorical data, where calibrated probabilities matter more than accuracy.
  • One-hot encoding, the hashing trick (with feature crosses) and target encoding are the main ways to feed categories to a model; the simplest often wins on small data.
  • Evaluate with log loss, AUC and a calibration table, and compare with the average-CTR baseline.
  • On real data, split by time, compute count features from the past only, and correct for position bias.

Next: a business problem with mixed numeric and categorical data and missing values: Loan Approval Prediction.