Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 41 ยท Data and Experiment Tracking

Feature Engineering and Feature Stores

Learn how to share feature code between training and serving, avoid training-serving skew, and use point-in-time correct joins, the core idea behind every feature store.

A model never sees raw data. It sees features: numbers computed from raw data, such as "orders in the last 30 days" or "average basket value". In a notebook you compute them once. In production you compute them twice: once in bulk when you train, and again, one row at a time, when a live request arrives. If those two computations differ even slightly, the model quietly gets worse. This problem has a name: training-serving skew.

This lesson shows where skew comes from, how to prevent it with one shared feature function, and how a feature store generalizes that idea for whole organizations.

Where features are computed

Training Bulk, over history.
Serving One request, milliseconds.
Feature store One definition for both.

Two code paths means two chances to disagree.

1 / 2

One feature function for both worlds

The simplest defense is a single function that both the training job and the prediction service import. Nobody re-implements it in SQL for training and in Python for serving.

Python Shared feature code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single serving row has exactly the same values as the second training row (index 1), because there is only one implementation. Note the explicit handling of None and negative amounts: decisions about missing values belong in the shared function, not scattered across callers.

Demonstrating skew

Here is what happens when someone re-implements a feature "almost" the same way.

Python A skew you can measure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two means differ by a constant factor (ln 10, about 2.30). The model would receive inputs on the wrong scale and make confident, wrong predictions. Nothing crashes. This is why monitoring input distributions matters, and why a test that compares both paths is worth writing.

Time travel: point-in-time correctness

The subtler skew is about time. Suppose you train on a label "customer churned in March" and a feature "number of support tickets" computed today. Tickets filed after the churn leak information the model could never have had at prediction time. This is data leakage.

The rule: for each training example, join features as of the example's timestamp, never later. This is a point-in-time join, and pandas has a tool for it, merge_asof.

Python A point-in-time join

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Customer a on 2025-03-10 gets tickets_30d = 0 (the value from 03-01), not 2 or 5, because those were not known yet. With direction="backward" the join can never peek into the future.

What a feature store adds

A feature store is a service built around these ideas. Typical parts:

Part Job
Feature registry Names, owners, definitions and data types of every feature.
Offline store Large historical tables (a warehouse or files) for training and point-in-time joins.
Online store A fast key-value lookup (Redis, DynamoDB) holding the latest value per entity for serving.
Materialization jobs Scheduled pipelines that compute features once and write them to both stores.

Because both stores are filled from the same definition, serving gets exactly what training saw. Teams also reuse features instead of rebuilding "customer lifetime value" five times. Open-source examples include Feast; cloud platforms ship their own.

Do you need one? Not at first. A shared Python module plus point-in-time joins covers a lot. Reach for a feature store when many models share features, when online latency needs pre-computed values, or when the same logic is being duplicated across teams.

Version your feature definitions like code. When you change how a feature is computed, bump a version and retrain; do not silently change it under a deployed model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is training-serving skew?

    Skew means the model is fed features computed differently at serving time than at training time, so its inputs do not match what it learned from.
  2. 2. Which approach best prevents skew?

    A single shared implementation (or a feature store built on one definition) removes the chance of the two paths disagreeing.
  3. 3. What does a point-in-time join guarantee?

    Point-in-time joins use the feature value as of the example's timestamp, which prevents leaking future information into training.
  4. 4. What is the online store in a feature store for?

    The online store (a key-value database) serves current feature values in milliseconds; the offline store holds history for training.

Summary

  • Features are computed in training and again in serving; any difference is training-serving skew.
  • Share one pure feature function, and handle missing values inside it.
  • Join history with point-in-time logic (merge_asof, direction="backward") to avoid leakage.
  • A feature store adds a registry, offline and online stores and materialization jobs, all fed from one definition.
  • Start simple; adopt a feature store when sharing and latency demand it.

Next: Experiment Tracking Basics