Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท MLflow and ML

Feature Engineering and Model Serving

Build features as governed tables, join them to training data point-in-time, score models in batch with Spark, and serve real-time endpoints with a REST call.

A model is only as good as its inputs. In production the same inputs must be computed the same way at training time and at prediction time; otherwise the model sees different data in the wild than it learned from (called training-serving skew). Databricks tackles this with feature tables in Unity Catalog and gives you two ways to deliver predictions: batch scoring and model serving endpoints.

Databricks cannot run on this site: product code is read-only. Stand-ins in plain Python show the two core ideas, point-in-time joins and shared feature code.

Features as tables

A feature table is an ordinary Unity Catalog Delta table with a primary key (and optionally a timestamp key) that holds computed features for an entity such as a customer.

Python
from databricks.feature_engineering import FeatureEngineeringClient
fe = FeatureEngineeringClient()

features_df = (spark.table("prod.sales.orders")
    .groupBy("customer_id")
    .agg(F.count("*").alias("orders_90d"),
         F.avg("amount").alias("avg_amount_90d")))

fe.create_table(
    name="prod.ml.customer_features",
    primary_keys=["customer_id"],
    df=features_df,
    description="Order behaviour per customer",
)

Because it is a table, it has permissions, lineage and history like any other. Newer workspaces can also declare a plain table with a primary key constraint (PRIMARY KEY (customer_id)) and use it directly as a feature table.

Training with feature lookups

You describe which features to join, not how, and Databricks records that in the model:

Python
from databricks.feature_engineering import FeatureLookup

lookups = [FeatureLookup(table_name="prod.ml.customer_features",
                         feature_names=["orders_90d", "avg_amount_90d"],
                         lookup_key="customer_id")]

training_set = fe.create_training_set(
    df=labels_df,                # customer_id + churned
    feature_lookups=lookups,
    label="churned",
)
train_df = training_set.load_df()

fe.log_model(model=model, artifact_path="model",
             flavor=mlflow.sklearn, training_set=training_set,
             registered_model_name="prod.ml.churn_model")

The model remembers its feature lookups. At scoring time you pass only customer_id and the same features are looked up automatically, using the same table, so skew is avoided.

Point-in-time correctness

If a feature changes over time (balance, order counts), training rows must see the feature value as of the label's timestamp, not today's value. Otherwise future information leaks into training. Feature tables with a timeseries key support point-in-time lookups.

Python Stand-in: point-in-time join versus a leaky join

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Batch scoring

Most predictions are made in bulk: scoring every customer nightly. Load the registered model as a Spark UDF and apply it:

Python
import mlflow
mlflow.set_registry_uri("databricks-uc")

model_uri = "models:/prod.ml.churn_model@champion"
predict = mlflow.pyfunc.spark_udf(spark, model_uri, result_type="double")

scored = (spark.table("prod.ml.customers_to_score")
          .withColumn("churn_score", predict(F.struct("orders_90d", "avg_amount_90d"))))

scored.write.mode("overwrite").saveAsTable("prod.ml.churn_scores")

With feature-engineering models, fe.score_batch(model_uri=..., df=ids_df) looks the features up for you. Schedule this as a job task (see Jobs and Schedules). Batch is cheapest and simplest, so choose it unless users need an answer within seconds.

Real-time model serving

For per-request predictions (fraud check at payment, recommendation on page load), create a Model Serving endpoint: a managed, auto-scaling REST service that hosts a registered model version (or alias), supports scale-to-zero, and logs requests.

Python
import requests, os

url = "https://<workspace-host>/serving-endpoints/churn-model/invocations"
headers = {"Authorization": f"Bearer {os.environ['DATABRICKS_TOKEN']}"}
payload = {"dataframe_records": [{"customer_id": 42}]}

response = requests.post(url, headers=headers, json=payload, timeout=30)
print(response.json())          # {"predictions": [0.17]}

Use a service principal token or OAuth in applications, never a personal token pasted into code. Endpoints can also serve foundation-model APIs and your own custom Python models, and they report latency, throughput and errors.

Batch Real time
Latency Minutes to hours Milliseconds to seconds
Cost Lowest (one job) Endpoint always available
Features From tables Online lookup or request payload
Good for Nightly scores, campaigns Checkout, search, assistants

One feature function for both worlds

Skew often comes from writing the feature logic twice. The stand-in below uses a single function for training rows and for a live request, and checks they agree.

Python Stand-in: one feature function, no skew

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Monitoring what you deployed

Models decay as the world changes. Track:

  • Inference tables: serving logs requests and responses to a Unity Catalog table.
  • Data drift: input distributions shifting from training data.
  • Quality: join predictions with real outcomes later and compute accuracy.
  • Operational health: latency, error rate, cost.

Lakehouse Monitoring can create profile and drift metrics tables on top of any Delta table.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is training-serving skew?

    Different feature logic means the model sees different data than it learned from.
  2. 2. Why use point-in-time lookups?

    Each label should see feature values as they were at its timestamp.
  3. 3. When should you choose batch scoring over a serving endpoint?

    Batch is cheaper and simpler for bulk predictions.
  4. 4. Which credential should an application use to call an endpoint?

    Use machine identities for applications.

Summary

  • Feature tables are governed Delta tables keyed by entity; lookups join them for training and scoring.
  • Point-in-time joins prevent leakage; shared feature code prevents skew.
  • Batch scoring with Spark UDFs is the default; serving endpoints handle real-time needs.
  • Monitor inputs, outputs and quality after deployment.

Next: Structured Streaming Basics