Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 41 ยท Projects and Practice

Practice: MLOps Exercises

Six hands-on MLOps exercises: schema checks, run fingerprints, gates, drift tests, delayed-label metrics and rollback logic, each with a starter and a solution.

You learn MLOps by making small systems behave. Here are six exercises, each with a runnable starter. Try each one before opening the solution below it. Every starter prints a result you can compare with what the exercise asks for.

Exercise 1: schema check

Write check_schema(df) that returns a list of problems: a missing column, a column with the wrong type, or values outside a range. Expected columns: age (int, 0 to 120) and country (string).

Python Exercise 1: starter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Solution:

Python
def check_schema(df):
    problems = []
    for col in ["age", "country"]:
        if col not in df.columns:
            problems.append(f"missing column: {col}")
    if "age" in df.columns and not df["age"].between(0, 120).all():
        problems.append("age outside [0, 120]")
    return problems

Exercise 2: run fingerprint

Write run_key(data, params, commit) returning a short hash that changes if any of the three change, and stays the same when dictionary key order differs.

Python Exercise 2: starter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Solution:

Python
def run_key(data, params, commit):
    payload = json.dumps({"d": data, "p": params, "c": commit}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:12]

Exercise 3: a promotion gate

Write can_promote(cand, champ) where each is a dict with auc and latency_ms. The candidate must have auc >= 0.80, be no more than 0.01 below the champion's AUC, and not be more than 1.5 times slower.

Python Exercise 3: starter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Solution:

Python
def can_promote(cand, champ):
    reasons = []
    if cand["auc"] < 0.80:
        reasons.append("AUC below 0.80")
    if cand["auc"] < champ["auc"] - 0.01:
        reasons.append("AUC more than 0.01 below champion")
    if cand["latency_ms"] > 1.5 * champ["latency_ms"]:
        reasons.append("latency over 1.5x champion")
    return reasons

Exercise 4: detect a shifted feature

Use a two-sample KS test to flag a feature as drifted when p < 0.01, and report the effect size (the KS statistic) so a large sample does not hide a trivial shift.

Python Exercise 4: starter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Solution:

Python
def drift_report(ref, cur):
    r = ks_2samp(ref, cur)
    return {"drifted": bool(r.pvalue < 0.01), "statistic": round(float(r.statistic), 3)}

Notice the third case: with 200,000 rows even a shift of 0.03 standard deviations is "significant", but the statistic is tiny. Use both numbers.

Exercise 5: accuracy with delayed labels

Given predictions with a day and labels that arrive later (label_day), compute accuracy per prediction day using only labels known by today, and report coverage.

Python Exercise 5: starter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Solution:

Python
def daily_accuracy(df, today):
    known = df[df.label_day <= today]
    out = known.groupby("day").agg(accuracy=("correct", "mean"), labeled=("correct", "size"))
    out["coverage"] = out["labeled"] / df.groupby("day").size()
    return out[["accuracy", "coverage"]].round(3)

Exercise 6: rollback decision

After a promotion, you receive the new model's live scores. Write decide(new_scores, old_level, tolerance=0.02, min_samples=5) returning "wait", "keep" or "rollback".

Python Exercise 6: starter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Solution:

Python
def decide(new_scores, old_level, tolerance=0.02, min_samples=5):
    if len(new_scores) < min_samples:
        return "wait"
    return "rollback" if np.mean(new_scores) < old_level - tolerance else "keep"

Where to go next

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use json.dumps with sort_keys=True in a run fingerprint?

    Key order must not change the hash of identical parameters.
  2. 2. Why return a list of reasons from a gate?

    Explainable failures are fixable failures.
  3. 3. With a huge sample, why look at the KS statistic and not only the p-value?

    Significance grows with sample size; the statistic measures how large the difference is.
  4. 4. What should decide return when there are too few live scores?

    With too little data the decision is unreliable, so keep watching.

Summary

  • Small, testable functions are the building blocks of MLOps systems: checks, keys, gates, tests, metrics and decisions.
  • Explain every failure, record enough to reproduce, and protect decisions from noise with sample-size rules.
  • Practice by combining these pieces into pipelines and monitors.

Next: Back to the MLOps course