Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 41 ยท Serving and Monitoring

Safe Rollouts: Canary, Shadow and A/B

Release a new model without betting everything on it: shadow mode, canary traffic splits with automatic rollback, A/B tests with significance, and blue-green swaps.

A model that passed every offline test can still behave badly in production. Real traffic differs from the test set, upstream systems have quirks, and users react to predictions in ways a dataset cannot show. Progressive delivery limits the damage: show the new model to a small part of the world first, watch closely, and expand only if everything looks fine.

This lesson covers the main strategies, shows how to split traffic consistently, and builds an automatic canary check and a statistical A/B test.

The strategies

Rollout strategies

Shadow New model sees live traffic but its answers are ignored.
Canary A few percent of users get the new model.
A/B test Two groups, compare business results.
Blue-green Two full stacks, flip traffic at once.

Each step trades speed for safety.

1 / 2
  • Shadow deployment: every request goes to the live model and a copy goes to the candidate. Users only see the live answer. You compare predictions, latency and errors with zero user risk. It cannot show how users react, because they never see the new output.
  • Canary release: a small share (1 to 5 percent) of real traffic is answered by the new model. If error rate, latency and prediction statistics stay healthy, the share grows in steps (5, 25, 50, 100 percent).
  • A/B test: a controlled experiment. Users are randomly split and a business metric (conversion, revenue, retention) is compared with statistics. It answers "is the new model better for the business", which offline metrics cannot.
  • Blue-green: run old (blue) and new (green) stacks side by side and switch the router. Rollback is just flipping back. Good for infrastructure changes; less granular than a canary.
flowchart LR
  R["Incoming requests"] --> S["Router"]
  S -->|"95%"| A["Champion model"]
  S -->|"5%"| B["Challenger model"]
  A --> M["Metrics and logs"]
  B --> M
  M --> D{"Healthy?"}
  D -->|"yes"| E["Increase share"]
  D -->|"no"| F["Roll back to 0%"]

Canary loop: split, observe, then expand or roll back.

Splitting traffic consistently

A user should not see model A on one request and model B on the next. Use a sticky split: hash a stable id (user id) and map the hash into buckets. The same user always lands in the same bucket, and raising the canary percentage only adds users; nobody flips back and forth.

Python Sticky hash-based traffic split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The salt (a string that changes per experiment) matters. Without it, the same users would always be in every experiment's treatment group, which makes results biased.

An automatic canary check

A human staring at dashboards does not scale. Define health rules in advance and let code decide whether to advance or roll back. A typical rule set compares the canary with the champion on the same time window: error rate, p95 latency, and a prediction statistic such as the average score (a sudden shift means something is off).

Python Canary analysis with automatic rollback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three decisions exist, not two. WAIT matters: with 80 requests the numbers are too noisy to judge, and acting on them would produce false alarms or false confidence. Also, the rollback should be automatic and fast; waking someone up at night to flip a switch defeats the purpose.

Running a proper A/B test

For business questions, split users randomly and compare outcomes with a significance test. A common metric is conversion rate. With two groups you can use a two-proportion z-test.

Python Two-proportion z-test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The relative improvement is the same in both cases (4.8 to 5.8 percent), but only the larger experiment is conclusive. Sample size matters.

Rules for trustworthy A/B tests:

  • Decide the sample size and duration before starting (a power calculation). Stopping the moment the result looks good inflates false positives ("peeking").
  • Randomize by user, not by request, so each person has one consistent experience.
  • Pick one primary metric in advance, plus guardrail metrics (latency, errors, complaints) that must not get worse.
  • Run whole weeks, because weekday and weekend behavior differ.

Rollback and kill switches

Every rollout needs a fast way back.

  • Keep the previous model version deployed and warm until the new one has proven itself.
  • Use a feature flag or config value for the traffic share, so rollback is a config change and not a redeploy.
  • Design a fallback (last good score, a rules-based default) for when any model path fails.
  • Practice the rollback. A procedure nobody has tried is a hope, not a plan.

Common mistakes

  • Promoting on offline metrics alone. Offline gains sometimes vanish in production.
  • Canary too small to measure. 1 percent of 500 requests is 5 requests; you learn nothing.
  • Comparing different time windows. Compare canary and champion over the same period.
  • Peeking at A/B tests and stopping at the first good-looking number.
  • No guardrail metrics, so a "winning" model silently harms latency or fairness.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main advantage of a shadow deployment?

    Users only see the live model's answers. You compare outputs and performance without exposing anyone to the new model.
  2. 2. Why hash a stable user id to assign canary traffic?

    Sticky bucketing gives a consistent experience and clean comparisons.
  3. 3. A canary has served only 80 requests. The best decision is

    Too few requests means noisy metrics; a WAIT state avoids false alarms and false confidence.
  4. 4. Which question does an A/B test answer that offline metrics cannot?

    Only a live, randomized experiment measures actual user and business outcomes.

Summary

  • Shadow, canary, A/B and blue-green releases reduce the risk of a new model.
  • Use sticky hash-based traffic splits with an experiment salt.
  • Automate canary decisions (advance, wait, roll back) from pre-agreed health rules.
  • Run A/B tests with a pre-set sample size, a primary metric and guardrails; do not peek.
  • Always keep a fast rollback and a fallback path.

Next: Monitoring Service Health