Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Build a Mini Experimentation Engine

Build a small experimentation engine in Python: hash-based assignment, exposure logging, metric aggregation, SRM check and a significance test, then validate it with A/A simulations.

The best way to understand an experimentation platform is to build a tiny one. In this project you write about 80 lines of Python that do what the big systems do: assign users deterministically, log exposures and events, compute per-user metrics, check the sample ratio and test the result. Then you validate the engine with A/A simulations, the way real platform teams do before trusting any new analysis code.

Before you start, skim assignment and bucketing and the platform architecture lesson, which this project puts into practice.

Requirements

  1. assign(user_id) returns a variant, stable for the same user and independent across experiments.
  2. expose(user_id) records the first exposure with the variant.
  3. track(user_id, value) records a metric event (a conversion or amount).
  4. analyze() returns per-variant stats, an SRM p-value and a test of the difference between each variant and control, using only users who were exposed, counting users with no events as zero.

The engine

Python The Experiment class and a demo run

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output the way a user of the platform would: the SRM p-value says the split is healthy; the treatment row gives a lift estimate near 1.5 points with a confidence interval and a p-value.

Validate with A/A tests

A trustworthy engine, run on data where nothing changed, should flag a "significant" difference in about 5% of runs, and the p-values should be roughly uniform. This single check catches most analysis bugs. The block below defines a compact engine again (only the pieces needed), so it runs by itself.

Python A/A validation: false positive rate and uniform p-values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the false positive rate were far from 5%, or the histogram piled up near 0, your engine (or the assignment) would have a bug such as correlated assignment, counting events twice or an invalid variance formula.

Challenges

Extend the engine; each idea reuses a lesson from this course.

  1. Exposure-time filter: store a timestamp at exposure and ignore events with earlier timestamps.
  2. Ramping: allow weights to change from (0.95, 0.05) to (0.5, 0.5) by expanding the treatment bucket range, and verify earlier users never switch (bucketing).
  3. CUPED: accept a covariate per user and implement the adjustment from the CUPED lesson.
  4. Sequential stopping: add O'Brien-Fleming-style boundaries from alpha spending and refuse to report significance outside planned looks.
  5. Multiple variants: apply a Holm correction when there are three or more arms (multiple comparisons).
  6. Guardrails: add a second metric with a non-inferiority margin.

Production engines add far more: distributed logging, late-arriving data, bot filtering, experiment layers, holdouts and heavy testing. Yours is a teaching model, but every concept in it is real.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the engine ignore events from users who were never exposed?

    Unexposed users did not see the treatment, so including them dilutes or biases the comparison.
  2. 2. Why must users with no events be counted as zeros?

    Conversion is converters divided by all exposed users, not by converters only.
  3. 3. What should an A/A validation show?

    When there is truly no effect, a valid test rejects at its nominal rate and its p-values spread evenly.
  4. 4. Using the same salt for every experiment in the engine would...

    Users would land in the same bucket for every experiment, entangling their effects.

Summary

  • A tiny engine needs deterministic assignment, exposure logging, event tracking and a statistical analysis.
  • Counting users with zero events and ignoring pre-exposure events are essential correctness details.
  • A/A simulations validate the whole pipeline: expect about 5% false positives and uniform p-values.
  • Every advanced feature (CUPED, sequential tests, corrections) plugs into this skeleton.

Next: Practice: A/B Testing Questions and Exercises