Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Experimentation Platforms

Experimentation Platform Architecture and Metrics Pipelines

Understand the parts of an A/B testing platform: assignment service, exposure logging, metrics pipeline, analysis engine and dashboards, plus build-versus-buy advice, with a runnable mini pipeline in pandas.

Running one A/B test by hand in a notebook is fine. Running hundreds at once across a company needs a platform: shared infrastructure that assigns users, records what they saw, computes metrics, and reports results consistently. Whether you build it, buy it or combine open-source parts, the same components appear.

The components

Experimentation platform parts

Config and flags Define experiments and targeting
Assignment service Hash users into variants
Exposure and event logs What users saw and did
Metrics pipeline Join, aggregate per user

Four pieces, then analysis

1 / 2
  1. Configuration and flags. A UI or config repository where owners define the experiment: key, variants, allocation, targeting rules, primary metric, guardrails, start and end dates.
  2. Assignment service (or SDK). Evaluates the rules and hash-buckets users, as in assignment and bucketing. Client-side SDKs cache rules; server-side SDKs evaluate locally for low latency.
  3. Exposure logging. One row per user-experiment-variant when the user first sees the change.
  4. Event logging. Clicks, purchases, page views, errors, latency, with user ids and timestamps.
  5. Metrics pipeline. Scheduled jobs join exposures with events, filter to events after exposure, compute per-user metric values (the unit of analysis) and store them.
  6. Analysis engine. Runs statistical tests with the methods in this course: SRM checks, variance reduction, sequential or fixed-horizon tests, multiple-comparison handling.
  7. Reporting and governance. Dashboards, alerts, an experiment registry that records hypotheses and outcomes so teams learn from history.
flowchart LR
    A["Experiment config"] --> B["Assignment SDK"]
    B --> C["Product (variant shown)"]
    C --> D["Exposure log"]
    C --> E["Event log"]
    D --> F["Metrics pipeline<br/>(per-user table)"]
    E --> F
    F --> G["Analysis engine"]
    G --> H["Dashboard and alerts"]
    H -->|"decision"| A

Data flows from the config to the product, back through logs, into per-user metrics and the analysis.

A mini pipeline in pandas

The heart of the pipeline is a join. We simulate an exposure log and an event log, keep only events that happened after the user's exposure, build the per-user table, and test the result.

Python Exposure log joined with events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every user from the exposure log appears in the final table, including those who never purchased (zero orders). Forgetting those zeros, or including events from before exposure, are the two most common pipeline bugs.

Pipeline checks every platform should run

Python Automated data-quality checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The SRM check flags that something is wrong in logging or assignment before anybody reads the effect (see sample ratio mismatch). Results from a flagged experiment should be hidden or marked untrustworthy until the cause is fixed.

Build, buy or combine?

  • Buy (commercial platforms such as Optimizely, Statsig, LaunchDarkly Experimentation, Eppo, VWO, Amplitude Experiment): fastest start, built-in stats, support. Costs rise with traffic, and you depend on the vendor's methodology and data handling.
  • Open source (GrowthBook, Unleash, Flagsmith, PostHog experiments, Wasabi-style projects): control and lower license cost; you run infrastructure and maintain the analysis.
  • Build in-house: worthwhile when you run many experiments, need custom units (marketplaces, switchbacks) or own your metrics layer. Expect a team, not a person.
  • Combine: use a flag tool for assignment and your data warehouse for analysis ("warehouse-native"): SQL on your own tables with a stats library on top.

Questions to ask before choosing: Where does metric data live? Who writes metric definitions? How are SRM, sequential testing and variance reduction handled? Can analysts reproduce numbers by hand? What happens to PII?

Start small: a script that reads exposure and event tables from your warehouse and prints a results table is a legitimate first platform. Add features only when real experiments demand them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why filter events to those after exposure?

    Only post-exposure behavior can reflect a treatment effect; earlier events add noise or bias.
  2. 2. What is a common pipeline bug when computing conversion?

    Every exposed user belongs in the table with zeros where nothing happened.
  3. 3. What does the experiment registry provide?

    Recording experiments lets teams avoid repeating failed ideas and learn from effect sizes.
  4. 4. Which approach keeps metric data in your own warehouse?

    Warehouse-native tools run analysis as queries over your own tables.

Summary

  • A platform = configuration, assignment, exposure and event logging, a metrics pipeline, an analysis engine and reporting.
  • The core data operation is joining exposures to post-exposure events at the user level, keeping users with zero events.
  • Automated checks (SRM, duplicates, missing data) must run before anyone trusts a result.
  • Buy, adopt open source, build or combine depending on scale, custom needs and data governance.

Next: Project: Analyze a Checkout Experiment End to End