Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 37 ยท Experiment Design

Sample Ratio Mismatch (SRM)

Learn what sample ratio mismatch is, why it invalidates A/B test results, how to detect it with a chi-square test in Python, and its most common causes.

You planned a 50/50 split. After a week the dashboard shows 50,000 users in A and 48,600 in B. Is that fine? It looks close, but with this many users, random chance almost never produces a gap that big. This is a sample ratio mismatch (SRM), and it is the most reliable alarm bell in experimentation: when it rings, the results are not trustworthy, no matter how significant they look.

What SRM is

SRM means the observed split between arms differs from the designed split by more than chance allows. It is a data-quality check on the assignment and logging pipeline, performed before looking at any metric. If the groups were not formed the way you intended, the comparison is not a fair experiment anymore.

Why does it matter? Imagine variant B has a bug that crashes the page for slow phones, so those users never register an exposure. B now silently loses its slowest users, the remaining B users look better, and B appears to win. The metric is "significant" yet entirely fake. The user-count imbalance is the only visible symptom.

Detecting SRM with a chi-square test

The check is a goodness-of-fit chi-square test. Compare the observed counts with the counts expected from the design split. With expected counts E and observed O:

chi2 = sum((O - E)^2 / E)

A tiny p-value means the split is unlikely to be chance. Practitioners use a strict threshold (such as p < 0.001) because SRM checks run on every experiment, and a false alarm only costs a quick investigation while a missed SRM costs a wrong decision.

Python An SRM check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second case shows how a seemingly small gap (about 1.4%) is decisive at this sample size, while the first, a similar-looking wobble, is normal. Intuition about "close enough" fails; always run the test.

How often does chance alone cause it?

To calibrate, simulate honest assignment and check how rarely a correct pipeline trips the threshold:

Python False alarm rate of the SRM check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A fair split almost never trips a 0.001 alarm, so when it does, take it seriously.

Common causes

flowchart TD
    S["SRM found"] --> A["Assignment bugs<br/>bad hash, caching, bucketing"]
    S --> E["Exposure logging<br/>variant crashes or loads slowly"]
    S --> R["Redirects<br/>one arm redirects and drops users"]
    S --> F["Filtering<br/>bot or eligibility rules applied unevenly"]
    S --> T["Timing<br/>arms launched or ramped differently"]
    S --> I["Interaction<br/>another experiment shares traffic"]

Look at where users can be lost or filtered differently between arms.

  • Redirect experiments: if B redirects to a new page, some users drop before the redirect finishes and never get counted.
  • Triggering differences: the treatment changes who qualifies (for example only B shows a banner that triggers logging).
  • Caching and CDN: cached pages serve one variant to everyone.
  • Bot filtering: bot rules that depend on page behavior remove different users from each arm.
  • Ramp mistakes: users assigned at 10% treatment are mixed into the 50% analysis.

What to do when SRM appears

  1. Stop interpreting the results. Do not "adjust" the numbers.
  2. Segment the mismatch by device, browser, country, date, and variant to find where users vanish: SRM is usually concentrated in one slice.
  3. Fix the pipeline, then rerun the experiment. Rarely can the data be rescued.
  4. Automate the check. Good platforms run SRM tests on every experiment, daily, and flag them.
Python Find where the SRM lives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only one browser breaks the 50/50 balance, which points straight at the cause: the new variant fails for Browser X users.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a significant SRM check tell you?

    SRM shows that groups were not formed or counted as designed, which undermines the comparison.
  2. 2. Which statistical test is typically used to detect SRM?

    Observed arm counts are compared with the counts expected from the designed split using a chi-square goodness-of-fit test.
  3. 3. Why is a strict threshold like p < 0.001 common for SRM alerts?

    Checking many experiments with 0.05 would flood teams with false alarms; real SRMs usually have very small p-values.
  4. 4. A redirect-based test shows SRM. What is a likely cause?

    Redirects can drop users before the exposure is logged, leaving that arm short of users.

Summary

  • SRM is a mismatch between the designed and observed split, tested with a chi-square goodness-of-fit test.
  • Treat a significant SRM (often p < 0.001) as a stop sign: do not interpret the metrics.
  • Segment by browser, device, country and date to find the cause, fix it and rerun.
  • Automate the check for every experiment.

Next: Experiment Duration and Novelty Effects