Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 37 ยท Projects and Practice

Practice: A/B Testing Questions and Exercises

Test your A/B testing skills with hands-on exercises on peeking, Bayesian updates, CUPED, multiple comparisons and sample ratio mismatch, plus common interview questions with answers.

Reading about experimentation is not the same as doing it. This lesson gives you five numeric exercises with runnable solutions and a set of conceptual questions of the kind asked in data-science and product-analytics interviews. Try to solve each part on paper or in your head first, then run the code and compare.

Exercise 1: Peeking arithmetic

A team checks their test once a day for 10 days and stops at the first p < 0.05. If the daily checks were independent (they are not, but this gives a rough guide), what is the chance of at least one false positive on an A/A test? Then simulate the correlated case to see the real number.

Python Exercise 1: independent bound and real simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The independent formula overstates the damage. With correlated looks the real inflation is smaller, but still well above 5% (see The Peeking Problem).

Exercise 2: Bayesian update

Control: 90 conversions in 1,000 users. Treatment: 112 conversions in 1,000 users. Use flat Beta(1, 1) priors. What is the probability that treatment beats control, and the expected loss of shipping treatment?

Python Exercise 2: P(treatment beats control)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Interpretation practice: if your team's threshold is "ship when expected loss is under 0.1 points", what do you do, and what would you ask before deciding? (Hint: the interval for the lift is wide with only 1,000 users per arm. Is it worth collecting more data?)

Exercise 3: How much does CUPED help?

Revenue per user has a standard deviation of 30 and a correlation of 0.6 with the same metric in the previous month. You want to detect a lift of 1.0 with 80% power at 5% significance (use n = 16 * sigma^2 / delta^2). What is the required sample size per group without and with CUPED?

Python Exercise 3: sample size with and without CUPED

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The saving equals rho^2, here 36%. See CUPED for the derivation.

Exercise 4: Multiple comparisons

A test reports these p-values for eight secondary metrics: 0.003, 0.009, 0.014, 0.030, 0.045, 0.30, 0.52, 0.81. How many are significant with no correction, Bonferroni, and Benjamini-Hochberg at 5%?

Python Exercise 4: count discoveries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how BH sits between the other two. Which would you choose for a screening list of metrics that will be confirmed in a follow-up test? Which for a ship decision that rests on all eight? (See multiple comparisons.)

Exercise 5: Sample ratio mismatch

The design is a 50/50 split. You observe 50,000 control users and 51,200 treatment users. Is that a problem?

Python Exercise 5: SRM test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 1.2% difference looks small but with 100,000 users it is far outside chance. Do not read the effect until you know why: see sample ratio mismatch.

Conceptual questions

Answer these aloud, then check the notes.

  1. Why not stop a test as soon as it is significant? Repeated looks inflate false positives and overstate effects. Use a fixed horizon or a sequential design.
  2. What is the difference between statistical and practical significance? A tiny effect can be significant with huge samples but worthless; always report effect size and interval, and compare with the minimum effect that matters to the business.
  3. When do you use a bandit instead of an A/B test? Short-lived, low-stakes optimization where reward matters more than precise knowledge (bandits).
  4. Your A/B test is flat. Is the idea bad? Not necessarily: it could be underpowered, the effect may be limited to a segment, or novelty may have cancelled a real effect. Check power, the confidence interval (what effects can you rule out?) and daily trends.
  5. Why does CUPED not bias the estimate? The covariate comes from before randomization, so it is independent of treatment.
  6. How do you handle a ratio metric such as CTR? Analyze at the randomization unit, using the delta method or a user-level bootstrap (delta method).
  7. What could cause a sample ratio mismatch? Bot filtering that differs by arm, redirects that lose users, crashes in one variant, bad hashing or logging differences.
  8. How would you test a feature on a marketplace? Worry about interference; consider cluster, switchback or geo designs (interference).
  9. Shipped winners deliver less than measured. Why? Winner's curse: selection on noise (winner's curse). Shrink estimates, replicate, keep holdbacks.
  10. A test shows +2% overall but -3% on mobile and +10% on desktop. What next? Check segment mix and sample sizes, apply multiple-comparison caution, then run a follow-up test targeting the pre-specified segment hypothesis.

When answering interview questions, structure the answer: state the concept, give a small numeric example, name the failure mode, and say what you would do in practice.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You observe 50,000 vs 51,200 users in a 50/50 test. What should you do first?

    A p-value far below 0.001 means the assignment or logging is broken, so effect estimates are unreliable.
  2. 2. With rho = 0.6 how much does CUPED shrink the required sample size?

    Variance falls by rho^2 = 0.36, and sample size is proportional to variance.
  3. 3. Which correction controls the false discovery rate?

    BH controls the expected proportion of false discoveries; Bonferroni and Holm control the family-wise error rate.
  4. 4. A flat test result with a very wide confidence interval means...

    Absence of evidence is not evidence of absence; the interval shows which effects remain plausible.

Summary

  • Practice turns the formulas into intuition: peeking inflation, Bayesian expected loss, CUPED savings, correction counts and SRM checks.
  • Always report effect sizes with intervals and check data quality before effects.
  • Interview answers should name the concept, a number, the failure mode and the practical action.
  • You now have the full toolkit: design, analysis, variance reduction, pitfalls, platforms and projects.

Next: back to the A/B Testing track page