Skip to content
Tutorials Center
My learning

Topics

A/B Testing Tutorial

Learn A/B Testing step by step: 37 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is A/B Testing? Learn what an A/B test is, why randomized experiments beat guesses and before-and-after comparisons, and run your first simulated experiment in Python.
  2. 2 The Experimentation Lifecycle Walk through every stage of an A/B test, from idea and hypothesis to design, launch, analysis and decision, and write a pre-registration plan in Python.
  3. 3 Key Terms and Metric Types Learn the vocabulary of A/B testing (control, treatment, lift, null hypothesis, p-value, significance) and the main metric types, with runnable pandas examples.
  4. 4 Simulating Experiments in Python Build a reusable A/B test simulator in Python to generate fake experiment data with a known effect, split users at random, and understand the A/A test.

Experiment Design

  1. 5 Writing Testable Hypotheses Learn to write A/B test hypotheses that name the change, metric, direction and reason, and understand null and alternative hypotheses and one-sided vs two-sided tests.
  2. 6 Choosing Metrics - OEC and Guardrails Choose the right A/B test metrics. Learn the overall evaluation criterion (OEC), primary and secondary metrics, guardrails, proxy metrics and how to avoid gaming.
  3. 7 Randomization and Assignment Units Learn how A/B tests assign users to variants: choosing the randomization unit, deterministic hash bucketing, stratified and cluster assignment, and common pitfalls.
  4. 8 Sample Ratio Mismatch (SRM) Learn what sample ratio mismatch is, why it invalidates A/B test results, how to detect it with a chi-square test in Python, and its most common causes.
  5. 9 Experiment Duration and Novelty Effects Decide how long an A/B test should run. Learn weekly cycles, seasonality, novelty and primacy effects, ramp-up plans, and how to spot a drifting effect in Python.

Hypothesis Tests and Metrics

  1. 10 The Two-Proportion Z-Test Analyze conversion-rate A/B tests with the two-proportion z-test. Learn the formula, pooled standard error, p-values, and check your work against statsmodels and SciPy.
  2. 11 Confidence Intervals for Lift Report A/B test results with confidence intervals. Compute absolute and relative lift intervals, understand coverage by simulation, and use the delta method.
  3. 12 T-Tests for Continuous Metrics Analyze continuous A/B test metrics like revenue per user with Welch's t-test. Learn the standard error, confidence interval, outliers and skew, and how to use SciPy.
  4. 13 Chi-Square and Fisher's Exact Test Test conversion differences with the chi-square test of independence and Fisher's exact test. Learn contingency tables, expected counts, and when to use each.
  5. 14 Bootstrap and Mann-Whitney Tests Analyze A/B tests without strong assumptions. Compute bootstrap confidence intervals, run permutation tests and use the Mann-Whitney U test for skewed metrics in Python.
  6. 15 Multiple Comparisons and False Discoveries Learn why testing many metrics, segments or variants inflates false positives, and fix it with Bonferroni, Holm and Benjamini-Hochberg corrections, shown in Python.

Power and Sample Size

  1. 16 Statistical Power and Error Types Understand Type I and Type II errors, alpha, beta and statistical power in A/B testing, and see how sample size, effect size and noise drive power, with Python plots.
  2. 17 Sample Size for Conversion Rates Calculate the sample size for a conversion-rate A/B test. Learn the formula, use statsmodels, and see how baseline, minimum detectable effect, alpha and power change the answer.
  3. 18 Sample Size for Means and the MDE Compute the sample size for A/B tests on means such as revenue per user, solve for the minimum detectable effect given your traffic, and understand the role of variance.
  4. 19 Power Analysis by Simulation Estimate A/B test power by simulation when formulas fall short. Build a reusable power simulator, handle skewed metrics and compare tests, with runnable Python.

Sequential and Bayesian Tests

  1. 20 The Peeking Problem See why checking an A/B test every day and stopping at the first significant result inflates false positives, and measure the damage with a runnable simulation.
  2. 21 Group Sequential Tests and Alpha Spending Learn how group sequential designs let you stop an A/B test early without inflating errors: Pocock and O'Brien-Fleming boundaries and Lan-DeMets alpha spending, with runnable code.
  3. 22 Bayesian A/B Testing with the Beta-Binomial Model Run a Bayesian A/B test on conversion rates: Beta priors and posteriors, probability that B beats A, credible intervals and expected loss, with runnable Python examples.
  4. 23 Multi-Armed Bandits and Thompson Sampling Compare fixed A/B splits with epsilon-greedy and Thompson sampling bandits, measure regret in a runnable simulation, and learn when bandits are the wrong tool.

Variance Reduction

  1. 24 Why Variance Reduction Matters Understand how metric variance drives A/B test sample size and duration, and try capping, winsorizing and metric choice to shrink noise, with runnable examples.
  2. 25 CUPED: Using Pre-Experiment Data Learn CUPED: adjust A/B test metrics with pre-experiment covariates to cut variance by 30-60%, derive theta, check validity and compare with regression adjustment in Python.
  3. 26 Stratification and Post-Stratification Reduce A/B test variance by comparing like with like: stratified randomization, post-stratification weights, and a simulation showing the narrower estimates, in runnable Python.
  4. 27 The Delta Method for Ratio Metrics Analyze click-through rate and revenue-per-session correctly when you randomize by user: why the naive test fails, the delta method standard error and a bootstrap check in Python.

Pitfalls and Ethics

  1. 28 Novelty, Primacy and Interference Why a lift can fade after launch: novelty and primacy effects, plus network and marketplace interference. Detect them with daily lift plots, cluster randomization and runnable simulations.
  2. 29 Multiple Comparisons and Segment Hunting Why testing many metrics and segments produces false wins, and how Bonferroni, Holm and Benjamini-Hochberg corrections control the damage, with runnable Python simulations.
  3. 30 Winner's Curse and Simpson's Paradox Why shipped A/B test wins shrink after launch, and how an uneven traffic mix can flip a result: simulate the winner's curse and Simpson's paradox in Python and learn the remedies.
  4. 31 Ethics, Privacy and Responsible Experimentation Run A/B tests responsibly: consent and transparency, harm guardrails, privacy-safe data handling, vulnerable users, dark patterns and regulation, with runnable guardrail and pseudonymization examples.

Experimentation Platforms

  1. 32 How Experiment Assignment Works: Hashing and Bucketing Build deterministic user assignment for experiments: hash-based bucketing, per-experiment salts, ramping traffic without reshuffling users, and chi-square checks, with runnable Python code.
  2. 33 Feature Flags and Gradual Rollouts Learn how feature flags power A/B tests and safe launches: targeting rules, percentage rollouts, kill switches, exposure logging and flag cleanup, with a runnable flag evaluator in Python.
  3. 34 Experimentation Platform Architecture and Metrics Pipelines Understand the parts of an A/B testing platform: assignment service, exposure logging, metrics pipeline, analysis engine and dashboards, plus build-versus-buy advice, with a runnable mini pipeline in pandas.

Projects and Practice

  1. 35 Project: Analyze a Checkout Experiment End to End Work through a complete A/B test analysis in Python: data checks, SRM, conversion and revenue tests, CUPED, segments with correction, Bayesian view, guardrails and a written decision.
  2. 36 Project: Build a Mini Experimentation Engine Build a small experimentation engine in Python: hash-based assignment, exposure logging, metric aggregation, SRM check and a significance test, then validate it with A/A simulations.
  3. 37 Practice: A/B Testing Questions and Exercises Test your A/B testing skills with hands-on exercises on peeking, Bayesian updates, CUPED, multiple comparisons and sample ratio mismatch, plus common interview questions with answers.