A/B Testing Tutorial
Learn A/B Testing step by step: 37 lessons in 9 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is A/B Testing? Learn what an A/B test is, why randomized experiments beat guesses and before-and-after comparisons, and run your first simulated experiment in Python.
- The Experimentation Lifecycle Walk through every stage of an A/B test, from idea and hypothesis to design, launch, analysis and decision, and write a pre-registration plan in Python.
- Key Terms and Metric Types Learn the vocabulary of A/B testing (control, treatment, lift, null hypothesis, p-value, significance) and the main metric types, with runnable pandas examples.
- Simulating Experiments in Python Build a reusable A/B test simulator in Python to generate fake experiment data with a known effect, split users at random, and understand the A/A test.
Experiment Design
- Writing Testable Hypotheses Learn to write A/B test hypotheses that name the change, metric, direction and reason, and understand null and alternative hypotheses and one-sided vs two-sided tests.
- Choosing Metrics - OEC and Guardrails Choose the right A/B test metrics. Learn the overall evaluation criterion (OEC), primary and secondary metrics, guardrails, proxy metrics and how to avoid gaming.
- Randomization and Assignment Units Learn how A/B tests assign users to variants: choosing the randomization unit, deterministic hash bucketing, stratified and cluster assignment, and common pitfalls.
- Sample Ratio Mismatch (SRM) Learn what sample ratio mismatch is, why it invalidates A/B test results, how to detect it with a chi-square test in Python, and its most common causes.
- Experiment Duration and Novelty Effects Decide how long an A/B test should run. Learn weekly cycles, seasonality, novelty and primacy effects, ramp-up plans, and how to spot a drifting effect in Python.
Hypothesis Tests and Metrics
- The Two-Proportion Z-Test Analyze conversion-rate A/B tests with the two-proportion z-test. Learn the formula, pooled standard error, p-values, and check your work against statsmodels and SciPy.
- Confidence Intervals for Lift Report A/B test results with confidence intervals. Compute absolute and relative lift intervals, understand coverage by simulation, and use the delta method.
- T-Tests for Continuous Metrics Analyze continuous A/B test metrics like revenue per user with Welch's t-test. Learn the standard error, confidence interval, outliers and skew, and how to use SciPy.
- Chi-Square and Fisher's Exact Test Test conversion differences with the chi-square test of independence and Fisher's exact test. Learn contingency tables, expected counts, and when to use each.
- Bootstrap and Mann-Whitney Tests Analyze A/B tests without strong assumptions. Compute bootstrap confidence intervals, run permutation tests and use the Mann-Whitney U test for skewed metrics in Python.
- Multiple Comparisons and False Discoveries Learn why testing many metrics, segments or variants inflates false positives, and fix it with Bonferroni, Holm and Benjamini-Hochberg corrections, shown in Python.
Power and Sample Size
- Statistical Power and Error Types Understand Type I and Type II errors, alpha, beta and statistical power in A/B testing, and see how sample size, effect size and noise drive power, with Python plots.
- Sample Size for Conversion Rates Calculate the sample size for a conversion-rate A/B test. Learn the formula, use statsmodels, and see how baseline, minimum detectable effect, alpha and power change the answer.
- Sample Size for Means and the MDE Compute the sample size for A/B tests on means such as revenue per user, solve for the minimum detectable effect given your traffic, and understand the role of variance.
- Power Analysis by Simulation Estimate A/B test power by simulation when formulas fall short. Build a reusable power simulator, handle skewed metrics and compare tests, with runnable Python.
Sequential and Bayesian Tests
- The Peeking Problem See why checking an A/B test every day and stopping at the first significant result inflates false positives, and measure the damage with a runnable simulation.
- Group Sequential Tests and Alpha Spending Learn how group sequential designs let you stop an A/B test early without inflating errors: Pocock and O'Brien-Fleming boundaries and Lan-DeMets alpha spending, with runnable code.
- Bayesian A/B Testing with the Beta-Binomial Model Run a Bayesian A/B test on conversion rates: Beta priors and posteriors, probability that B beats A, credible intervals and expected loss, with runnable Python examples.
- Multi-Armed Bandits and Thompson Sampling Compare fixed A/B splits with epsilon-greedy and Thompson sampling bandits, measure regret in a runnable simulation, and learn when bandits are the wrong tool.
Variance Reduction
- Why Variance Reduction Matters Understand how metric variance drives A/B test sample size and duration, and try capping, winsorizing and metric choice to shrink noise, with runnable examples.
- CUPED: Using Pre-Experiment Data Learn CUPED: adjust A/B test metrics with pre-experiment covariates to cut variance by 30-60%, derive theta, check validity and compare with regression adjustment in Python.
- Stratification and Post-Stratification Reduce A/B test variance by comparing like with like: stratified randomization, post-stratification weights, and a simulation showing the narrower estimates, in runnable Python.
- The Delta Method for Ratio Metrics Analyze click-through rate and revenue-per-session correctly when you randomize by user: why the naive test fails, the delta method standard error and a bootstrap check in Python.
Pitfalls and Ethics
- Novelty, Primacy and Interference Why a lift can fade after launch: novelty and primacy effects, plus network and marketplace interference. Detect them with daily lift plots, cluster randomization and runnable simulations.
- Multiple Comparisons and Segment Hunting Why testing many metrics and segments produces false wins, and how Bonferroni, Holm and Benjamini-Hochberg corrections control the damage, with runnable Python simulations.
- Winner's Curse and Simpson's Paradox Why shipped A/B test wins shrink after launch, and how an uneven traffic mix can flip a result: simulate the winner's curse and Simpson's paradox in Python and learn the remedies.
- Ethics, Privacy and Responsible Experimentation Run A/B tests responsibly: consent and transparency, harm guardrails, privacy-safe data handling, vulnerable users, dark patterns and regulation, with runnable guardrail and pseudonymization examples.
Experimentation Platforms
- How Experiment Assignment Works: Hashing and Bucketing Build deterministic user assignment for experiments: hash-based bucketing, per-experiment salts, ramping traffic without reshuffling users, and chi-square checks, with runnable Python code.
- Feature Flags and Gradual Rollouts Learn how feature flags power A/B tests and safe launches: targeting rules, percentage rollouts, kill switches, exposure logging and flag cleanup, with a runnable flag evaluator in Python.
- Experimentation Platform Architecture and Metrics Pipelines Understand the parts of an A/B testing platform: assignment service, exposure logging, metrics pipeline, analysis engine and dashboards, plus build-versus-buy advice, with a runnable mini pipeline in pandas.
Projects and Practice
- Project: Analyze a Checkout Experiment End to End Work through a complete A/B test analysis in Python: data checks, SRM, conversion and revenue tests, CUPED, segments with correction, Bayesian view, guardrails and a written decision.
- Project: Build a Mini Experimentation Engine Build a small experimentation engine in Python: hash-based assignment, exposure logging, metric aggregation, SRM check and a significance test, then validate it with A/A simulations.
- Practice: A/B Testing Questions and Exercises Test your A/B testing skills with hands-on exercises on peeking, Bayesian updates, CUPED, multiple comparisons and sample ratio mismatch, plus common interview questions with answers.