Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 37 ยท Hypothesis Tests and Metrics

T-Tests for Continuous Metrics

Analyze continuous A/B test metrics like revenue per user with Welch's t-test. Learn the standard error, confidence interval, outliers and skew, and how to use SciPy.

Not every metric is a yes/no. Revenue per user, minutes spent, order value and latency are numbers on a scale. To compare their averages between arms, the standard tool is the t-test. In A/B testing, the version you want is almost always Welch's t-test, which does not assume the two groups have the same variance.

The logic

The idea mirrors the z-test. Compute the difference in means d = mean_B - mean_A, estimate how much that difference would wobble from sampling noise, and ask how many standard errors it is from zero.

SE = sqrt( s_A^2 / n_A + s_B^2 / n_B )

t = d / SE

where s^2 are the sample variances. With thousands of users per arm the t distribution is practically the normal distribution, so p-values and intervals look like the z versions. Welch's method adjusts the degrees of freedom so it stays reliable when the variances or group sizes differ.

Run it in SciPy

Python Welch's t-test on time spent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always pass equal_var=False. The default (True) is Student's t-test, which assumes equal variances and can mislead when arms differ. Welch's version costs almost nothing when the variances are equal and protects you when they are not.

Revenue: skew and the mass at zero

Revenue per user is the classic hard case. Most users spend nothing (a spike at zero), and a few spend a great deal (a long right tail). The t-test is about means, and means of skewed data are noisy and dominated by a handful of big spenders.

Python A skewed revenue metric

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You see a median of zero, a 99th percentile far above the mean, and a small group of heavy spenders carrying a big share of the revenue. In such data the variance is huge, so the test needs many users, and a handful of big spenders can swing the result. Here the observed lift is not significant even though the treatment truly buys a little more often. Tools for this: capping (winsorizing) extreme values at a high percentile, a log transform (only for positive values and it changes the question to geometric means), the bootstrap (next lessons), and variance reduction techniques.

Winsorizing

Winsorizing replaces values above a chosen percentile (say the 99th) with that percentile. It trades a little bias for much lower variance. Decide the cap before looking at the effect, and apply the same rule to both arms.

Python Capping outliers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The capped version has a smaller standard error, so it can see smaller effects. The cost is that it no longer measures exactly the same mean, which you should state in the report.

Does it matter that the data are not normal?

For comparing means with thousands of users, no, thanks to the central limit theorem: the sample mean is close to normal even if the data are skewed. The issue is not validity but efficiency and robustness: heavy tails mean a bigger standard error and sensitivity to outliers. With very small samples or extreme skew, prefer the bootstrap or a rank test.

Paired designs are different

If the same unit is measured twice (before and after), you use a paired t-test on the differences. In a standard A/B test each user belongs to one arm, so the two samples are independent and ttest_ind is right.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why prefer Welch's t-test over Student's in A/B tests?

    Welch's test adjusts for unequal variances and group sizes, protecting the error rate at little cost.
  2. 2. What is the main problem with revenue per user as a metric?

    A few large purchases dominate the average, inflating variance and reducing power.
  3. 3. What does winsorizing do?

    Capping outliers reduces variance at the cost of a small bias; the rule should be fixed in advance and applied to both arms.
  4. 4. With 20,000 users per arm and skewed data, why can the t-test for means still be valid?

    Sample means of large samples are close to normal even for skewed data, though heavy tails still inflate the standard error.

Summary

  • Use Welch's t-test (equal_var=False) to compare means of continuous metrics; report the interval.
  • Revenue-like metrics are skewed with many zeros; outliers drive variance.
  • Cap outliers with a pre-registered rule, or use the bootstrap or variance-reduction methods.
  • For large samples, the normal approximation of the mean makes the t-test valid even for skewed data.

Next: Chi-Square and Fisher's Exact Test