Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 Β· Pitfalls and Ethics

Novelty, Primacy and Interference

Why a lift can fade after launch: novelty and primacy effects, plus network and marketplace interference. Detect them with daily lift plots, cluster randomization and runnable simulations.

A test result is a measurement of what happened during the test, to the users in it. Two common pitfalls break the link between that measurement and what you will see after launch: the effect changes over time, and users in different groups affect each other. This lesson covers both.

Novelty and primacy effects

  • Novelty effect: people click on something new simply because it is new. The lift is big in week 1 and fades as the novelty wears off. Shipping on the week-1 number overstates the long-term gain.
  • Primacy (change aversion) effect: people are annoyed by change and perform worse at first, then adapt. The measured lift is too low early and rises later. Stopping early kills a good idea.

Returning users show both effects; brand-new users, who have no habit to break, usually show neither. That gives you a diagnostic.

Effects that move over time

Novelty Starts high, fades
Primacy Starts low, recovers
New users Cleaner signal, no habits

Plot the daily lift, always

1 / 2

Detecting it: plot lift by day

If the effect is stable, the daily lift hovers around a constant. A trend means trouble.

Python A novelty effect in daily lift

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first week shows a lift of about 1.2 points and the whole-test average about 0.7, but the last week is only about 0.3 points (daily values are noisy, so look at weekly averages): that is the number to plan around.

How to handle it:

  1. Run longer than the novelty period, at least one or two full weekly cycles, and report the last-week lift.
  2. Segment by new versus returning users. A gap between them hints at novelty or primacy.
  3. Use holdbacks. Keep a small percentage of users on the old experience for weeks after launch, so you can measure the long-term effect.
  4. Mind the weekly cycle. Even without novelty, weekday and weekend users differ, so run in whole weeks.

Interference: when users affect each other

A/B tests assume the stable unit treatment value assumption (SUTVA): a user's outcome depends only on their own assignment. That fails often:

  • Social networks: if you give some users a better sharing feature, their friends (in either group) are affected.
  • Marketplaces: if treated buyers grab more of a limited supply, control buyers face fewer items. Treated sellers who are cheaper take sales from control sellers.
  • Shared resources: a ranking change that makes some users consume more server capacity slows everybody down.

In these cases the test result compares users who are all living in the same mixed world, which is not the same as a world where everyone has the feature.

Python Spillover hides part of the effect

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With user-level randomization both arms enjoy the same spillover (half their friends are treated), so it cancels out and the test sees only the direct effect. Randomizing whole clusters captures the full effect that a real launch would have.

Designs that handle interference

  • Cluster randomization: randomize connected groups (friend circles, cities, sellers and their buyers). Costs power, since there are fewer independent units.
  • Switchback tests: alternate treatment and control over time on the same system (for example, ride-sharing in a city alternating every hour). Good for marketplaces; needs enough periods and care with carry-over.
  • Geo experiments: treat whole regions, using matched control regions.
  • Two-sided designs: randomize buyers and sellers independently and analyze the interaction.

You cannot detect interference from a plain A/B result alone. Think about how your product connects users and ask: "if the treated group did well, could that have hurt (or helped) the control group?"

Check your understanding

Quick check Β· 4 questions

of 4 answered You scored / 4 Β· perfect!

  1. 1. What does a novelty effect look like in daily lift?

    People try the new thing eagerly at first, then habituate.
  2. 2. Which group is the cleanest for judging long-term effects of a UI change?

    New users cannot show change aversion or novelty relative to an old experience.
  3. 3. In a marketplace test, buyers in the treatment group win more items, leaving fewer for control buyers. What is violated?

    Outcomes of control users depend on treatment assignment of other users.
  4. 4. Which design randomizes whole friend groups to capture spillover?

    Treating connected groups together keeps spillover inside the group, so it counts toward the measured effect.

Summary

  • Effects can change over time: novelty fades, primacy recovers. Plot daily lift and prefer the last-week figure.
  • Run whole weeks, segment new versus returning users and keep holdbacks for long-term effects.
  • Interference breaks the assumption that users' outcomes are independent of others' assignments.
  • Cluster, switchback and geo designs deal with interference at the cost of power.

Next: Multiple Comparisons and Segment Hunting