Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท Experiment Design

Experiment Duration and Novelty Effects

Decide how long an A/B test should run. Learn weekly cycles, seasonality, novelty and primacy effects, ramp-up plans, and how to spot a drifting effect in Python.

"How long should we run it?" has two parts. The first is statistical: you need enough users, which later lessons turn into a number. The second is behavioral: you need the right time window, because users behave differently on Mondays than Saturdays, and a shiny new feature behaves differently in week one than in week four. This lesson covers the time side of the design.

Run whole weeks

Most products have a weekly cycle. B2B tools peak on weekdays; entertainment and shopping may peak on weekends. If your test runs Wednesday to Sunday, weekdays and weekend days are not represented evenly, and the result may not describe the average week.

Rule of thumb: run for at least one full week, and for whole multiples of weeks (7, 14, 21 days), so each day of the week appears equally often. Both arms always run at the same time, so day-of-week effects cancel between arms, but they still limit how well your result generalizes.

Python Weekly pattern in a metric

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A weekday-only window overstates the weekly average here. Full weeks keep the sample representative.

Seasonality and special events

Holidays, sales, paydays, school terms and product launches change behavior. A test during Black Friday may not describe an ordinary week, and the effect of a discount banner will be exaggerated. Avoid launching across big events when possible, or run long enough to include both the event and normal days, and report that context.

Novelty and primacy effects

Two opposite effects distort early results:

  • Novelty effect: users click a new thing because it is new. The lift fades after the curiosity wears off. Early numbers overstate the long-term effect.
  • Primacy (change aversion) effect: regular users resist change and use the new design less at first, then adapt. Early numbers understate the long-term effect.
flowchart LR
    subgraph novelty["Novelty: lift decays"]
      direction LR
      N1["Week 1<br/>+8%"] --> N2["Week 2<br/>+4%"] --> N3["Week 3<br/>+2%"]
    end
    subgraph primacy["Primacy: lift grows"]
      direction LR
      P1["Week 1<br/>-3%"] --> P2["Week 2<br/>0%"] --> P3["Week 3<br/>+3%"]
    end

The effect measured depends on when you measure. Look at it over time before trusting a pooled number.

Existing users are most affected by both. New users have no habit to break, so segmenting new versus returning users is a quick way to tell them apart.

Is the effect stable? Plot it by week

Compute the lift for each week separately and see whether it trends. A flat line suggests a stable effect; a steady decline suggests novelty.

Python A fading (novelty) effect

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weekly lifts shrink, and the pooled figure hides that story: it is higher than what you will actually get once the novelty has worn off. When you see a trend, consider running longer, reporting the final week's effect, or restricting analysis to users first exposed after the novelty period.

Ramp-up and holdback

  • Ramp-up: start at 1-5% traffic to catch crashes and SRM, then go to the full planned split. The analysis uses only the full-split period, because ramping mixes different assignment rates over time.
  • Long-term holdback: after shipping, keep a small (for example 1-5%) group on the old experience for weeks or months to measure long-term effects and detect decay.

Planning the duration

Combine the two requirements: the duration is the larger of "days needed to reach the planned sample" and "one or two full weeks," rounded up to whole weeks.

Python From sample size to run time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should an experiment usually run in whole weeks?

    Behavior differs by weekday, so whole weeks keep the mix of days balanced and the result representative.
  2. 2. A new button gets many clicks in week one and fewer each week afterwards. What is this called?

    Users are drawn to novelty; the lift fades as the curiosity disappears.
  3. 3. What should you do if the weekly lifts show a steady decline?

    A trend means the pooled effect is not the steady-state effect. Examine the time pattern and plan accordingly, documenting the choice.
  4. 4. What is the purpose of a long-term holdback?

    A holdback group stays on the old experience so long-run differences, including decay, can still be measured.

Summary

  • Run whole weeks (at least one, often two or more) so the weekly cycle is represented.
  • Beware holidays and special events; they change behavior.
  • Novelty inflates early effects and primacy depresses them; plot the effect by week.
  • Duration is the larger of "days to reach the sample" and the minimum weeks, rounded up to whole weeks.

Next: The Two-Proportion Z-Test