Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 37 ยท Experiment Design

Randomization and Assignment Units

Learn how A/B tests assign users to variants: choosing the randomization unit, deterministic hash bucketing, stratified and cluster assignment, and common pitfalls.

Randomization looks trivial (flip a coin) until you try to do it in production. Users return on different days, on different devices, logged out and logged in. The same person must see the same variant every time, assignment must be fast and not need a database lookup, and you must be able to explain exactly who was in which group afterwards. This lesson covers the design choices that make that work.

Choosing the unit of randomization

The unit is the thing that gets assigned. Common choices:

Unit Good for Watch out for
User / account ID Most product changes; consistent experience across sessions Needs a stable ID (logged-in users)
Cookie / device ID Anonymous visitors, landing pages One person on two devices lands in two groups; cleared cookies re-randomize
Session Short-lived effects such as ranking within a visit A returning user may see both versions
Page view / request Changes with no memory (ad creative) Inconsistent experience, contaminated effects
Cluster (store, city, class) Effects that spill over between people Few clusters means low power

Rule of thumb: pick the unit that matches how the change is experienced, and keep the experience consistent for that unit. If a price change is shown by user, never show the same user two prices.

Deterministic hashing

Production systems rarely store "user 123 is in B." Instead they compute the assignment from the user ID with a hash function. The same input always gives the same bucket, no storage is needed, and assignment is identical across servers. Adding an experiment name (a "salt") keeps assignments in different experiments independent.

flowchart LR
    ID["user_id + experiment name"] --> H["hash function<br/>(e.g. MD5, SHA-256)"]
    H --> N["number 0..9999"]
    N --> B{"bucket < 5000?"}
    B -- yes --> A["variant A"]
    B -- no --> T["variant B"]

Hashing turns an identifier into a repeatable, evenly spread number.

Python Hash-based bucketing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Hash assignment means a user always gets the same variant, assignments are reproducible during analysis, and ramping up (changing the split from 10% to 50%) can be done so that users already in B stay in B if you design the buckets as ranges.

Independence between experiments

If two experiments use the same hash salt, users in A of one are also in A of the other, and their effects tangle together. Using the experiment name as part of the key makes assignments statistically independent. Check it:

Python Are two experiments independent?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With different salts, about half the B users of one experiment are in B of the other, which is what independence looks like. With a reused salt it is 100%, and the two experiments are completely confounded.

Stratified and blocked randomization

Simple randomization balances traits only on average. With a small sample, an unlucky split can put most big spenders in one arm. Stratified randomization randomizes separately inside groups (strata) such as country or past-spend bucket, which guarantees balance on those traits. Analysis then accounts for the strata (Variance Reduction shows how that sharpens the test).

Clusters and interference

Standard tests assume one unit's outcome does not depend on another unit's assignment (the no-interference assumption). It breaks when users influence each other: a social feed, a two-sided marketplace (more buyer demand with variant B eats supply for the control), a shared-driver pool. The fix is to randomize at a higher level (city, group, time slice) at the cost of fewer units and less power.

Practical checks

  • Stable ID: confirm the ID does not change at login. Switching from cookie to user ID mid-experiment can move people between arms.
  • Assign at the right moment: log exposure when the user actually reaches the changed experience, so you do not dilute the effect with users who never saw it (the next lesson shows how to check balance).
  • Log the assignment: store user, experiment, variant and timestamp at assignment time, not reconstructed later.
  • Exclude bots and employees consistently in both arms.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why include the experiment name in the hash input?

    Salting by experiment name makes each experiment's split statistically independent of the others.
  2. 2. You are testing a feature in a marketplace where buyer behavior in B changes what is available to buyers in A. Which approach is safest?

    Interference violates the independence assumption. Randomizing clusters keeps the interacting units in the same arm, at the price of fewer units.
  3. 3. What is the main advantage of deterministic hash assignment?

    Hashing gives repeatable assignment across servers and sessions without storing per-user state.
  4. 4. What is stratified randomization for?

    Randomizing within strata ensures each arm has the same mix of those traits, which reduces noise and avoids unlucky imbalances.

Summary

  • Pick the randomization unit that matches how users experience the change and keep it consistent.
  • Hash the ID plus an experiment name for stable, independent, storage-free assignment.
  • Stratify to guarantee balance on key traits; randomize clusters when users interfere.
  • Log assignment and exposure at the right moment.

Next: Sample Ratio Mismatch