Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 38 ยท Projects and Practice

Project: Plan, Run and Report a Marketing Experiment

A hands-on project: design a landing page experiment from hypothesis to sample size, analyze simulated results with a seeded script and write a decision memo.

This project walks through one complete experiment as a marketer would run it: a hypothesis, a plan, results and a decision memo. You will use a seeded simulation, so the numbers are reproducible, and the same code pattern works with real exports from your testing tool. It builds on A/B testing and conversion rate optimization.

The situation

A company sells online accounting software. Its trial sign-up page converts 5 percent of visitors. Heatmaps and interviews show that visitors do not understand what happens after they sign up. Support also hears, "Do I need a credit card?"

Step 1: Write the experiment brief

  • Hypothesis: Because visitors are unsure about commitment, adding the line "No credit card needed, cancel anytime" under the button and a three-step "how it works" strip will raise trial sign-ups.
  • Primary metric: trial sign-up rate (sign-ups / unique visitors).
  • Guardrails: trial-to-paid rate after 30 days must not fall by a relevant amount; page load time must not increase.
  • Audience: all new visitors from paid and organic sources, split 50/50, tracked consistently per visitor.
  • Minimum detectable effect: a 15 percent relative lift (5.0 to 5.75 percent) is worth acting on.
  • Duration: whole weeks, set from the sample size.

Step 2: Calculate sample size

JS Plan the sample size

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Commit to this duration. Looking at results early is allowed for checking that tracking works, not for deciding.

Step 3: Simulate the experiment run

Here we create data with a seeded random generator so everyone gets the same result. In real life you export visitors and conversions from your tool.

JS Simulate and analyze

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Real data would differ, and a smaller true effect might not reach significance. Run the code with a different seed or a smaller true lift to see how often a real improvement looks inconclusive; that is why the sample size matters.

Step 4: Check data quality first

Before reading the result, run quality checks:

JS Sample ratio and sanity checks

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Step 5: Guardrails and segments

A lift in sign-ups that brings low-quality trials is a false win. Check the guardrail (trial-to-paid after 30 days) once enough time has passed, and look at segments (mobile versus desktop, paid versus organic) as exploratory observations. Segment results are hypotheses for the next test, not conclusions, because splitting data many ways produces chance differences.

Step 6: The decision memo

Write one page:

  1. Summary: what we tested, the result and the decision ("Ship variant B to all traffic").
  2. Result: sign-up rate for both versions, relative lift, confidence interval, sample size, dates.
  3. Quality checks: split ratio, tracking, external events.
  4. Guardrails: trial-to-paid and speed.
  5. What we learned: for instance, "Reducing perceived commitment helps; test the same on the pricing page."
  6. Next experiments: ranked with ICE (see CRO).
  7. Estimated value: extra trials per month x trial-to-paid x LTV (see unit economics).

Add the result to your test log so the organization remembers it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the sample size decided before starting the test?

    A fixed plan prevents peeking and stopping at a lucky moment.
  2. 2. What does a sample ratio mismatch indicate?

    An unexpected traffic split suggests broken randomization.
  3. 3. Why check the trial-to-paid guardrail?

    A lift in the top metric is only valuable if downstream quality holds.
  4. 4. How should segment findings be treated?

    Many segment cuts produce chance differences, so confirm them with new tests.

Summary

  • A good experiment starts with a hypothesis, one primary metric, guardrails and a planned sample.
  • Run whole weeks, check data quality and sample ratio before reading results.
  • Report lift with a confidence interval, not only a p-value.
  • Treat segments as ideas; decide from the primary metric and guardrails.
  • Finish with a short memo and a test log entry.

Next: Marketing Practice Problems and Interview Questions