Skip to content

Marketing & Growth Systems

Experimentation: decide before you look

Run a test, check it on day three, see a winner, ship it. A month later the numbers look like they did before. A useful test is a plan written before it starts: one change, one metric, a guardrail, how many people need to see it, and what result means ship. Then the decision is made by the plan, not by whoever looked first.
Discuss your systemSee the plan ↓
A7 fieldsGet a quotetarget · 400 visitors eachday 3 · A 6/60 · 10.0%day 14 · A 41/402 · 10.2%B4 fields, photo afterGet a quotetarget · 400 visitors eachday 3 · B 12/58 · 20.7%day 14 · B 54/398 · 13.6%not yet · 58 of 400day 3 · A 6/60 · 10.0% · B 12/58 · 20.7% · not yet · 58 of 400guardrail · booked rate A 48% · B 46% · holdsdecision · ship B

The plan comes first

Test plan · quote form · written Sep 29

change
7 fields → 4; photo asked for after submitting
metric
quote requests ÷ visitors to the form
guardrail
share of requests that book: no more than 5 points lower
sample
400 visitors per version
longest run
3 weeks
decision rule
ship B if it's at least 2 points better and the guardrail holds
owner
marketing lead

Every line is decided before the first visitor arrives. Nobody can move the finish line after seeing the score.

The guardrail matters as much as the metric. A shorter form that brings more requests but fewer bookings isn't a win.

Four ways tests mislead

  1. 01

    Peeking

    Stopping at the first good day

    Early results swing wildly. Decide at the planned sample, not when the chart looks good.

    “let's just check how it's going.”

  2. 02

    Many changes at once

    Which one worked?

    New headline, new colour and fewer fields in one test tells you the bundle won, not why.

    a redesign called a test.

  3. 03

    The wrong metric

    Clicks aren't customers

    Measure the thing you want more of, and guard the thing you can't afford to lose.

    wins on a number nobody is paid on.

  4. 04

    Too few people

    Small numbers lie

    With a few dozen visitors, chance looks like a result. Some sites can't test small changes at all.

    tests that need a year to finish.

Test it, or just change it

Something is broken or confusingFix it. No test needed
Fewer than about 100 visitors a week to the pageMake the change and watch the trend
The change is expensive or riskyTest it first
Two good options and no clear favouriteTest them

What a test hands off

In

The plan
change, metric, rule
Events
visits, requests, bookings
Assignment
who saw which version
Test
split → count → apply the rule

Out

Decision
recorded against the plan
Results
with their counts
Guardrail
checked before shipping

Worth testing when the answer isn't obvious

Good fit

  • A page or form gets a few hundred visitors a week or more.
  • The change could plausibly go either way.
  • You can measure the outcome you care about, not just clicks.

Not the right fit

  • The traffic is too low to reach a sample in reasonable time.
  • The current version is plainly broken. Fix it.

Where tests run

Next to experimentation

Questions about experimentation

How long should a test run?

Until it reaches the sample you planned, and at least a full week so weekdays and weekends are both in. Decide that before it starts.

We don't get much traffic. Can we still test?

Only big changes, and slowly. Often it's better to make the change, watch the trend, and save tests for pages with more visitors.

What tools do you use?

Often PostHog or the experiment features in your existing tools. Starting candidates, not requirements; the plan matters more.

Can a test hurt us?

That's what the guardrail is for. If the protected number drops past its limit, the test stops.