A holdout compares an eligible group that can receive a treatment with a comparable group withheld from it. For an advertiser, the treatment might be a defined campaign. A well-designed comparison can help estimate incremental effect within the tested population and period. It cannot automatically reveal the value of every campaign, future season or customer segment.
For a small advertiser, the central question is whether the experiment can meaningfully inform a decision at the available scale. This is a design guide, not a claim that a particular account has measured lift or a promise that a small test will produce a decisive answer.
Define the treatment boundary
“Advertising” is usually too broad. Write which campaign, audience, geography and period belong to the test. List other activity that continues in both groups. If the holdout can still receive closely related messages through another campaign, interpretation must account for that exposure.
The unit of assignment also matters. Individuals, households and regions have different practical limitations. A regional comparison can be distorted by market differences or local events. A user-level comparison can be affected by incomplete identity and cross-device exposure. Choose a method whose limitations can be described, not simply the one with the easiest dashboard.
Specify the outcome and decision
A useful test plan separates the outcome from the action. The outcome might be eligible-customer orders or contribution over a defined observation period. The action might be to maintain spending, increase it within a limit, or redesign the campaign. Decide what magnitude of difference would matter commercially before seeing the result.
| Design question | Why it matters |
|---|---|
| Who is eligible? | Defines the population to which the finding applies |
| How is assignment made? | Determines whether groups can be compared credibly |
| What remains unchanged? | Reduces competing explanations |
| When do outcomes mature? | Avoids prematurely closing the observation window |
| What effect would change the decision? | Connects statistical precision to business relevance |
| What could contaminate exposure? | Makes spillover and implementation risks visible |
Check whether precision is plausible
Small purchase counts can leave wide uncertainty even when assignment is sound. Before launching, estimate the baseline event rate, available population and commercially meaningful effect. A power analysis can help assess whether the planned design is capable of detecting that effect under its assumptions. Statsig’s power-analysis documentation describes the relationship between sample planning, effect size and statistical power.
If the available scale is insufficient, do not promise certainty by extending a spreadsheet. Consider a larger decision horizon, a broader but still coherent treatment, or a different research question. An upstream behavior may be easier to observe, but its effect is not equivalent to an effect on purchases or profit.
Protect the comparison during execution
Record assignment integrity, delivery, exclusions and significant business changes. A payment outage affecting one region, a stockout concentrated in one group, or a concurrent promotion can change interpretation. Investigate these events without selectively removing inconvenient observations after seeing the result.
Keep a separate operational stop rule for unacceptable harm or spending. Stopping to protect the business is legitimate; it should not be disguised as a statistically complete test. Preserve the reason and the evidence available at the time.
Interpret uncertainty honestly
A result compatible with both a useful gain and a meaningful loss is inconclusive. It is not proof that the campaign has no effect. Similarly, a positive estimate with substantial uncertainty should not be converted into a precise annual revenue forecast.
Report the estimate, uncertainty, implementation checks and scope together. Discuss whether the test population represents the customers the next budget decision will target. A campaign may create value in the tested setting while requiring a new assessment when spending, audience or offer changes.
The prospecting review sequence places this evidence alongside order economics and customer quality. A holdout is most useful when it answers one bounded question well; it is least useful when an ambiguous result is asked to justify an entire growth strategy.