Small changes need a lot of traffic
Button colors and single-word swaps make easy tests, but they rarely move conversion enough to measure. The smaller the difference you’re trying to detect, the more visitors you need.
A rough guide, assuming your page converts at about 2% and you want the usual 95% confidence and 80% power:
| Lift you want to detect | Visitors needed per variation |
|---|---|
| +20% (2.0% → 2.4%) | about 21,000 |
| +50% (2.0% → 3.0%) | about 3,800 |
Most stores won’t push 40,000 visitors through a single test before Black Friday is over. The practical move is to test changes big enough to produce big differences.
Test in this order
- The offer. Percentage off vs. dollars off vs. a bundle vs. a free gift. The offer usually matters more than how it’s described.
- The headline and opening promise. Lead with the outcome, the problem or the offer — and match the ad that sends the traffic.
- The page structure. A short page that goes straight to the buy button vs. a longer page that handles objections first. These are different pages, not tweaks.
- Proof and where it sits. Reviews, guarantees, press or comparisons, and whether they appear before or after the price.
- The details. Button copy, imagery, colours. Worth testing once the big things are settled and traffic allows.
Keep each test clean
- Test one idea at a time. A variation can change many elements, but they should all express one hypothesis — “bundle-first” or “guarantee-first” — so you know what you learned.
- Split the same traffic. Send the same ad and audience to both pages. If different ads feed different pages, you’re testing the ads too.
- Choose the goal before you start. Purchases or revenue per visitor, measured over an agreed window. Changing the metric afterwards is how teams talk themselves into false winners.
- Run full weeks. Weekday and weekend shoppers behave differently, so seven days is a sensible minimum even if one version looks ahead early.
When there isn’t time to wait
Black Friday compresses everything. If you can’t reach a confident result before the sale, test in the weeks leading up to it, carry the winner into the sale, and keep testing smaller ideas during it with the traffic you have.