Ask when a test is done and you'll usually hear one of two answers: "when it hits significance" or "after a couple of weeks." Both sound reasonable. Both lead to bad calls.
The first stops tests on lucky streaks. The second ignores how much traffic you actually have. The right length comes from a number you set before the test starts, plus a few calendar rules that keep the result honest.
Decide the sample size before you launch
A test needs enough visitors to tell a real difference from noise. How many depends on three inputs you choose up front:
- Baseline conversion rate. What the current page converts at for the traffic you'll test. Pull it from the same campaigns and the same type of visitor, not a sitewide average.
- Minimum detectable effect. The smallest lift worth knowing about. Pick the lift that would change a decision, not the lift you hope for.
- Confidence and power. The usual standard is 95% confidence and 80% power. In plain terms: a low chance of calling a winner that isn't real, and a good chance of catching one that is.
Plug those into any two-proportion sample size calculator and you get the visitors needed per variation. Divide by the daily traffic you can send to the test and you have a duration.
Here's an example with made-up round numbers. Say your page converts at 3% and you want to detect a 20% relative lift, from 3.0% to 3.6%. That takes about 13,900 visitors per variation, or roughly 27,800 in total. If the page gets 2,000 paid visitors a day split evenly, you're looking at about two weeks.
Now say you want to detect a 10% lift instead, from 3.0% to 3.3%. The requirement jumps to about 53,000 visitors per variation. Same page, same traffic, and the test now takes close to eight weeks. Halving the lift you want to detect roughly quadruples the sample.
That's why the order you test things in matters so much when traffic is limited. Small tweaks need samples many stores can't reach.
Run whole weeks
Shoppers behave differently on a Tuesday morning than on a Sunday night. Paydays, weekend browsing and lunch-break scrolling all change who shows up and how ready they are to buy.
A ten-day test counts some days of the week twice and others once, so the result over-represents whichever days got the extra turn. If one variation happens to suit weekend browsers, a test with two weekends and one set of weekdays will flatter it.
So:
- Run at least one full week, even if the sample size math says fewer days. Two full weeks is a better floor.
- Round up to the next full week once you've hit the sample size. If you reach it on day 16, run to day 21.
- Start and stop on the same weekday so every day is counted the same number of times.
Watch the calendar too. A test that overlaps a sale, a holiday or a big email send is measuring shoppers in an unusual state. Either avoid those windows or note them and plan to re-test.
Don't stop when it first looks significant
This is where a lot of tests go wrong. You check the dashboard on day three, one version is ahead at 95% confidence, and you call it.
The problem is that the confidence number assumes you looked once, at the planned end. Results swing hard early on, when each side has only a handful of conversions. Check often enough and one of those swings will cross the line by chance.
The math isn't subtle. If two pages are actually identical and you check every day for two weeks, stopping the first time either one shows 95% confidence, the odds of declaring a false winner are roughly one in five. Not one in twenty. It's also one of the main reasons test winners stop winning after rollout.
A few ways to handle it:
- Look, but don't act. Check for broken tracking or a variation that isn't loading. Don't make a winner call before the planned end.
- Use a tool built for checking as you go. Some testing tools use sequential statistics designed for continuous monitoring. If yours does, follow its stopping rules rather than your gut.
- Write the stopping rule down before launch: the sample size, the end date and the primary metric. Picking that metric is its own decision, covered in choosing the metric that decides a test.
Use conversions per variation as a sanity check
Visitor counts can look impressive while the number of actual purchases is tiny. Before you trust any result, look at how many conversions each side has.
Run the sample size math at a 1%, 2% or 3% baseline and a pattern appears: the number of conversions you need barely changes. Detecting a 20% lift takes roughly 400 to 500 purchases per variation. Detecting a 50% lift takes roughly 75 to 115.
So a test that's "winning" with 30 purchases on each side hasn't proven anything except that it needs more time. Treat the conversion count as a floor you check alongside the visitor count, not a replacement for the sample size calculation.
When traffic is too low
Plenty of stores run the numbers and find a meaningful test would take months. You still have options.
- Test bigger changes. A different offer, a different angle, a different page structure. Bigger differences produce bigger effects, and bigger effects need far smaller samples. In the 3% example, detecting a 50% lift takes about 2,500 visitors per variation instead of 13,900.
- Read a higher-funnel metric as a directional signal. Add-to-cart rate and checkout starts happen more often than purchases, so they settle faster. Use them to spot a likely loser early or to choose what to test next, not to declare a winner. A page can win add-to-carts and lose purchases if it attracts window shoppers.
- Accept a longer test, with care. Four to six weeks is workable if nothing else changes underneath it. The longer it runs, the more likely a new ad, a promo or a seasonal shift muddies the comparison, so log anything that changes during the test.
- Concentrate traffic. Run one test at a time on the traffic you have, with two variations rather than four. Every extra variation splits the sample further.
- Lower the bar on purpose, if the stakes are low. Accepting 90% confidence for a cheap, easily reversed change is a legitimate business call. Make it before the test, and know you're trading more false winners for speed.
The short version
- Set the baseline, minimum detectable effect, confidence and power before launch, and let that sample size set the length.
- Run whole weeks, start and end on the same weekday, and avoid sale windows unless the test is about the sale.
- Don't stop the first time a result looks significant. Daily peeking turns a one-in-twenty false-positive risk into something closer to one in five.
- Check conversions per variation before trusting any result.
- With low traffic, test bigger ideas, treat higher-funnel metrics as directional, and log what changes during a longer test.