The test looked clean. Variation B beat the control, you rolled it out to everyone, and a month later conversion rate is about where it was before. Sometimes it's worse.
This is common enough that it should be expected, not treated as a mystery. A fading winner usually comes down to one of a handful of causes, and most of them can be spotted early or prevented outright.
Regression to the mean
Every test result is part signal, part noise. When you pick the variation with the best result, you're also picking the one that got the luckiest noise. That luck doesn't come with it into the rollout.
So even when a winner is real, its measured lift is usually inflated. Say a test shows a 25% lift, but the test was only big enough to reliably detect large effects. The true effect might be closer to 10%. After rollout you see something like the 10%, and it feels like the winner lost most of its edge. It didn't. The test overstated it.
This effect is strongest in underpowered tests. The smaller the sample, the bigger a lift has to look before it crosses the significance line, so any winner from a small test arrives with an exaggerated number attached.
What to do: size tests properly (how long a test should run covers the math), and when you forecast the impact of a winner, plan on less than the measured lift.
False positives from peeking and too many variations
Some winners were never winners. At 95% confidence, about one in twenty tests of two identical pages will still show a "significant" difference. That's the trade you accept when you pick that threshold.
Two habits make the risk much larger:
- Peeking. Checking results daily and stopping as soon as something hits 95% gives luck many chances to cross the line. Two identical pages checked every day for two weeks will produce a false winner roughly one time in five.
- Many variations. Each comparison against the control is another roll of the dice. With five challengers and no real differences, the chance that at least one shows a "significant" result is close to one in five, not one in twenty.
What to do: set the stopping point before launch, keep the number of variations low, and treat the winner of a five-way test as a candidate for a head-to-head re-test rather than a final answer.
Novelty effects
Returning visitors notice change. A new layout, hero image or offer block can draw attention simply because it's different, and that attention fades once it's familiar.
For cold paid social traffic this matters less, since those visitors are seeing the page for the first time anyway. It matters more for retargeting, email and returning customers. And it matters a lot for ads: a winning ad will wear out as frequency climbs, which looks like a fading winner but is really fatigue. Spotting creative fatigue covers how to tell the difference.
What to do: if a test leans on returning traffic, run it long enough for novelty to wear off, and compare the first week's lift with later weeks. A lift that shrinks week over week is a warning sign.
Seasonality and promo periods
A page that wins during a sale may have won because of the sale. Urgency-heavy layouts, countdown timers and discount-forward headlines can do well when shoppers arrive ready to buy, then underperform in a quiet month when the same visitors need more convincing.
The reverse happens too. A long, objection-handling page might lose in November and win in February.
What to do: log the promo calendar next to every test. If a test ran during a sale, label it a sale-period result and re-test before treating it as the year-round default.
The traffic changed
A landing page result is only true for the traffic that saw it. Change the traffic and the result can change with it.
That happens constantly in paid social:
- New ads with different angles bring people with different motivations. A page built around a pain-point angle may not suit traffic from a gifting angle.
- Broad targeting and Advantage+ campaigns shift who sees your ads as delivery adjusts over time, even when you don't touch the settings.
- New channels and budget moves. Adding TikTok, or shifting budget from prospecting to retargeting, changes the mix landing on the page.
What to do: record which campaigns, ads and audiences fed each test. When the mix changes meaningfully, check the page's performance by traffic source with UTMs before assuming the winner still holds.
The rollout isn't the page you tested
Sometimes the winner stops winning because it isn't the same page anymore. The test version lived in a testing tool; the rollout was rebuilt in the theme. Small differences creep in:
- Speed and layout. A section that loads slower, or shifts around on mobile.
- Missing pieces. A trust badge, review widget or shipping note that didn't make the move.
- Cart behavior. A slide-out cart drawer instead of a cart page, or the reverse.
- Tracking. Events that fire differently, so part of the "drop" is really a measurement change.
What to do: QA the rolled-out page against the tested version side by side, on real phones. Compare load time, every section, cart behavior and tracking before you call the rollout done.
How to protect your winners
None of these causes can be removed completely. You can make them visible.
- Keep a holdout. After rollout, leave a small slice of traffic on the old version for a few weeks. A 10% holdout, for example, gives you an ongoing comparison instead of a before-and-after guess. On a smaller store it won't be precise, but it will flag a big reversal.
- Re-test big wins. If a result will reshape your funnel, run it again, ideally in a different month with fresh traffic. A real effect should show up twice.
- Document the conditions. For every test, record the dates, traffic sources, ads, offer, promo status, sample size, primary metric, and the measured lift with its range, not just "B won." A testing roadmap and log makes this routine.
- Discount your forecasts. Plan on less than the measured lift, especially from smaller tests.
The short version
- Winners from small or peeked-at tests usually overstate the lift, and some aren't real at all.
- Novelty, seasonality and shifts in traffic mix can make a real winner stop working.
- Rollouts often differ from the tested page. QA them side by side.
- Holdouts, re-tests and a written record of test conditions turn "it stopped working" into something you can diagnose.