Two people can look at the same test and declare different winners. One points to conversion rate, the other to revenue, and both are reading their numbers correctly.
That isn't a statistics problem. It's a decision that should have been made before the test started: which metric decides, and which ones are only there to catch problems.
What each metric actually measures
Each common metric answers one narrow question. Trouble starts when you treat it as the answer to a bigger one.
CTR (click-through rate)
What it tells you: how well the ad earns a click from the people who saw it.
Where it misleads: CTR rewards curiosity. A vague or provocative hook can pull lots of clicks from people who never meant to buy. Those visitors drag down conversion on the page, and the "winning" ad ends up with a worse cost per purchase. Meta also reports more than one version of CTR, such as all clicks versus link clicks, so make sure you're comparing the same one.
CTR is a useful diagnostic for ads, especially early, when purchases are too sparse to judge. It's rarely the right metric to decide a test.
Conversion rate
What it tells you: the share of visitors who buy. For page tests, it's the most direct read on whether the page persuades.
Where it misleads: conversion rate ignores order value. A page that pushes a single low-priced item can convert more visitors and still bring in less money than a page that sells a bundle. It also depends on the denominator. Sessions, users and clicks give different rates, so define it once and stick with it.
CPA (cost per acquisition)
What it tells you: how much ad spend each purchase cost. It folds the ad's efficiency and the page's conversion into one number.
Where it misleads: CPA treats a $30 order and a $150 order as equal. It also moves with auction conditions that have nothing to do with your test, which is one reason page tests should split traffic after the click rather than compare separate campaigns. Testing ads and landing pages together covers how to set that up.
ROAS (return on ad spend)
What it tells you: revenue per dollar of ad spend.
Where it misleads: in two ways. First, it ignores margin, so a variation that leads with a deep discount can win on ROAS and still lose money. Second, the revenue side depends on attribution. Meta, Shopify and GA4 often report different numbers, and attribution windows and view-through credit can shift results between variations. Attribution when platforms disagree goes deeper on this.
Per-visitor metrics usually work better for page tests
For page tests, two metrics capture more of what you care about than conversion rate alone.
Revenue per visitor (RPV) is total revenue divided by visitors. It moves when conversion rate moves and when order value moves, so it catches the bundle-versus-single-item problem.
Profit per visitor goes one step further: revenue minus product cost, discounts, shipping and payment fees, divided by visitors. It's the right metric when variations differ in offer, price or discount, because that's exactly where revenue and profit split apart.
Here's an example with made-up round numbers. Ignore sample size for a moment; this is only about what each metric measures.
- Page A gets 1,000 visitors and 30 orders at an average of $60. Conversion rate is 3.0%, revenue is $1,800, and RPV is $1.80.
- Page B leads with a discounted bundle. It gets 1,000 visitors and 26 orders at an average of $80. Conversion rate is 2.6%, revenue is $2,080, and RPV is $2.08.
On conversion rate, A wins. On RPV, B wins. Now add margin: say A keeps a 60% gross margin and the discounted bundle keeps 45%. A earns $1,080 in gross profit, or $1.08 per visitor. B earns $936, or about $0.94 per visitor. On profit, A wins again.
Three metrics, and the winner flips twice.
One caution: revenue and profit metrics are noisier than conversion rate, because order values vary from one customer to the next. They need more traffic to reach a confident result, and they need a different statistical test than a simple conversion-rate calculator uses. If your testing tool reports RPV with confidence intervals, use those.
Match the metric to the test
The right primary metric depends on what's changing.
- Page tests with similar offers: conversion rate or RPV. If the variations could change order size, use RPV.
- Page tests that change the offer, price or discount: profit per visitor, or RPV with margin as a guardrail.
- Ad tests: cost per purchase or purchase value per dollar spent, measured on the same landing page. Use CTR and cost per click to diagnose why an ad won or lost, not to pick the winner.
- Matched ad-and-page angle tests: cost per purchase or gross profit per dollar spent, since you're judging the whole path from impression to order.
For how these connect to the health of the business overall, see CAC, MER, ROAS and contribution margin.
Pick one primary metric and a few guardrails
Before the test starts, write down two things.
One primary metric. This is the number that decides the test. One, not three. Choosing the metric after you've seen the results gives luck extra chances to hand you a winner, the same problem as stopping a test early because it looks good.
Two to four guardrails. These don't pick the winner, but they can veto one. Useful guardrails include:
- Average order value, so a conversion-rate win doesn't hide smaller baskets.
- Discount rate or margin per order, for anything touching the offer.
- Refund or return rate, checked once enough time has passed to see it.
- Subscription opt-in rate, for brands that sell subscriptions.
- Mobile load time, for page variations that add heavier sections.
Then write the decision rule in one sentence. For example: "Ship page B if RPV is higher at 95% confidence and average order value doesn't fall by more than 5%." The threshold is yours to set. What matters is that it exists before anyone sees a result.
The short version
- CTR measures the click, not the customer. Use it to diagnose ads, not to decide tests.
- Conversion rate ignores order value, CPA ignores order size, and ROAS ignores margin while leaning on attribution.
- For page tests, revenue per visitor or profit per visitor usually gives the most honest answer.
- Choose one primary metric and a few guardrails before launch, and write the decision rule down.