You suspect that your brand ads on Google mostly catch people who would have found you anyway. Attribution does not help here. It gives these ads many orders because they sit just before the purchase. Whether those orders would have happened without the ads, it cannot say.
The idea
An on-off test switches the advertising on and off in turns, in periods of equal length, and compares orders or leads per visit. You switch at least twice, for example one week on, one off, one on, one off. If only one week with advertising stood against one without, every difference between those two weeks would go into the result, including the one between the start and the end of the month.
A worked example
Suppose that in the weeks with advertising you count 4,000 visits and 120 orders, and in the weeks without, also 4,000 visits and 100 orders. That is 3.0 against 2.5 per cent, a difference of 0.5 percentage points. Looks like a result.
| Visits per side | With advertising | Without advertising | Difference in percentage points | 95 per cent interval |
|---|---|---|---|---|
| 4,000 | 3.0% (120) | 2.5% (100) | +0.5 | −0.2 to +1.2 |
| 16,000 | 3.0% (480) | 2.5% (400) | +0.5 | +0.1 to +0.9 |
The interval shows how precisely the difference is measured. Metrunio calculates it at 95 per cent, using the standard comparison of two proportions. With 4,000 visits per side it runs from minus 0.2 to plus 1.2 percentage points and includes zero. The result is then “no measurable difference”: the gap cannot be told apart from ordinary fluctuation. Only with four times as many visits does the whole interval lie above zero.
“No measurable difference” is a full result. Metrunio then extrapolates nothing and states no additional orders. If the interval lies entirely above zero, the result states them: in the example 0.5 per cent of 16,000 visits, so 80 orders in the weeks with advertising. Below 100 visits per side, or with fewer than ten conversions in total, Metrunio does not calculate at all and says the basis is missing.
What no calculation can check
The test assumes the periods differ only in the advertising. A discount campaign, a public holiday, a newsletter in one of the weeks with advertising: all of it ends up in the result. So plan the switches around such days. And write down what you expect before you read the result; Metrunio asks for this hypothesis when you create the test, and the caveat stays attached to every result.
For two set-ups in the same period there is a second design, the comparison of two campaign groups. It says which of the two did better. What would have happened without advertising, it does not say, because both groups are advertised.
The on-off test comes with the Growth plan, the comparison of two campaign groups with Scale and the agency plan. On Start and Pro, “Experiments” appears in the navigation with a lock, and the page says what it does.
Idea clave
120 against 100 orders looks like a result. With 4,000 visits per side it is not one.