Performance measurement looks like the most objective work in search. It is among the least, because the instrument has more degrees of freedom than the thing being measured. Five of them accounted for every wrong number we produced on one engagement.
The five
- Cache state. A warm cache and a cold cache are different sites. A single run tells you which one you happened to load.
- Consent. With a banner unaccepted, the third-party scripts behind it never execute. The number describes a page no visitor sees.
- Third parties. Tag managers, pixels and chat widgets load on their own schedule. Two runs a minute apart are not the same page.
- Run count. One run is an anecdote. The variance between runs on the same URL routinely exceeds the effect size being investigated.
- No interval. A difference quoted without a confidence interval cannot be distinguished from the noise in the four defects above.
What controlling them produced
On a live A/B of two third-party pixels — 20 runs per arm, cache state and consent fixed, everything else held — the result was not the one a single-variable test would have found:
| Arm | TBT | LCP | Performance score |
|---|---|---|---|
| Google pixel gated | −637 ms (p<0.0001) | −1.94 s (p<0.0001) | +1.0 — not significant (p=0.246) |
| Meta pixel gated | −817 ms (p<0.0001) | −0.52 s (p=0.054) | +1.0 (p=0.037), CI [0, +2] |
| Both gated | −1,640 ms (p<0.0001) | −4.17 s (p=0.0001) | +6.0 (p=0.0017), CI [+5, +9] |
Neither pixel alone moves the score. Gating both moves it six points. A test that changed one variable at a time — the textbook approach — would have concluded there was nothing worth doing here, twice.
The protocol
Fix the cache state and state which one. Accept or block consent explicitly and say which. Enumerate the third parties present in each arm. Run at least twenty times per arm. Publish the interval, not just the delta. If any of the five is uncontrolled, the honest output is not a smaller number — it is no number.
Sources — Lighthouse, 20 runs per arm, cache and consent state fixed, third parties enumerated per arm. 30 August 2026.