Some infrastructure work never touches a line of product code and still quietly changes what users experience — a service mesh setting, a change to which compute instances back a node pool, anything that shifts how fast a request comes back. That raises a question worth taking seriously before dismissing it: does infrastructure deserve A/B testing at all, or is that overkill for something users can't consciously feel?
What Counts as "Infra" Here
Two concrete examples worth having in mind. First, a service mesh configuration change — something like an Istio setting. Istio sits between microservices and manages how requests get routed, retried, load-balanced, and timed out, without the application code itself changing at all. Tweak a retry policy or a timeout threshold, and request latency can shift even though nobody touched the service that's actually handling the request.
Second, a change to the compute instances behind a node pool — swapping which EC2 instance family backs a Kubernetes node pool, for example. Different instance families have different CPU, memory, and network characteristics, so the exact same application code can run measurably faster or slower purely because of what it's running on.
In both cases, the thing that moves is something like API call response time — the round-trip from a request going out to a response coming back. Nobody sees "response time" directly; what they experience is a page or a feature feeling a little faster or slower than it did yesterday, if they notice anything at all.
The Objection: This Is Too Subtle to Matter
The natural first reaction is skepticism. These changes typically move latency by single-digit or low double-digit milliseconds — nowhere near anything a person would consciously notice, and plausibly too small to show up reliably in noisy, aggregate business metrics like conversion or revenue per user. Building out proper experiment infrastructure — traffic splitting, guardrails, analysis pipelines — for something this small can feel like a lot of machinery for a change nobody will ever feel.
The Rebuttal: Monitor the Infra Metric Directly
The objection assumes the only thing worth measuring is a downstream business metric. It isn't. Splitting traffic across the two infra variants — half the requests through the new Istio config or the new node pool, half through the old — and watching the infra metrics themselves (p50/p95/p99 latency, error rate) directly is cheap and immediately informative. You don't need a fully powered, business-metric-level experiment to find out whether the infra change moved the thing it was actually meant to move.
The Point That Actually Settles It
Kohavi, Tang, and Xu's Trustworthy Online Controlled Experiments documents a controlled experiment where a 100-millisecond increase in latency reduced revenue by roughly 0.6%. That's not a rounding error. At any real scale, a fraction of a percent of revenue is genuine money, moved by a delay most users would never consciously register.
This reframes the original objection rather than answering it directly: "too subtle to matter" was measuring subtlety against human perception, not against what actually moves outcomes. A change nobody notices can still move revenue, because the mechanism doesn't run through conscious awareness — it runs through abandonment rates, session continuation, and dozens of small behavioral margins that don't require anyone to say "this feels slow." Infra latency is exactly the kind of thing that's dangerous to dismiss precisely because it doesn't need to be perceptible to be consequential.
A Cheaper First Step Before Building the Full Thing
Given that latency clearly can matter, but building a fully randomized, properly powered infra-level A/B testing pipeline is a real engineering investment, what's a reasonable first check before committing to it?
Look at the infra metric itself — API response time, say — at fine granularity, second by second or minute by minute, in a tight window right around the moment the infra change actually deploys. Compare the period just before the deployment to the period just after it.
This is, conceptually, a regression discontinuity design: treat deployment time as the running variable, the deployment moment as the cutoff, and read a level jump in the metric right at that cutoff as evidence of an effect — under the standard RDD assumption that nothing else relevant is changing discontinuously at that exact instant besides the deployment itself.
The caveat matters as much as the technique: this before/after window is not a substitute for genuine randomization. It can be confounded by anything else that happens to coincide with the deployment — a simultaneous traffic spike, an unrelated concurrent deploy, a time-of-day effect if the window is too wide to be a clean cutoff. It's a fast, low-cost first pass for deciding whether a change is worth chasing further — not a final causal answer on its own.
Putting the Sequence Together
Don't dismiss infra changes as too subtle to matter — even an imperceptible latency shift can move revenue at a fraction-of-a-percent scale that's real money at any meaningful volume. Before investing in a full randomized infra experiment, run the cheap version first: a tight before/after window around the deployment moment, read the way a regression discontinuity design would read it. If that shows a real jump, it justifies the investment in a proper controlled experiment with traffic split by variant. If it doesn't, you've saved the effort — and either way, you found out fast.