A category-page experiment removed GIF-preview thumbnails for one arm and kept them for the other. Reviewing the readout, an ML engineer asked two questions about how to analyze it properly. Both sounded like the same kind of question — "can we narrow the analysis down to the users who matter?" One of them had to be rejected outright. The other turned out to be encouragement design, and answering it properly required not narrowing anything down at all.
The Experiment, Briefly
Control kept GIF previews on the category page as normal. Treatment removed them entirely, replacing every card with a static image. At the full population — about 151,000 users in control against 149,000 in treatment — the guardrail metric (orders per active user) came back at (p = 0.02): a real, if modest, negative signal.
Question 1: "Can We Just Analyze Users Who Saw a GIF at Least Once?"
The question: "What if we define activation as 'saw a GIF-preview card at least once' and only analyze that subgroup? Isn't that the cleanest way to isolate the users who could actually have been affected?"
It sounds like exactly the right instinct — why analyze people the treatment could never have touched? The problem is what "saw a GIF card" actually means in each arm. In control, it's a low, passive bar: GIF cards are everywhere, so plenty of ordinary users clear it just by scrolling normally. In treatment, the category page shows zero GIF cards at all — the only way to satisfy the same condition is to have gone looking for one somewhere else in the app entirely. Clearing the bar in treatment requires active, self-selected exploration that clearing it in control never demanded.
The actual counts confirmed the damage: roughly 100,000 users in the control subgroup against 50,000 in the treatment subgroup — a 67:33 split where a legitimate filter should land close to even. A chi-square test rejects equal allocation decisively. That's not a treatment effect. It's survivorship bias — the treatment arm's "activated" users are a biased sample of unusually heavy, exploratory app users, because that's the only kind of user who could clear a bar the treatment itself made nearly impossible to reach locally.
The answer to Question 1 is no. This subgroup breaks randomization the moment it's defined, regardless of what the resulting numbers say. The full post-mortem on exactly how this SRM happened walks through it in detail.
Question 2: "Then How Do We Handle Control Mixing Exposed and Unexposed Users?"
The question that was actually being asked: "Control is really two groups at once — (1) users who saw a GIF card on the category page, and (2) users who didn't happen to see one that session. Treatment only ever has type (2), since GIFs are gone for everyone there. Doesn't having type (2) mixed into control dilute the result? What analysis method accounts for that?"
This is a genuinely different question from the first one, even though it's tempting to reach for the same fix — filter control down to just the type (1) users, and compare like with like. That would just be Question 1 again, wearing a different justification, and it would break randomization exactly the same way.
Why "Ratio," Not "Selection," Is the Right Frame
The dilution in Question 2 doesn't need any selection at all. Every original user stays in the analysis, in both arms, at full sample size. What's needed instead is scaling the estimate by a measured proportion — the share of control users who were actually exposed to a GIF card:
, where is the exposure rate and is the true effect among exposed users. Solving for :
With the measured exposure rate at and the observed effect at : . No user was ever removed from the analysis to get that number — it comes from dividing, not filtering.
That's the real difference between the two questions. Question 1's instinct — narrow the sample down to the people who match a post-treatment condition — is the move that breaks randomization, no matter how reasonable it sounds. Question 2's dilution is resolved by leaving the sample alone entirely and rescaling the estimate by a ratio measured from it. This is exactly what an encouragement design is: you can't force exposure to a GIF card, only randomize who gets the opportunity to be exposed, and you recover the effect on those who would actually comply by dividing by that opportunity's realized rate — never by filtering down to them.
Where This Comes From, Formally
The "divide by the ratio" move isn't a bespoke trick invented for this one experiment — it's a specific, simplified case of instrumental variables (IV), the causal-inference technique for recovering a valid effect when you can't randomize the thing you actually care about, only something upstream of it. This companion article in the Causal Inference section reconstructs the same GIF experiment explicitly in IV terms — relevance, the exclusion restriction, the Wald estimator, and exactly which population that number actually describes.
Two Questions, Two Lessons
Question 1 looked like a reasonable filtering idea and turned out to quietly break randomization — the kind of mistake worth rejecting outright, independent of what the resulting numbers show. Question 2 looked like the harder problem and turned out to have a clean, principled answer that keeps every original user in the analysis and simply rescales by a measured exposure ratio. The next article checks that division against the published instrumental-variables literature, and the full worked example carries the numbers the rest of the way, including a third question about what the resulting significance level actually means.