Ask why an experiment shipped six months after it launched, and the honest answer is usually "let me go find out" — a Notion doc with the original plan, a Slack thread where the actual reasoning happened, and an experiment platform that only stored the numbers, none of them cross-linked, none of them the obvious single place to look. This is a reflection on that recurring friction, and an idea for closing it with an agent instead of a new process document nobody will maintain.
The Problem Isn't a Missing Tool, It's a Missing Document
None of the individual tools are broken. The planning doc genuinely captures the hypothesis at kickoff. The Slack thread genuinely captures the real debate — the objection someone raised, the tradeoff the team accepted, the reason a metric got reframed midstream. The experiment platform genuinely captures the results. The problem is that none of these were ever meant to be the canonical record, and by the time someone needs to understand what actually happened and why, the trail runs through all three, in whatever state they happen to be in months later — a doc that's been edited since, a thread that's scrolled off into history, a dashboard that only shows the current numbers with no memory of the original intent.
The Idea: Let an Agent Write the One-Pager
If the experiment analysis tool simply let you attach a link — a Notion doc, a Slack thread — to an experiment's planning field, an agent with read access to both (via MCP) could do something a human is unlikely to ever get around to doing consistently: read the stated goal and hypotheses, read the surrounding discussion, and consolidate all of it into a single, standardized write-up. Borrowing the shape of an Amazon-style one-pager — narrative, working backward from the goal, written for someone who wasn't in the room — gives every experiment the same predictable structure regardless of how scattered its source material was. Do this consistently across every experiment, and it stops being a one-off summary and starts being the seed of an actual wiki: a growing, structured record of what the team has tried and why, instead of a graveyard of Slack threads.
Capturing Intent Before Outcome Bias Sets In
The detail that makes this worth automating rather than doing by hand later: a daily Airflow DAG that checks for newly started experiments and, the moment one appears, pulls whatever Notion doc or Slack thread is attached as of that day and drafts the one-pager immediately. Written this early, the summary captures the hypothesis as it was actually reasoned about at kickoff — before anyone knows how the experiment turns out, and before hindsight quietly rewrites the story of what the team "really" expected. A write-up produced after the results are in is never fully free of outcome bias, even when written in good faith. A write-up drafted on day one has no result to bias it yet.
Closing the Loop When the Experiment Ends
The same mechanism can run a second pass after the experiment concludes. If there's a Slack thread discussing the readout — the debate over whether to ship, the guardrail that raised eyebrows, the reasoning that ultimately won — the agent can fold a summary of that business-context discussion into the same document. The result is one page that holds both halves of the story that are normally scattered and eventually lost: why the team started this, in their own words, at the time, and what they actually decided and why, once the data came in.
Why This Is Worth the Trouble
The honest reason this matters is a little uncomfortable: half of what makes this "Retrospective" category useful is exactly this kind of consolidation, done manually, well after the fact, from memory and whatever scraps of documentation survived. An automated one-pager doesn't replace the judgment in a retrospective — deciding what a result actually means still takes a person — but it would remove the part of the job that's pure archaeology: digging through old threads to reconstruct what was even being tested and why. That's institutional memory a team shouldn't have to depend on any one person's recollection to preserve, and it's exactly the kind of task an agent with access to the right documents is well suited to handle quietly, in the background, once a day, whether or not anyone remembers to ask for it.
This one's still an idea, not a shipped system — worth writing down precisely because the pain it addresses is real and recurring, and because the pieces needed to build it (a scheduled job, read access to a couple of internal tools, an agent that can summarize) are all things that already exist individually. What's missing is someone deciding it's worth wiring together.