Every team wants their experiment to succeed. That single fact is enough to make the way A/B tests get monitored in practice dangerous. Nobody calculates a sample size, patiently waits for the pre-computed date, and checks the result exactly once on the day they promised themselves they would. Instead, the dashboard gets refreshed every morning, every afternoon, sometimes every hour — and the experiment quietly stops the moment the desired result shows up, the moment the goal metric ticks up.
This is called the peeking problem, and it's one of the most common ways an experimentation program manufactures false wins without anyone realizing it.
The Behavior: Refresh Until It Looks Good
Picture a typical experiment at an ordinary startup. A PM ships a new checkout flow. The team is excited about the new policy. Someone opens the experiment dashboard the next morning to check the conversion rate. It's not significant yet — fine, still too early. They check again the next day. Still well short of what they're hoping for. By day four, the p-value drops below 0.05.
Champagne emojis flood the Slack channel. The experiment is declared a win and shipped to all traffic.
Nobody involved notices anything wrong with this. Everyone would say they were being careful — they didn't stop on day one, they patiently waited for "real" significance. But the behavior itself is exactly what statisticians call optional stopping: checking the result repeatedly and stopping the moment it crosses a threshold. Here's why that's a problem.
Why Refreshing the Dashboard Inflates False Positives
The significance level of commonly used in A/B testing means the following: if the new policy truly has no effect, a single test has only a 5% chance of falsely showing significance. That guarantee is valid only when you check the data exactly once.
If you check repeatedly and stop the moment things look favorable, you're no longer running a single test — you're running many, and each additional check adds another chance for random noise to cross the threshold. Suppose you check once a day for four days. Even if the new checkout flow has no effect whatsoever, the probability that at least one of those four checks turns out "significant" by pure chance is:
You believed you were running the experiment at a 5% false-positive rate. In reality, because of this repeated checking, you were running it at close to 18.5%. A pattern we see often — teams that check daily for two or three weeks — pushes this number well past 50%. Almost every experiment, looked at long enough, will eventually appear to "succeed" on noise alone.
The Guardrail Version Is Just as Dangerous
Most teams have, at least loosely, put guardrail metrics in place — revenue, latency, unsubscribe rate — to make sure a win on the primary metric isn't causing harm somewhere else. It looks like responsible behavior. In practice, it's just one more round of the same problem.
The team keeps refreshing the dashboard until the primary metric looks good and the guardrail doesn't look obviously bad. But checking repeatedly for "doesn't look obviously bad" commits the same mistake — repeating a noisy check until it happens to come out the way you want.
That's because guardrail metrics fluctuate day to day just as much as the goal metric does. If you keep checking until the guardrail happens to look fine, you haven't verified safety — you've simply found one noisy snapshot, out of many, that happened to let you stop.
Why This Keeps Happening
Peeking isn't a team-discipline problem that gets fixed by asking people to "just wait longer." It happens as the incentive to check early and frequently exists within the team. Engineers want to know if their work paid off, and PMs are evaluated on the impact of what they ship. For them, waiting three weeks when the dashboard updates in real time — when they could succeed quickly and move on to the next thing — feels wasteful. Refreshing costs little resistance, as nothing on the team stops anyone from doing it.
The most honest fix isn't "stop looking." It's designing a monitoring plan that controls the error rate even when people check the dashboard more often.
What Comes Next
The first common response to the peeking problem is to pick a fixed check-in day — say, day seven — and refuse to make any decision before then. It looks like a reasonable compromise. But teams still have an incentive to check early and roll back the moment something looks wrong. The next article, why a fixed 7-day rule doesn't actually solve peeking, shows exactly where that response falls apart.