After understanding how serious the peeking problem really is, many teams adopt the following fix: pick a fixed number of days — seven, say — and refuse to make any judgment about the policy's effect before then. This is a reasonable instinct. But this approach doesn't solve every problem in A/B testing.
What the Fixed Window Actually Fixes
If a team commits to making the decision at a fixed point — one look at day 7, with no informal peeking and no attempt at an early call — the fixed-horizon test is statistically valid. The 5% false-positive rate holds exactly.
One observation ends the test.
The rule is appealing. It's simple, it's easy to explain to stakeholders, and it solves the peeking problem described in the previous article.
What the Fixed-Window Approach Doesn't Solve
Suppose the treatment is clearly, significantly causing harm on day 2. The fixed-window rule says to wait five more days anyway. This approach has no mechanism to stop early for safety — because the entire point of the rule, for better or worse, is to block any decision before day 7 arrives.
The Price of That Control
A strictly-enforced fixed window isn't "statistically" wrong. It properly controls the false-positive rate on the primary metric, and that does solve the peeking problem.
But that control comes at a steep price.
As just described, it isn't only that real harm can go unstopped — a clear win can't be stopped early either, no matter how strong the evidence is. A good policy that could have shipped in a day or two, or three, sits waiting until day 7. The new policies you can't decide on or test during that wait, and the money you don't make from this win in the meantime, are the hidden cost.
What You Actually Want
The right solution is neither "look whenever you feel like it" nor "look only once." It's looking a small, pre-planned number of times — say at day 1, day 3, and day 7 — with each observation assigned a statistically calibrated threshold. This way, checking early doesn't inflate the overall false-positive rate, and neither the guardrail metric nor the goal metric is left unwatched until the very end. That framework is Group Sequential Testing, and the next article introduces its definition.