A/B Test Results: Reading the Difference, the Interval, and the Mismatch Check

Last updated October 30, 2026·4 min read

A pricing page test, even split, two weeks:

The worked test

• Control: 24 conversions from 200 visitors, 12.0 percent
• Treatment: 36 conversions from 200 visitors, 18.0 percent
• Difference: plus 6.0 points, Newcombe 95 percent interval minus 1.0 to plus 13.0 points
• Chi-square 2.824, p = 0.093
• Bayesian probability that treatment beats control: 0.953

Four readings of the same data follow, three of them wrong.

Wrong reading one: the p value verdict

"p is greater than 0.05, so there is no effect." The interval refutes the sentence: the data are compatible with effects from one point negative to thirteen points positive, and most of that range is a meaningful win. A nonsignificant result on 400 visitors is a statement about the sample size before it is a statement about the treatment. The correct sentence is narrower: this test, at this size, does not separate the difference from zero at the conventional line.

Wrong reading two: the point estimate alone

"Treatment wins by six points, ship it." Six points is the center of a wide interval, quoted from a test where a one-point loss remains compatible with the data. Acting on the point estimate alone is sometimes defensible, when the change is cheap and reversible, but the decision record should quote the interval it acted under, so the next team knows the risk that was accepted.

Wrong reading three: the Bayesian number as a verdict

"There is a 95.3 percent chance treatment is better, so the case is closed." The probability is real and it is the most decision-shaped number on the page, but it is the probability that the treatment rate exceeds the control rate by any amount, including a tenth of a point. Expected loss completes the thought: how much is given up, on average, if the choice is wrong. A 95 percent chance of better, where better might be tiny and the interval admits harm, is a lean, not a landslide.

The reading that holds

The point estimate favors treatment by six points. The interval spans small harm to large gain. The Bayesian summary leans toward treatment at roughly twenty to one on direction, with magnitude unresolved. The decision then turns on costs, as it should: cheap and reversible argues for acting on the lean; expensive or hard to reverse argues for more data, and the sample size arithmetic prices that data exactly (a test four times this size would roughly halve the interval width).

Before any of that: the mismatch check

All four readings assume the randomization worked. The sample ratio mismatch check tests the assumption: observed variant counts against the intended allocation, by chi-square. A significant mismatch on an even split, 200 against 200 intended and 263 against 137 observed, say, signals a redirect fault, a logging fault, or a bot filter eating one arm, and in every case the comparisons downstream rest on a broken comparison. The worked test here splits 200 and 200, clean. Where the check fires, the experiment gets fixed and rerun, not interpreted harder. This is why the analyzer in Analysis Studio blocks on a mismatch until a person acknowledges it in writing: the most expensive A/B test error is a confident read of a broken test.

Three habits that separate careful teams

• Name the primary metric before launch, and hold it. Secondary metrics inform; they do not take turns being the headline until one of them moves.
• Set the runtime by a power calculation, aimed at the smallest lift worth acting on, and let the test run to it. Stopping rules invented mid-flight, especially stopping at the first significant peek, manufacture significance at rates no adjustment on the results page can undo.
• Report the interval and the split with every result, so the next reader can re-derive the quality of the evidence without asking for the raw data.

Analyze a test

Module 10 of Analysis Studio runs the mismatch check first, then the comparisons, with Newcombe intervals, CUPED where a pre-period covariate exists, and the Bayesian summary on the same screen:

Module 10: A/B Test AnalyzerOpen in Analysis Studio →

Sources and method notes

Interval method: Newcombe hybrid score interval for the difference of proportions. Bayesian summary: independent Beta posteriors under flat priors, matching the module implementation. Worked figures are shared with the calculator page and re-verified at build.