What this analyzer does
The A/B Test Analyzer is Module 10 of Analysis Studio. It analyzes randomized experiments with two or more variants and exactly one primary metric, chosen before the results are read. Optional secondary metrics ride along and are labeled as secondary. Metric types are binary (converted or not), continuous (revenue, time, score), count, and ratio metrics built from a numerator and a denominator summed per randomization unit.
Three things set it apart from the two-proportion calculators that fill most search results:
- A sample ratio mismatch check runs first. If the observed split across variants departs from the intended split by more than chance allows, the analyzer stops and requires an acknowledgment before it shows comparisons, because a broken split usually means broken randomization or broken logging.
- CUPED variance reduction is available with a pre-period covariate, which tightens intervals for continuous metrics without touching the estimate.
- A Bayesian summary for binary metrics reports the probability that each variant beats control and the expected loss of choosing it, next to the frequentist interval, so the decision conversation has both readings on one screen.
Comparisons use Holm adjustment across variants. Binary differences carry Newcombe hybrid score intervals, the interval method with the best coverage record for this quantity. Continuous metrics use Welch t tests, with winsorizing or a bootstrap interval on request. The decision summary is a fixed template that states which variant the data favor and how strong that reading is. It never says ship, launch, or keep, because that call belongs to the team that owns the costs.
Promotion, stated with its date: through March 31, 2027, the A/B Test Analyzer runs free on screen, like every module. Downloads and Copy Results stay with the subscription throughout. After the promotion, the analyzer returns to the subscription set at $299 per year.
The method, in plain terms
For a binary metric, each variant converts at some rate. The comparison of interest is the difference in rates, and the honest summary of a difference is its interval. The analyzer computes each rate with a Wilson score interval, then combines the two into the Newcombe interval for the difference:
Lower bound = difference − √((treatment rate − its lower bound)² + (control upper bound − control rate)²), and symmetrically for the upper bound.
The sample ratio mismatch check is a chi-square test of the observed variant counts against the intended allocation proportions. The Bayesian summary places flat Beta priors on each rate and integrates the posterior, which for flat priors has a closed form the analyzer evaluates directly.
Worked example
A pricing page test runs for two weeks with an even split:
- Control: 24 conversions from 200 visitors, a rate of 12.0 percent
- Treatment: 36 conversions from 200 visitors, a rate of 18.0 percent
The analyzer reports:
- Difference: plus 6.0 percentage points
- Newcombe 95 percent interval for the difference: minus 1.0 to plus 13.0 points
- Chi-square test of the difference: 2.824, p = 0.093
- Bayesian probability that the treatment rate beats the control rate: 0.953
- Sample ratio mismatch check: 200 against 200 on an intended even split, no mismatch
Read together, the honest summary is: the point estimate favors treatment by six points, the interval includes small negative values, so the data are suggestive rather than settled, and the Bayesian reading puts the chance that treatment is better at about nineteen in twenty. A team that needs certainty keeps the test running. A team with a cheap rollback may act on the balance of evidence. The analyzer reports; it does not decide.
When the analyzer fits, and when it does not
It fits when:
- Assignment was genuinely random, at the level the metric is measured.
- One metric matters most and was named before the results came in.
- The variants ran concurrently over the same period, so weekday and seasonality effects hit all arms alike.
It does not fit when:
- The split was not random, for example a before-and-after change. That is an observational comparison, and the propensity tools in Module 18 are the honest route.
- The team wants to peek daily and stop the moment a p value dips under 0.05. Sequential testing needs sequential methods, which this module does not offer, and peeking inflates false positives no matter which calculator reads the data.
- Many metrics compete for the headline. Picking the metric that moved most, after the fact, manufactures significance. The analyzer holds the primary metric fixed for this reason.
Run it
Module 10: A/B Test AnalyzerOpen in Analysis Studio →Load the experiment data, or enter per-variant summaries where the module accepts them, name the primary metric, and read the mismatch check before the comparisons.
Questions
My test shows p = 0.09. Is the treatment better? The worked example on this page is exactly that case. The point estimate favors treatment, the interval runs from minus one to plus thirteen points, and the Bayesian probability of better is 0.953. That is evidence leaning toward treatment, short of the conventional line. Whether it is enough depends on the cost of being wrong, which the analyzer states but does not judge.
What is a sample ratio mismatch? A departure between the split the experiment intended and the split the data show, larger than chance explains. It usually signals a randomization fault, a logging fault, or a redirect that treats variants unequally. Results from a mismatched test rest on a broken foundation, so the analyzer blocks until the mismatch is acknowledged.
What does CUPED do? It uses a pre-period measurement of the same users, collected before the experiment, to remove the part of the outcome variance that the covariate explains. The estimate stays unbiased and the interval narrows, often substantially for revenue and engagement metrics.
Can it run more than two variants? Yes. Two or more variants are supported, each compared with control, with Holm adjustment across the family of comparisons.
Does the analyzer support sequential testing or bandits? No. Both are out of scope by design, and the page says so rather than implying coverage it does not have.