Which Statistical Test Should You Use?

Last updated October 30, 2026·4 min read

Test choice is not a matter of taste. Four questions, answered in order, land on the right procedure nearly every time:

The four questions that decide it

• What type is the outcome: continuous, binary, categorical, count, ordered, or time to event?
• What structure connects the cases: independent groups, paired or repeated measures, clusters, or a survey design?
• What is the question: a difference, an association, a prediction, an effect adjusted for other variables, or a ranking of importance?
• What do the data support: sample size, distribution shape, events per variable, and the assumptions each candidate carries?

The roster walk below follows those questions. Each entry names the module in Analysis Studio that runs it.

Describing before testing

Some questions end at description, and forcing a test onto them adds ceremony without information. Means, medians, spreads, and frequency tables with intervals are Module 1, Descriptive Statistics and Frequencies. Under a survey design those summaries are design-based, which is usually the entire requirement.

Two variables, one question

• Continuous outcome, one grouping variable with two levels: independent t test, Welch by default (Module 3). Paired measurements: paired t test (Module 3). Ordinal outcome, or a continuous outcome badly skewed at small n: Mann-Whitney or Wilcoxon signed-rank (Module 8).
• Continuous outcome, three or more groups: one-way ANOVA with post hoc comparisons (Module 3), or Kruskal-Wallis where the rank route is safer (Module 8).
• Two categorical variables: crosstab with chi-square, Fisher exact for small two by two tables, Rao-Scott under a survey design (Module 2).
• Two continuous or ordinal variables: correlation, Pearson, Spearman, or Kendall by measurement level, with partial correlation for a control (Module 4).
• Repeated conditions within the same cases: Friedman for the rank route (Module 8); richer structures belong to mixed models (Module 15).

Modeling an outcome on several predictors

• Continuous outcome: linear regression (Module 5), with robust or cluster-robust standard errors matched to the design.
• Binary outcome: logistic regression (Module 6), watching for separation, where the Firth fit takes over.
• Count outcome: Poisson, then negative binomial when dispersion demands it (Module 14).
• Ordered categories: proportional-odds logistic regression, with the Brant test checking the proportionality the name promises (Module 14).
• Unordered categories: multinomial logistic regression (Module 14).
• Clustered or repeated outcomes with the clustering itself of interest: mixed-effects models (Module 15).
• Time to an event, with censoring: Kaplan-Meier for curves and log-rank for comparisons, Cox models for adjustment (Module 16).
• A series over time: decomposition, trend tests, and ETS or ARIMA forecasting scored against a benchmark (Module 17).

Questions that are not tests

• Which predictors matter most: key driver analysis, Shapley importance shares with intervals (Module 9). The answer is a ranking with uncertainty, not a significance list.
• Did the treatment work in a randomized experiment: the A/B analyzer, one primary metric, mismatch check first (Module 10).
• What would the effect be if exposure had been assigned fairly: propensity score analysis, balance before results (Module 18).
• Is the sample representative, and what would make it so: weighting and raking (Module 11), including propensity adjustment where the sample was never drawn by probability at all.
• Did the population change across waves: trend analysis on repeated cross-sections with design-based tests (Module 12).
• Do these items form a scale: reliability and factor analysis (Module 13).
• How large should the study be: the sample size and power planner (Module 7), asked before data collection, when the answer can still change the design.

The design overrides the textbook default

One rule outranks every row above. Where data come from a complex survey, clustered, stratified, weighted, the procedure must be its design-based form, or the standard errors will understate the uncertainty by the design effect and every significance claim inherits the error. The Studio enforces this: with a design declared, a module either runs design-based methods or declines with the reason stated. The only exit is an explicit unweighted analysis, and the output then carries a banner saying so, so the choice is never silent.

A closing discipline

Choose the test from the design and the question, in that order, and write the choice down before the results exist. A test selected after browsing the outcomes is a story about the browsing, whatever its p value says.

Run the choice

The module picker in Analysis Studio organizes all nineteen procedures by the questions in this guide:

Analysis StudioOpen Analysis Studio →

Sources and method notes

Decision structure follows the Studio catalog and the module specifications, which encode the same decision table the app recommendation flow uses.