Map
Framework

Four steps of hypothesis testing

The question it answers

After running a test on a sample of customers, what steps should a marketing or analytics team follow to decide whether the data give enough evidence to reject the default assumption?

Origin
Lane, n.d.; Illowsky & Dean, 2023; Greenland et al., 2016
Diagram type
Process by phase
Level of analysis
Individual
Topics
Data & analytics
Magnifying lens over a line chart tile, icon for data and analytics

Diagram

Four steps of hypothesis testing

  1. 1. State the null hypothesis
  2. H0 and the alternativeH0 is a statement of no difference, often the status quo. The alternative contradicts H0 and is usually what you are trying to show.
  3. 2. Set the alpha level
  4. Threshold α, fixed before computing pTypical values are 0.05 and 0.01. α is the probability of a Type I error; weigh which error has the heavier consequences.
  5. 3. Compute the p value
  6. Test statistic and pp is the probability of a sample result at least as different from H0 as the one observed, if H0 is true. The test statistic, such as t or chi-squared, measures the distance between the data and the model's prediction.
  7. 4. Compare p with α and decide
  8. Reject or do not reject H0If p is lower than α, reject H0, but this is not all-or-none: the lower p is, the more confidence you can have that H0 is false. Failing to reject H0 does not support H0; it only means the data are not strong enough.

When to use it, and when not to

Use it when

Use it when there is a specific default assumption, usually no difference, and a sample of data to try to reject it. Greenland et al. argue that in most scientific settings, estimating the size of the effect and its uncertainty matters more than sorting results into significant and non-significant, so report both.

Pitfalls

Pitfall

Treating p as the probability that H0 is true; treating p ≤ 0.05 as proof that H0 is false or that the finding matters in practice; treating p > 0.05 as showing H0 is true (Greenland et al., 2016, misinterpretations 1, 3, 4 and 7). Selecting analyses for presentation by the p values they produce can give small p values even when the hypothesis is correct.

Sources

References

Listed A–Z by author surname, Harvard style. A source with a DOI links to https://doi.org/…; one without links to its page or says plainly that it has no DOI.

  1. Greenland, S. et al. (2016) ‘Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations’, European Journal of Epidemiology, 31(4), pp. 337–350.doi:10.1007/s10654-016-0149-3Cited in 1 article
  2. Illowsky, B. and Dean, S. (2023) Introductory statistics 2e. Houston, Texas: OpenStax.No DOIopenstax.orgCited in 1 article
  3. Lane, D.M. (n.d.) Steps in hypothesis testing. Online Statistics Education: an interactive multimedia course of study (Rice University, University of Houston Clear Lake and Tufts University).No DOIonlinestatbook.comSource page