Framework

Where numbers go wrong

The question it answers

When a manager is handed a report or chart to base a decision on, what should they check in the data, its selection, presentation and interpretation before trusting it?

Origin
De Veaux and Hand, 2005; Jansen et al., 2022
Diagram type
Process by phase
Level of analysis
Individual
Topics
Data & analytics
Magnifying lens over a line chart tile, icon for data and analytics

Diagram

Where numbers go wrong

  1. Data quality
  2. Missing dataYou see the survivors, not the cases that vanished.
  3. False proxyMeasuring what is easy (impressions) instead of what you need to know.
  4. Data selection
  5. Small or self-selected samplesThe sample does not represent the population.
  6. Cherry-pickingChoosing the period or calculation that looks best.
  7. Presentation
  8. Axes and scaleTruncated axes and mixed scales change perception.
  9. How the data is partitionedSame data, different grouping, different conclusion.
  10. Interpretation
  11. Signal or noiseShort-term swings hide the real trend.
  12. Correlation, causation, SimpsonCheck common causes, reverse direction, and pooling of groups.

Sources

References

Listed A–Z by author surname, Harvard style. A source with a DOI links to https://doi.org/…; one without links to its page or says plainly that it has no DOI.

  1. De Veaux, R.D. and Hand, D.J. (2005) ‘How to lie with bad data’, Statistical Science, 20(3), pp. 231–238.doi:10.1214/088342305000000269Source page
  2. Jansen, B.J. et al. (2022) ‘The illusion of data validity: why numbers about people are likely wrong’, Data and Information Management, 6(4), article 100020.doi:10.1016/j.dim.2022.100020Source page