Map
Framework

The five-rung verification ladder

The question it answers

Before signing, sending or publishing a draft written by AI, which layers should a manager check, from the facts to the person who answers for it?

Origin
Ji et al., 2023; Sharma et al., 2024; Kasai et al., 2022; Parasuraman et al., 2000; Bainbridge, 1983
Diagram type
Process by phase
Level of analysis
Individual
Topics
AI for leadersEthics & corporate governance
Glass sphere with a glowing neural-network core, icon for AI for leaders

Diagram

The five-rung verification ladder

  1. Five rungs of verification
  2. FactsCan every figure, name and citation be traced to a real source?
  3. AssumptionsWhich assumptions does the answer rest on, and are they your own, repeated back to you?
  4. Source recencyIs the information still true today?
  5. Business riskIf it is wrong, who bears the loss and how much?
  6. Human approvalWho reads the sources and puts their name to it?

When to use it, and when not to

Use it when

Use it when an AI-written draft, table or summary is about to leave your hands: sent to a client, handed to your manager, published. Start at rung 1 and stop to fix whichever rung fails.

Avoid it when

The ladder is this site’s grouping of the sources above; none of them prescribes these five rungs or this order. For very low-risk work, such as rough notes for yourself, climbing all five rungs is wasted effort.

How to apply it, step by step

  1. Facts: open the source of every figure, name and citation. Hallucination is widely documented ((Ji et al., 2023)); in Mata v. Avianca the filing cited court cases that did not exist ((Mata v. Avianca, Inc., 2023)).
  2. Assumptions: write down the assumptions the answer rests on. AI assistants tend to change their answers to match the asker’s view, even a weakly stated one ((Sharma et al., 2024)), so assumptions you put into the question need a check of their own.
  3. Source recency: look at the date of each source. Models can answer with out-of-date information ((Kasai et al., 2022); (Lazaridou et al., 2021)).
  4. Business risk: ask who bears the loss if it is wrong, and how much. The cost of consequences is one criterion for choosing how much to hand to a machine ((Parasuraman et al., 2000)); the airline in Moffatt v. Air Canada was held responsible for what its chatbot said ((Civil Resolution Tribunal (British Columbia), 2024)).
  5. Human approval: a named person reads the sources and signs. The court in Mata stressed the lawyer’s “gatekeeping” role over what is filed ((Mata v. Avianca, Inc., 2023)). A person who only monitors a system that usually works loses vigilance and skill ((Bainbridge, 1983)), so this step has to be real reading, not a click on “approve”.

Pitfalls

Pitfall

Stopping at rung 1. A figure that matches its source can still be wrong if the source is out of date (rung 3) or if the question already carried a false assumption (rung 2).

Sources

References

Listed A–Z by author surname, Harvard style. A source with a DOI links to https://doi.org/…; one without links to its page or says plainly that it has no DOI.

  1. Bainbridge, L. (1983) ‘Ironies of automation’, Automatica, 19(6), pp. 775–779.doi:10.1016/0005-1098(83)90046-8Cited in 1 article
  2. Civil Resolution Tribunal (British Columbia) (2024) Moffatt v. Air Canada, 2024 BCCRT 149, file SC-2023-005609, decision issued 14 February 2024.No DOIdecisions.civilresolutionbc.caCited in 3 articles
  3. Ji, Z., Lee, N. and Frieske, R. (2023) ‘Survey of hallucination in natural language generation’, ACM Computing Surveys, 55(12), pp. 1–38.doi:10.1145/3571730Cited in 2 articles
  4. Kasai, J. et al. (2022) RealTime QA: what's the answer right now? (arXiv preprint arXiv:2207.13332).doi:10.48550/arXiv.2207.13332Cited in 2 articles
  5. Lazaridou, A. et al. (2021) Mind the gap: assessing temporal generalization in neural language models. (NeurIPS 2021; arXiv:2102.01951).doi:10.48550/arXiv.2102.01951Cited in 1 article
  6. Mata v. Avianca, Inc. (2023) 678 F. Supp. 3d 443 (S.D.N.Y. 2023), No. 22-cv-1461, Opinion and Order on Sanctions, 22 June 2023.No DOInhd.uscourts.govCited in 3 articles
  7. Parasuraman, R., Sheridan, T.B. and Wickens, C.D. (2000) ‘A model for types and levels of human interaction with automation’, IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, 30(3), pp. 286–297.doi:10.1109/3468.844354Cited in 2 articles
  8. Sharma, M., Tong, M. and Korbak, T. (2024) Towards understanding sycophancy in language models. (ICLR 2024; arXiv:2310.13548).doi:10.48550/arXiv.2310.13548Cited in 3 articles