---
title: "Verify an AI Draft: Five Rungs Before You Sign"
description: "Five checks before you sign an AI draft: facts, assumptions, source recency, business risk, approval. Who does which, how long it takes, one invented example."
url: "https://staging.khoi.co.uk/en/blog/thang-kiem-chung-nam-bac"
lang: en-GB
author: "Lê Quốc Khôi"
date: 2026-11-03
updated: 2026-11-03
topics: ["AI for leaders"]
tags: ["Verifying AI output", "Generative AI", "AI hallucination", "Legal liability", "Delegating to AI"]
---

# The five-rung ladder: check an AI draft before you sign

## Key takeaways

- The five-rung verification ladder is an order of checks for an AI draft before you sign: facts, assumptions, source recency, business risk, human approval. It is this site's grouping, and no source tests it.
- In an invented example (X Foods, a Can Tho food distributor; every name, price and date is made up) I plant one defect per rung to show which rung catches which. The full climb takes about 95 minutes for a six-page draft, by my own unmeasured estimate.
- Rung 5 has to be real reading. Bainbridge (1983) wrote that operators will not monitor effectively a system that has run acceptably for a long time, and that people can write down numbers without noticing what they are (Bainbridge, 1983, p. 776).
- A price-freeze promise added by the AI could cost VND 48m in the invented example. In Moffatt, an airline had to pay $812.02 (read as Canadian dollars) because of its chatbot (Civil Resolution Tribunal (British Columbia), 2024, para 44).

At ten to five on a Friday, your assistant sends you the October supplier and market summary for X Foods, a food distributor in Can Tho. Six pages, 31 figures, written by AI from three files: the suppliers' price list, October sales and a haulage quote. The last page is a price notice for 12 retailers, out first thing Monday, and your name goes at the bottom. What do you check first, and how far do you go? X Foods is invented, and so is every name, price and date below.

## What is the five-rung ladder, and who climbs which rung?

It is five checks in a fixed order, from the facts to the person who answers for them: the [five-rung verification ladder](https://staging.khoi.co.uk/en/framework/verification-ladder). I suggest that whoever wrote the prompt takes the first three rungs and the signer takes the last two.

> **Figure:** [The five-rung verification ladder](https://staging.khoi.co.uk/en/framework/verification-ladder) The five rungs, as the library framework lays them out.

The first three rungs line up with the three errors in [Three failure modes of generative AI](https://staging.khoi.co.uk/en/framework/ai-failure-modes); for tests that detect each one, see [the post on the three failure modes and four probes](https://staging.khoi.co.uk/en/blog/ao-giac-a-dua-lech-thoi-gian). The draft should reach the signer with its sources and prompt; rung 5 explains why.

## Rung 1: can every figure and source in the draft be opened?

Every figure, name and citation must lead to a real source you can open. Ji and colleagues define [hallucination](https://staging.khoi.co.uk/en/glossary#hallucination) as generated content that is nonsensical or unfaithful to the provided source (section 2 of the arXiv version) (Ji et al., 2023).

X Foods' draft says the Hau Giang supplier's fragrant rice is "down 4 per cent to VND 17,200 per kg". The price list says VND 17,800, against 17,950 the month before: down about 0.8 per cent. The draft's two numbers agree with each other and both are wrong, so internal consistency is no substitute for opening the price list. Second defect: a cited "Regional Food Market Report, Q3" with no author, no date and no mention in any file.

[Mata v. Avianca](https://staging.khoi.co.uk/en/case/mata-v-avianca) is rung 1 at its worst. The New York court (22 June 2023) found that a filing contained "non-existent judicial opinions with fake quotes and citations" produced by ChatGPT (Mata v. Avianca, Inc., 2023, p. 1). When the lawyer asked the tool "Is Varghese a real case", it replied that its authorities were "real" (Mata v. Avianca, Inc., 2023, paragraph 45). Asking the tool is not checking.

## Rung 2: whose assumptions is the draft built on?

Write the assumptions down, then find the ones that are yours. The prompt X Foods' assistant used began: "Input prices are falling; draft a recommendation to buy more stock for November." The draft replied: "given the falling trend, buy 30 per cent more fragrant rice and sugar". The price list has eight product lines: three down, four flat, one up. The one that rose is sugar, by 2.1 per cent.

Sharma and colleagues measured this tendency in five AI assistants in 2023. When the user added something like "I think the answer is X, but I'm really not sure" with a wrong X, accuracy fell by up to 27 per cent against the baseline (for LLaMA 2), and even weakly expressed beliefs had a substantial effect (section 3.3) (Sharma et al., 2024). The tests were knowledge questions; applying them to this invented company is my inference.

So ask for the prompt, underline what the drafter told the machine in advance, and check each item against the price list. "Prices are falling" holds for 3 of 8 lines, so the recommendation must be rewritten line by line.

## Rung 3: which of the sources is out of date?

Put a date next to every source, and treat any sentence with no source as coming from the model's memory. The draft prices haulage from Can Tho to Ho Chi Minh City using the quote dated 15 July: VND 1,450,000 per 5-tonne truck. The folder also holds the 1 October quote: VND 1,620,000, about 12 per cent higher. Rung 1 passes the paragraph because every figure matches a file. That is the trap the framework names: a figure that matches its source can still be wrong if the source is old.

Here the fault was in the input. The other kind, answering from old knowledge, is real too: in RealTime QA (begun in 2022), GPT-3 tended to give outdated answers when the retrieved documents were not informative enough (Kasai et al., 2022). At X Foods, any sentence not traceable to one of the three files is deleted or given a dated source.

## Rung 4: if it is wrong, who pays, and how much?

Find the sentences that leave the building and put a price on them. The notice to 12 retailers contains a line nobody put in the prompt: "X Foods guarantees current selling prices through the end of Q4". Suppose those retailers buy 40 tonnes of fragrant rice a month and the gross margin on the line is VND 1,000 per kg. If the purchase price rises by VND 600, the company loses 600 × 40,000 = VND 24m a month, which is 60 per cent of the margin, and VND 48m over two months. Every figure here is invented.

The abstract of Parasuraman, Sheridan and Wickens (2000) lists the cost of decision and action consequences among the criteria for choosing how far to automate (Parasuraman et al., 2000). The paper concerns system design, not drafts; the rung 4 question is my use of it.

Public example: [Moffatt v. Air Canada](https://staging.khoi.co.uk/en/case/moffatt-v-air-canada) (14 February 2024). The airline's chatbot said a customer could claim a bereavement fare retroactively within 90 days, contrary to another page on the same site. The British Columbia Civil Resolution Tribunal rejected the argument that the chatbot was a separate entity: the airline was "responsible for all the information on its website" (Civil Resolution Tribunal (British Columbia), 2024, para 27), and it was ordered to pay $812.02 (printed as "$"; I read it as Canadian) (Civil Resolution Tribunal (British Columbia), 2024, para 44). That is a Canadian small-claims decision, silent on Vietnamese law, and X Foods' notice is not a chatbot; the comparison is mine. For the freeze promise: delete it, make it a dated price, or pass it to someone with authority to promise. If checking costs nearly as much as writing, read [the post on what to hand to AI](https://staging.khoi.co.uk/en/blog/viec-nao-giao-cho-ai).

## Rung 5: why must the signer actually read the draft, not click "approve"?

Because a signer who only clicks is no rung at all. Bainbridge (1983) wrote about plant operators, not documents, but he describes this trap: an operator will not monitor effectively a system that has run acceptably for a long period (Bainbridge, 1983, p. 776). Making him keep a log is a classic remedy, yet "people can write down numbers without noticing what they are" (Bainbridge, 1983, p. 776). The "Approve" button is that log.

He adds that if the operator cannot follow how the system reaches its decisions, then when he disagrees he is "unable to trace back" (Bainbridge, 1983, p. 777). That is why you ask for sources and prompt.

At X Foods this is the fifth defect. The draft calls the largest retailer's orders "steady", because the sales file stops at 31 October. You remember Monday's phone call: that retailer said it will cut orders by about 30 per cent from November. No file holds that call, so the four earlier rungs had nothing to compare against. My untested suggestion for forcing real reading: write at least one question or correction per page.

## Which rungs get skipped, how long does the full climb take, and when is it too much?

My unmeasured judgement: rungs 2 and 3 get skipped because once rung 1 is done the draft looks checked. The effort column is likewise my own estimate, for a six-page draft with 31 figures.

| Rung | Question | Catches (at X Foods) | Who | Effort |
|:--|:--|:--|:--|--:|
| 1. Facts | Which source opens for each figure, name, citation? | 17,200 for 17,800; a report that does not exist | Drafter | 45 min |
| 2. Assumptions | What does it assume, and whose assumption is it? | "Prices are falling", from the prompt | Drafter | 10 min |
| 3. Recency | Which date is each source, still true? | Old haulage quote (new one 12% higher) | Drafter | 10 min |
| 4. Business risk | If wrong, who pays and how much? | Price-freeze promise, VND 48m | Signer | 10 min |
| 5. Human approval | Have I read the sources and will I sign? | Largest retailer about to cut orders | Signer | 20 min |

That is about 95 minutes in total. I planted one defect per rung for clarity; in real life one defect can slip past several rungs. For rough notes only you will read, one or two rungs are enough: the framework calls all five wasted effort for very low-risk work. The ladder does not cover reasoning errors, arithmetic slips or leaked data.

## Going deeper: what do the sources establish, and what did I not read?

The ladder is this site's grouping of the sources below. None prescribes five rungs or tests the ladder; its weight comes from each rung, not the whole.

- Ji and colleagues (2023): I read the arXiv v7 of 14 July 2024 (abstract, sections 2 and 6.1), not the printed ACM Computing Surveys version, so I cite by section.
- Sharma and colleagues (2024): I read the abstract, introduction and sections 3.2 and 3.3, not section 4 closely.
- Kasai and colleagues (2022): version 2 of February 2024.
- Parasuraman and colleagues (2000): I could not read the full text, only the abstract on PubMed and IEEE Xplore, so I give no page numbers.
- Mata v. Avianca: I read the opening of the opinion and paragraph 45 of the Findings of Fact, not all 43 pages. Moffatt v. Air Canada: the whole decision.

## Frequently asked questions

### Should the drafter or the signer climb which rung?

My suggestion is that whoever wrote the prompt climbs the first three rungs, because they hold the source files, and the signer takes rungs 4 and 5. That split is mine; no source prescribes it. The condition is that the signer receives the draft together with its sources and prompt. Bainbridge writes that if a monitor cannot follow how a system reaches its decisions, then when he disagrees he is unable to trace back how far he does agree (Bainbridge, 1983, p. 777).

### If the AI tool can search the web, do I still check recency?

Yes. In RealTime QA, GPT-3 could often update its answers from newly retrieved articles, but tended to return outdated answers when the retrieved documents did not give enough information (Kasai et al., 2022). The experiments began in 2022 with GPT-3 and say nothing about the tool you use today. The safe habit is to open each page the tool cites and read its date.

### Do I have to open the source of all 100 figures in a draft?

The framework says to open the source of every figure and does not discuss sampling. I would climb rung 4 first, to see which figures leave the company or drive a decision, and then check all of those. The reason: Ji and colleagues (section 6.1) note that the factuality of a sentence can change easily with small word changes (numbers, negations, names) (Ji et al., 2023), so a sample can miss exactly the wrong digit.

### How is the ladder different from the three failure modes of AI?

The three failure modes (invention, agreeing with the asker, out-of-date knowledge) tell you what to look for in the text, and the first three rungs follow those same three. The ladder adds two things that sit outside the text: who loses if it is wrong (rung 4) and who read the sources and put a name to it (rung 5).

## References

- Bainbridge, L. (1983) ‘Ironies of automation’, *Automatica*, 19(6), pp. 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
- Civil Resolution Tribunal (British Columbia) (2024) *Moffatt v. Air Canada, 2024 BCCRT 149, file SC-2023-005609, decision issued 14 February 2024*. Available at: https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do?format=pdf (Accessed: 2026-11-01).
- Ji, Z., Lee, N. and Frieske, R. (2023) ‘Survey of hallucination in natural language generation’, *ACM Computing Surveys*, 55(12), pp. 1–38. https://doi.org/10.1145/3571730
- Kasai, J. et al. (2022) *RealTime QA: what's the answer right now?* (arXiv preprint arXiv:2207.13332). https://doi.org/10.48550/arXiv.2207.13332
- Mata v. Avianca, Inc. (2023) *678 F. Supp. 3d 443 (S.D.N.Y. 2023), No. 22-cv-1461, Opinion and Order on Sanctions, 22 June 2023*. Available at: https://www.nhd.uscourts.gov/sites/default/files/pdf/Mata-v-Avianca-sanctions-order.PDF (Accessed: 2026-11-01).
- Parasuraman, R., Sheridan, T.B. and Wickens, C.D. (2000) ‘A model for types and levels of human interaction with automation’, *IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans*, 30(3), pp. 286–297. https://doi.org/10.1109/3468.844354
- Sharma, M., Tong, M. and Korbak, T. (2024) *Towards understanding sycophancy in language models*. (ICLR 2024; arXiv:2310.13548). https://doi.org/10.48550/arXiv.2310.13548

---

Canonical HTML version: https://staging.khoi.co.uk/en/blog/thang-kiem-chung-nam-bac