Map
Framework

Three failure modes of generative AI

The question it answers

Before AI output goes into a report or a decision, which error should a manager look for: invented content, agreement with their own view, or answers built on out-of-date knowledge?

Origin
Ji et al., 2023; Sharma et al., 2024; Kasai et al., 2022
Diagram type
Comparison table
Level of analysis
Individual
Topics
AI for leadersEthics & corporate governance
Glass sphere with a glowing neural-network core, icon for AI for leaders

Diagram

Three failure modes of generative AI

CriterionHallucinationSycophancyRecency lag
What it isinventing plausible-sounding content with no real sourcechanging the answer to fit the asker’s view instead of the factsanswering from old knowledge about a world that has moved on
Warning signcitations, court cases or figures that do not existapologises and switches its answer when you ask “Are you sure?”, even if the first answer was rightstates an old event, price or rule as if it still applied
How to detectopen the original source of every citation and figureask neutrally, push back once, compare two separate sessionscheck against dated sources; let the AI search, then check again
Public evidenceJi et al., 2023; Mata v. Avianca (2023)Sharma et al., 2024Kasai et al., 2022; Lazaridou et al., 2021

When to use it, and when not to

Use it when

Use it as a checklist before you send, publish or decide on the strength of AI output. Each failure mode has its own test in the “How to detect” row. Hallucination is surveyed in (Ji et al., 2023), sycophancy is measured in (Sharma et al., 2024) and out-of-date answers are tested in (Kasai et al., 2022).

Avoid it when

These three are not a complete list of AI errors, and the sources do not call them “the three core failures”: grouping them is this site’s choice. Reasoning errors, data bias and information leaks sit outside the frame.

Pitfalls

Pitfall

Asking the AI whether it is right. In Mata v. Avianca the lawyer asked ChatGPT “Is Varghese a real case” and was told the authorities were “real” and could be found on Westlaw; the court records this in its findings of fact ((Mata v. Avianca, Inc., 2023), paragraph 45). Check against an outside source, not against the tool’s own assurance.

Sources

References

Listed A–Z by author surname, Harvard style. A source with a DOI links to https://doi.org/…; one without links to its page or says plainly that it has no DOI.

  1. Ji, Z., Lee, N. and Frieske, R. (2023) ‘Survey of hallucination in natural language generation’, ACM Computing Surveys, 55(12), pp. 1–38.doi:10.1145/3571730Cited in 2 articles
  2. Kasai, J. et al. (2022) RealTime QA: what's the answer right now? (arXiv preprint arXiv:2207.13332).doi:10.48550/arXiv.2207.13332Cited in 2 articles
  3. Lazaridou, A. et al. (2021) Mind the gap: assessing temporal generalization in neural language models. (NeurIPS 2021; arXiv:2102.01951).doi:10.48550/arXiv.2102.01951Cited in 1 article
  4. Mata v. Avianca, Inc. (2023) 678 F. Supp. 3d 443 (S.D.N.Y. 2023), No. 22-cv-1461, Opinion and Order on Sanctions, 22 June 2023.No DOInhd.uscourts.govCited in 3 articles
  5. Sharma, M., Tong, M. and Korbak, T. (2024) Towards understanding sycophancy in language models. (ICLR 2024; arXiv:2310.13548).doi:10.48550/arXiv.2310.13548Cited in 3 articles