AI governance glossary
AI hallucination (confabulation)
What is an AI hallucination?
An AI hallucination is output from a generative AI system that is presented confidently but is false, fabricated or unsupported by its inputs, such as invented facts, citations or legal authorities. NIST's Generative AI Profile calls this risk confabulation.
Why it matters
- Courts have sanctioned lawyers for filing briefs that cited cases invented by AI.
- In regulated advice (legal, medical, financial) a confident wrong answer can be a compliance breach, not just an error.
- It is one of the twelve risks NIST's Generative AI Profile asks organisations to manage (NIST AI 600-1).
In detail
It follows from how language models generate text, predicting plausible continuations rather than retrieving verified facts, so it can be reduced but not eliminated. Mitigations include grounding answers in retrieved sources with citations, restricting use to tasks where errors are caught, human review of consequential outputs, and evaluating factual accuracy before release. OWASP lists the related risk as LLM09 Misinformation.
Source: NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. A plain-language explanation for orientation, not the legal text; follow the source for the binding wording.
Laws and policies on record that use it
- Expanded ASEAN Guide on AI Governance and Ethics – Generative AI (2025)ASEAN (Association of Southeast Asian Nations) · Guidance
- Singapore starter kit for testing LLM applicationsSingapore · Guidance
Related concepts
- AI explainabilityAI explainability is the ability of an AI system, or the people operating it, to give reasons for an output th...
- Human oversightDesign and operating measures that let people effectively oversee an AI system while it is in use: understand...
- Red teaming (AI)AI red teaming is structured adversarial testing in which testers try to make an AI system fail, by producing...
- OWASP Top 10 for LLM ApplicationsThe OWASP Top 10 for LLM Applications is the Open Worldwide Application Security Project's list of the ten mos...
All glossary terms · Think a definition is off? Tell us; corrections are logged on the corrections page.
Frequently asked questions
What is an AI hallucination?
An AI hallucination is output from a generative AI system that is presented confidently but is false, fabricated or unsupported by its inputs, such as invented facts, citations or legal authorities. NIST's Generative AI Profile calls this risk confabulation.
Where does the term AI hallucination come from?
This explanation follows NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. It is a plain-language paraphrase for orientation; the source has the binding wording.
Which laws and policies use the term AI hallucination?
Among the records on this site: Expanded ASEAN Guide on AI Governance and Ethics – Generative AI (2025) (ASEAN) and Singapore starter kit for testing LLM applications (Singapore).