MIT AI Risk Repository · Risk Sub-Category · 47.02.12
Influence, overreliance and dependence (overreliance)
Category: Ethical and social risks
Description
"Beyond being simply influenced, humans may become overreliant on generative AI. Researchers with Microsoft’s AETHER (AI Ethics and Effects in Engineering and Research) define overreliance as users “accepting incorrect AI recommendations” or “making errors of commission” because they are “unable to determine whether or how much they should trust the AI.”
From Regulating under Uncertainty: Governance Options for Generative AI (G'sell2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
- Subdomain
- 5.1 Overreliance and unsafe use
- Causal entity
- Human
- Intent
- Unintentional
- Timing
- Post-deployment
Subdomain definition: Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Real-world incidents in this subdomain
- Lawsuit Alleged ChatGPT (GPT-4o) Encouraged Colorado Man's Suicide After Prolonged 'AI Companion' Chats
- Large-Scale Mental Health Crises Allegedly Associated with ChatGPT Interactions
- Google Gemini Reportedly Reinforced Delusions, Allegedly Contributing to Florida User's Near-Harm Episode and Suicide
- Family Reportedly Discovers ChatGPT Logs Detailing Suicidal Ideation Prior to Daughter's Death
- Purported AI Monitoring Software Reportedly Flags Unsent Joke Threat, Leading to Arizona Student Suspension
- ChatGPT Allegedly Reinforced Delusions Before Greenwich, Connecticut Murder-Suicide
How other frameworks describe this risk
Other entries from G'sell2024
- Technical and operational risks
- Technical vulnerabilities (Robustness - unexpected behaviour)
- Technical vulnerabilities (Robustness - unexpected behaviour)
- Technical vulnerabilities (Robustness - vulnerability to jailbreaking
- Technical vulnerabilities (Robustness - vulnerability to jailbreaking
- Technical vulnerabilities (The risk of misalignment)
- Technical vulnerabilities (The risk of misalignment)
- Factually incorrect content (inaccuracies and fabricated sources)
- Factually incorrect content (inaccuracies and fabricated sources)
- Opacity (the black box problem)
- Opacity (industry opacity)
- Opacity (industry opacity)