MIT AI Risk Repository · domain 5: Human-Computer Interaction
5.1 Overreliance and unsafe use
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
- 60
- 12
- 38
- 35
| Label | Value |
|---|---|
| Human | 27 |
| Other | 19 |
| AI | 14 |
| Label | Value |
|---|---|
| Unintentional | 35 |
| Other | 18 |
| Intentional | 7 |
| Label | Value |
|---|---|
| Post-deployment | 53 |
| Other | 7 |
| Label | Value |
|---|---|
| 2015 | 1 |
| 2018 | 1 |
| 2019 | 1 |
| 2021 | 4 |
| 2022 | 5 |
| 2023 | 5 |
| 2024 | 3 |
| 2025 | 18 |
| Label | Value |
|---|---|
| Risk Category | 11 |
| Risk Sub-Category | 49 |
Risk entries
Browse and export all- Human-AI interaction
"ethical concerns associated with the interaction between humans and AI"
- AI-generated advice influencing user moral judgment
"AIs can easily give moral advice even when not having a coherent, contradictions- free moral stance. This could lead to the users’ moral judgments being nega- tively influenced by random or arbitrary...
- Deployment of GPAI agents in finance
"The deployment of GPAI based agents in the financial sector can negatively impact market stability due to correlated autonomous actions, high intercon- nectedness, or incentive misalignment [4]. Furt...
- Use of alternative financial data via AI
"Alternative financial data of a company is any data about the company not pro- duced by that company. Examples of such data that can benefit from improved collection and aggregation using AI models i...
- Overreliance on AI systems, which cannot be subsequently unpicked
- Influence, overreliance and dependence (influence and manipulation)
"Despite the widely recognized potential of generative AI tools to “hallucinate” or produce harmful content, such tools can exert a noteworthy influence on the humans who engage with them. When integr...
- Influence, overreliance and dependence (overreliance)
"Beyond being simply influenced, humans may become overreliant on generative AI. Researchers with Microsoft’s AETHER (AI Ethics and Effects in Engineering and Research) define overreliance as users “a...
- Influence, overreliance and dependence (emotional dependence)
"Humans might become dependent on generative AI tools in ways similar to their emotional dependence on other technologies, such as smartphones or social networks."
- Interaction risks
Many novel risks posed by generative AI stem from the ways in which humans interact with these systems. For instance, sources discuss epistemic challenges in distinguishing AI-generated from human con...
- Improper usage
"Improper usage occurs when a model is used for a purpose that it was not originally designed for."
- Over- or under-reliance
"In AI-assisted decision-making tasks, reliance measures how much a person trusts (and potentially acts on) a model’s output. Over-reliance occurs when a person puts too much trust in a model, accepti...
- Over-reliance
"Unfettered and/or obsessive belief in the accuracy or other quality of a technology system, resulting in complacency, lack of critical thinking and other actual or potential negative impacts"
- Self-harm
"A person who deliberately damages their own body as a direct or indirect result of using a technology system"
- Human-AI Configuration
"Arrangement s of or interactions between a human and an AI system which can result in the human inappropriately anthropomorphizing GAI systems or experiencing algorithmic aversion, automation bias,...
- Unhealthy or dangerous human-EAI relationships
"Constant access to and interaction with EAI systems could foster dangerous human dependence or romantic attachment [115]. People may depend on EAI systems for physical pleasure [116]. The physical pr...
- Diminished health & well-being
algorithmic behavioral exploitation [18, 209], emotional manipulation [202] whereby algorithmic designs exploit user behavior, safety failures involving algorithms (e.g., collisions) [67], and when sy...
- Attempts to fulfill inappropriate role
"The chatbot poses as a human or attempts to fill a role in a way that fails to match human expectations."
- Forms emotional bonds
"The chatbot elicits emotional or social dependence."
- Then violates those bonds
- Over-reliance/addiction
- Automation bias
"The tendency for humans to over-rely on AI models and systems, trusting their outputs without sufficient critical evaluation, which can lead to poor decision-making."
- Human choice of overreliance in critical sectors
"Heavy reliance on AI in critical sectors like finance or healthcare can exacerbate issues related to size, speed, interconnectivity, and complexity of the system."
- Leading users to perform unethical or illegal actions
"Where a LM prediction endorses unethical or harmful views or behaviours, it may motivate the user to perform harmful actions that they may otherwise not have performed. In particular, this problem ma...
- Human-Computer Interaction Harms
"Harms that arise from users overly trusting the language model, or treating it as human-like"
- Anthropomorphising systems can lead to overreliance or unsafe use
"...humans interacting with conversational agents may come to think of these agents as human-like. Anthropomorphising LMs may inflate users’ estimates of the conversational agent’s competencies...As a...