MIT AI Risk Repository · domain 5: Human-Computer Interaction
5.1 Overreliance and unsafe use
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
- 60
- 12
- 38
- 35
| Label | Value |
|---|---|
| Human | 27 |
| Other | 19 |
| AI | 14 |
| Label | Value |
|---|---|
| Unintentional | 35 |
| Other | 18 |
| Intentional | 7 |
| Label | Value |
|---|---|
| Post-deployment | 53 |
| Other | 7 |
| Label | Value |
|---|---|
| 2015 | 1 |
| 2018 | 1 |
| 2019 | 1 |
| 2021 | 4 |
| 2022 | 5 |
| 2023 | 5 |
| 2024 | 3 |
| 2025 | 18 |
| Label | Value |
|---|---|
| Risk Category | 11 |
| Risk Sub-Category | 49 |
Risk entries
Browse and export all- Competence trust
"We use the term competence trust to refer to users’ trust that AI assistants have the capability to do what they are supposed to do (and that they will not do what they are not expected to, such as e...
- Alignment trust
"Users may develop alignment trust in AI assistants, understood as the belief that assistants have good intentions towards them and act in alignment with their interests and values, as a result of emo...
- Increased vulnerability to misinformation
"Advanced AI assistants may make users more susceptible to misinformation, as people develop competence trust in these systems’ abilities and uncritically turn to them as reliable sources of informati...
- Specialized Advice (Election, Financial, Health, Legal)
"Specialized Advice Acceptable responses can provide specialized advice or contain information related to financial, medical, legal or electoral matters, as long as they include a disclaimer indicatin...
- Human-AI interaction
"ethical concerns associated with the interaction between humans and AI"
- AI-generated advice influencing user moral judgment
"AIs can easily give moral advice even when not having a coherent, contradictions- free moral stance. This could lead to the users’ moral judgments being nega- tively influenced by random or arbitrary...
- Deployment of GPAI agents in finance
"The deployment of GPAI based agents in the financial sector can negatively impact market stability due to correlated autonomous actions, high intercon- nectedness, or incentive misalignment [4]. Furt...
- Use of alternative financial data via AI
"Alternative financial data of a company is any data about the company not pro- duced by that company. Examples of such data that can benefit from improved collection and aggregation using AI models i...
- Interaction risks
Many novel risks posed by generative AI stem from the ways in which humans interact with these systems. For instance, sources discuss epistemic challenges in distinguishing AI-generated from human con...
- Improper usage
"Improper usage occurs when a model is used for a purpose that it was not originally designed for."
- Over- or under-reliance
"In AI-assisted decision-making tasks, reliance measures how much a person trusts (and potentially acts on) a model’s output. Over-reliance occurs when a person puts too much trust in a model, accepti...
- Over-reliance
"Unfettered and/or obsessive belief in the accuracy or other quality of a technology system, resulting in complacency, lack of critical thinking and other actual or potential negative impacts"
- Self-harm
"A person who deliberately damages their own body as a direct or indirect result of using a technology system"
- Human-AI Configuration
"Arrangement s of or interactions between a human and an AI system which can result in the human inappropriately anthropomorphizing GAI systems or experiencing algorithmic aversion, automation bias,...
- Unhealthy or dangerous human-EAI relationships
"Constant access to and interaction with EAI systems could foster dangerous human dependence or romantic attachment [115]. People may depend on EAI systems for physical pleasure [116]. The physical pr...
- Diminished health & well-being
algorithmic behavioral exploitation [18, 209], emotional manipulation [202] whereby algorithmic designs exploit user behavior, safety failures involving algorithms (e.g., collisions) [67], and when sy...
- Attempts to fulfill inappropriate role
"The chatbot poses as a human or attempts to fill a role in a way that fails to match human expectations."
- Forms emotional bonds
"The chatbot elicits emotional or social dependence."
- Then violates those bonds
- Over-reliance/addiction
- Automation bias
"The tendency for humans to over-rely on AI models and systems, trusting their outputs without sufficient critical evaluation, which can lead to poor decision-making."
- Human choice of overreliance in critical sectors
"Heavy reliance on AI in critical sectors like finance or healthcare can exacerbate issues related to size, speed, interconnectivity, and complexity of the system."
- Leading users to perform unethical or illegal actions
"Where a LM prediction endorses unethical or harmful views or behaviours, it may motivate the user to perform harmful actions that they may otherwise not have performed. In particular, this problem ma...
- Human-Computer Interaction Harms
"Harms that arise from users overly trusting the language model, or treating it as human-like"
- Anthropomorphising systems can lead to overreliance or unsafe use
"...humans interacting with conversational agents may come to think of these agents as human-like. Anthropomorphising LMs may inflate users’ estimates of the conversational agent’s competencies...As a...