MIT AI Risk Repository · domain 5: Human-Computer Interaction
5.1 Overreliance and unsafe use
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
- 60
- 12
- 38
- 35
| Label | Value |
|---|---|
| Human | 27 |
| Other | 19 |
| AI | 14 |
| Label | Value |
|---|---|
| Unintentional | 35 |
| Other | 18 |
| Intentional | 7 |
| Label | Value |
|---|---|
| Post-deployment | 53 |
| Other | 7 |
| Label | Value |
|---|---|
| 2015 | 1 |
| 2018 | 1 |
| 2019 | 1 |
| 2021 | 4 |
| 2022 | 5 |
| 2023 | 5 |
| 2024 | 3 |
| 2025 | 18 |
| Label | Value |
|---|---|
| Risk Category | 11 |
| Risk Sub-Category | 49 |
Risk entries
Browse and export all- Creating avenues for exploiting user trust, nudging or manipulation
"In conversation, users may reveal private information that would otherwise be difficult to access, such as thoughts, opinions, or emotions. Capturing such information may enable downstream applicatio...
- Risk area 5: Human-Computer Interaction Harms
"This section focuses on risks specifically from LM applications that engage a user via dialogue, also referred to as conversational agents (CAs) [142]. The incorporation of LMs into existing dialogue...
- Anthropomorphising systems can lead to overreliance and unsafe use
Anticipated risk: "Natural language is a mode of communication particularly used by humans. Humans interacting with CAs may come to think of these agents as human-like and lead users to place undue co...
- Avenues for exploiting user trust and accessing more private information
Anticipated risk: "In conversation, users may reveal private information that would otherwise be difficult to access, such as opinions or emotions. Capturing such information may enable downstream app...
- Human-like interaction may amplify opportunities for user nudging, deception or manipulation
Anticipated risk: "In conversation, humans commonly display well-known cognitive biases that could be exploited. CAs may learn to trigger these effects, e.g. to deceive their counterpart in order to a...
- Overreliance
"Causing people to become emotionally or materially dependent on the model"
- Transformation of H2M interaction
"Human interaction with machines is a big challenge to society because it is already changing human behavior. Meanwhile, it has become normal to use AI on an everyday basis, for example, googling for...
- Decreasing human interaction as AI systems assume human tasks, disturbing well-being
- Operational misuses (Advice in heavily regulated industries)
- Economic harm (High-Risk Financial Activities)