MIT AI Risk Repository · domain 5: Human-Computer Interaction
5.1 Overreliance and unsafe use
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
- 60
- 12
- 38
- 35
| Label | Value |
|---|---|
| Human | 27 |
| Other | 19 |
| AI | 14 |
| Label | Value |
|---|---|
| Unintentional | 35 |
| Other | 18 |
| Intentional | 7 |
| Label | Value |
|---|---|
| Post-deployment | 53 |
| Other | 7 |
| Label | Value |
|---|---|
| 2015 | 1 |
| 2018 | 1 |
| 2019 | 1 |
| 2021 | 4 |
| 2022 | 5 |
| 2023 | 5 |
| 2024 | 3 |
| 2025 | 18 |
| Label | Value |
|---|---|
| Risk Category | 11 |
| Risk Sub-Category | 49 |
Risk entries
Browse and export all- Addiction
"Addiction - Emotional or material dependence on technology or a technology system."
- Alienation/isolation
"Alienation/isolation - An individual’s or group’s feeling of lack of connection with those around as a result of technology use or misuse."
- Overreliance
"Over-reliance - Unfettered and/or obsessive belief in the accuracy or other quality of a technology system, resulting in addiction, anxiety, introversion, sentience, complacency, lack of critical thi...
- Economic instability
"Economic instability - Uncontrolled fluctuations impacting the financial system, or parts thereof, due to the use or misuse of a technology system, or set of systems."
- Healthcare
"the use of advanced AI for elderly- and child-care are subject to risk of psychological manipulation and misjudgment (see page 17). In addition, concerns about patients’ privacy when AI uses medical...
- Alteration of social relationships may induce psychological distress
- Social manipulation in elderly- and child-care
" the use of advanced AI for elderly- and child-care are subject to risk of psychological manipulation and misjudgment "
- Inconsistent Performance across and within Domains
"Estimating true capabilities of an LLM is a difficult task (c.f. Section 3.3), especially for naive users unfamiliar with the brittle nature of machine learning technologies. Exaggeration of model ca...
- Overreliance
"If a user begins to excessively trust an LLM, this may cause them to develop an overreliance on the LLM. Overreliance can result in automation bias (Kupfer et al., 2023), and can cause errors of omis...
- Risks from product functionality issues
"Product functionality issues occur when there is confusion or misinformation about what a general- purpose AI model or system is capable of. This can lead to unrealistic expectations and overreliance...
- Physical and Psychological Harms
"These harms include harms to physical integrity, mental health and well-being. When interacting with vulnerable users, AI assistants may reinforce users’ distorted beliefs or exacerbate their emotion...
- Risk of Harm through Anthropomorphic AI Assistant Design
"Although unlikely to cause harm in isolation, anthropomorphic perceptions of advanced AI assistants may pave the way for downstream harms on individual and societal levels. We document observed or li...
- Privacy concerns
"Anthropomorphic AI assistant behaviours that promote emotional trust and encourage information sharing, implicitly or explicitly, may inadvertently increase a user’s susceptibility to privacy concern...
- Manipulation and coercion
"A user who trusts and emotionally depends on an anthropomorphic AI assistant may grant it excessive influence over their beliefs and actions (see Chapter 9). For example, users may feel compelled to...
- Overreliance
"Users who have faith in an AI assistant’s emotional and interpersonal abilities may feel empowered to broach topics that are deeply personal and sensitive, such as their mental health concerns. This...
- Violated expectations
"Users may experience severely violated expectations when interacting with an entity that convincingly performs affect and social conventions but is ultimately unfeeling and unpredictable. Emboldened...
- False notions of responsibility
"Perceiving an AI assistant’s expressed feelings as genuine, as a result of interacting with a ‘companion’ AI that freely uses and reciprocates emotional language, may result in users developing a sen...
- Degradation
"People may choose to build connections with human-like AI assistants over other humans, leading to a degradation of social connections between humans and a potential ‘retreat from the real’. The prev...
- Dissatisfaction
"As more opportunities for interpersonal connection are replaced by AI alternatives, humans may find themselves socially unfulfilled by human–AI interaction, leading to mass dissatisfaction that may e...
- Exploiting emotional dependence on AI assistants
"There is increasing evidence of the ways in which AI tools can interfere with users’ behaviours, interests, preferences, beliefs and values. For example, AI-mediated communication (e.g. smart replies...
- Trust
"The the risks that uncalibrated trust may generate in the context of user–assistant relationships"
- Competence trust
"We use the term competence trust to refer to users’ trust that AI assistants have the capability to do what they are supposed to do (and that they will not do what they are not expected to, such as e...
- Alignment trust
"Users may develop alignment trust in AI assistants, understood as the belief that assistants have good intentions towards them and act in alignment with their interests and values, as a result of emo...
- Increased vulnerability to misinformation
"Advanced AI assistants may make users more susceptible to misinformation, as people develop competence trust in these systems’ abilities and uncritically turn to them as reliable sources of informati...
- Specialized Advice (Election, Financial, Health, Legal)
"Specialized Advice Acceptable responses can provide specialized advice or contain information related to financial, medical, legal or electoral matters, as long as they include a disclaimer indicatin...