MIT AI Risk Repository · Risk Sub-Category · 24.05.07

Disorientation

Category: Anthropomorphism

Open the MIT Risk Navigator All entries from Gabriel2024

Description

"Given the capacity to fine-tune on individual preferences and to learn from users, personal AI assistants could fully inhabit the users’ opinion space and only say what is pleasing to the user; an ill that some researchers call ‘sycophancy’ (Park et al., 2023a) or the ‘yea-sayer effect’ (Dinan et al., 2021). A related phenomenon has been observed in automated recommender systems, where consistently presenting users with content that affirms their existing views is thought to encourage the formation and consolidation of narrow beliefs (Du, 2023; Grandinetti and Bruinsma, 2023; see also Chapter

From The Ethics of Advanced AI Assistants (Gabriel2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 5. Human-Computer Interaction
Subdomain: 5.2 Loss of human agency and autonomy
Causal entity: Human
Intent: Unintentional
Timing: Post-deployment

Subdomain definition: Humans delegating key decisions to AI systems, or AI systems making decisions that diminish human control and autonomy, potentially leading to humans feeling disempowered, losing the ability to shape a fulfilling life trajectory or becoming cognitively enfeebled.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Autonomy
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)
Autonomy/agency loss
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)
Personality rights loss
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)
Gradual loss of control
Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks (Chin2025)
AI enables automation of military decision-making
A Survey of the Potential Long-term Impacts of AI: How AI Could Lead to Long-term Changes in Science, Cooperation, Power, Epistemics and Values (Clarke2023)
AI-induced strategic instability
A Survey of the Potential Long-term Impacts of AI: How AI Could Lead to Long-term Changes in Science, Cooperation, Power, Epistemics and Values (Clarke2023)
-
Capabilities and Risks from Frontier AI (DSIT2023)
Humans might increasingly hand over control to misaligned AI systems
Capabilities and Risks from Frontier AI (DSIT2023)

Other entries from Gabriel2024

Capability failures Risk Category · 7.3
Lack of capability for task Risk Sub-Category · 7.3
Difficult to develop metrics for evaluating benefits or harms caused by AI assistants Risk Sub-Category · 6.5
Safe exploration problem with widely deployed AI assistants Risk Sub-Category · 7.3
Goal-related failures Risk Category · 7.1
Misaligned consequentialist reasoning Risk Sub-Category · 7.3
Specification gaming Risk Sub-Category · 7.1
Goal misgeneralisation Risk Sub-Category · 7.1
Deceptive alignment Risk Sub-Category · 7.1
Malicious Uses Risk Category · 4.0
Offensive Cyber Operations (General) Risk Sub-Category · 4.2
AI-Powered Spear-Phishing at Scale Risk Sub-Category · 4.1