MIT AI Risk Repository · Risk Category · 56.16.00
Misalignment
Description
"A highly agentic, self-improving system, able to achieve goals in the physical world without human oversight, pursues the goal(s) it is set in a way that harms human interests. For this risk to be realised requires an AI system to be able to avoid correction or being switched off."
From Future Risks of Frontier AI (GOS2023), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
- Causal entity
- AI
- Intent
- Intentional
- Timing
- Post-deployment
Subdomain definition: AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.
Real-world incidents in this subdomain
- Reinforcement Learning Reward Functions in Video Games
- Predictive Policing Program by Florida Sheriff’s Office Allegedly Violated Residents’ Rights and Targeted Children of Vulnerable Groups
- Image Classification of Battle Tanks
How other frameworks describe this risk
- Natural Language Underspecifies Goals
- Loss of control
- Loss of control
- Sudden loss of control
- AI leads to humans losing control of the future
- Risks from delegating decision-making power to misaligned AIs
- Risks from AIs developing goals and values that are different from humans
- Future AI systems might actively reduce human control
Other entries from GOS2023
- Discrimination
- Inequality
- Environmental impacts
- Amplification of biases
- Harmful responses
- Lack of transparency and interpretability
- Intellectual property rights
- Providing new capabilities to a malicious actor
- Misapplication by a non-malicious actor
- Poor performance of a model used for its intended purpose, for example leading to biased decisions
- Unintended outcomes from interactions with other AI systems
- Impacts resulting from interactions with external societal, political, and economic systems