MIT AI Risk Repository · Risk Sub-Category · 54.03.02
Emergent goals
Category: Harm caused by unaligned competent systems
Description
"As well as optimizing a subtly wrong goal, systems can develop harmful instrumental goals in the service of a given goal—without these emergent goals being specied in any way [434, 218, 339, 17]. For instance, a theorem in reinforcement learning suggests that optimal and near-optimal policies will seek power over their environment under fairly general conditions [560]. This power-seeking behavior is plausibly the worst of these emergent goals [92], and may be an attractor state for highly capable systems, since most goals can be furthered through gaining resources, self-preservation, preventi
From Ten Hard Problems in Artificial Intelligence We Must Get Right (Leech2024 ), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
- Causal entity
- AI
- Intent
- Intentional
- Timing
- Other
Subdomain definition: AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.
Real-world incidents in this subdomain
- Reinforcement Learning Reward Functions in Video Games
- Predictive Policing Program by Florida Sheriff’s Office Allegedly Violated Residents’ Rights and Targeted Children of Vulnerable Groups
- Image Classification of Battle Tanks
How other frameworks describe this risk
- Natural Language Underspecifies Goals
- Loss of control
- Loss of control
- Sudden loss of control
- AI leads to humans losing control of the future
- Risks from delegating decision-making power to misaligned AIs
- Risks from AIs developing goals and values that are different from humans
- Future AI systems might actively reduce human control
Other entries from Leech2024
- Negative impacts of AI use
- Under-recognized work
- Environmental cost
- Discrimination, toxicity, and bias
- Privacy
- Security
- Harm caused by incompetent systems
- Harm caused by unaligned competent systems
- Specification gaming
- Deceptive alignment
- Within-country issues: domestic inequality
- Demographic diversity of researchers