Risks from AIs developing goals and values that are different from humans

Classification

Domain

7. AI system safety, failures, & limitations

Subdomain

7.1 AI pursuing its own goals in conflict with human goals or values

Causal entity

AI

Intent

Intentional

Timing

Other

Subdomain definition: AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.

Real-world incidents in this subdomain

• 22 Dec 2016Reinforcement Learning Reward Functions in Video Games
• 1 Sep 2015Predictive Policing Program by Florida Sheriff’s Office Allegedly Violated Residents’ Rights and Targeted Children of Vulnerable Groups
• 20 Sep 2011Image Classification of Battle Tanks

How other frameworks describe this risk

• Natural Language Underspecifies Goals

  Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

• Loss of control

  International Scientific Report on the Safety of Advanced AI (Bengio2024)

• Loss of control

  International AI Safety Report 2025 (Bengio2025)

• Sudden loss of control

  Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks (Chin2025)

• Future AI systems might actively reduce human control

  Capabilities and Risks from Frontier AI (DSIT2023)

• Value specification

  AGI Safety Literature Review (Everitt2018 )

• Reliability

  AGI Safety Literature Review (Everitt2018 )

• Corrigibility

  AGI Safety Literature Review (Everitt2018 )

Other entries from Clarke2023

• Risks from accelerating scientific progress Risk Category
• Eased development of technologies that make a global catastrophe more likely Risk Sub-Category · 4.2
• Eased development of technologies that make a global catastrophe more likely Additional evidence
• Faster scientific progress makes it harder for governance to keep pace with development Risk Sub-Category · 6.5
• Faster scientific progress makes it harder for governance to keep pace with development Additional evidence
• Worsened conflict Risk Category · 6.4
• AI enables development of weapons of mass destruction Risk Sub-Category · 4.2
• AI enables automation of military decision-making Risk Sub-Category · 5.2
• AI-induced strategic instability Risk Sub-Category · 5.2
• Resource conflicts driven by AI development Risk Sub-Category · 6.4
• Increased power concentration and inequality Risk Category · 6.1
• Unequal distribution of harms and benefits Risk Sub-Category · 6.1