Loss of control of autonomous systems and unforeseen behaviour due to lack of transparency and self-programming/ reprogramming

Classification

Domain

7. AI system safety, failures, & limitations

Subdomain

7.1 AI pursuing its own goals in conflict with human goals or values

Causal entity

Other

Intent

Other

Timing

Other

Subdomain definition: AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.

Real-world incidents in this subdomain

• 22 Dec 2016Reinforcement Learning Reward Functions in Video Games
• 1 Sep 2015Predictive Policing Program by Florida Sheriff’s Office Allegedly Violated Residents’ Rights and Targeted Children of Vulnerable Groups
• 20 Sep 2011Image Classification of Battle Tanks

How other frameworks describe this risk

• Natural Language Underspecifies Goals

  Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

• Loss of control

  International Scientific Report on the Safety of Advanced AI (Bengio2024)

• Loss of control

  International AI Safety Report 2025 (Bengio2025)

• Sudden loss of control

  Dimensional Characterization and Pathway Modeling for Catastrophic AI Risks (Chin2025)

• AI leads to humans losing control of the future

  A Survey of the Potential Long-term Impacts of AI: How AI Could Lead to Long-term Changes in Science, Cooperation, Power, Epistemics and Values (Clarke2023)

• Risks from delegating decision-making power to misaligned AIs

  A Survey of the Potential Long-term Impacts of AI: How AI Could Lead to Long-term Changes in Science, Cooperation, Power, Epistemics and Values (Clarke2023)

• Risks from AIs developing goals and values that are different from humans

  A Survey of the Potential Long-term Impacts of AI: How AI Could Lead to Long-term Changes in Science, Cooperation, Power, Epistemics and Values (Clarke2023)

• Future AI systems might actively reduce human control

  Capabilities and Risks from Frontier AI (DSIT2023)

Other entries from Wirtz2022

• Technological, Data and Analytical AI Risks Risk Category
• Programming error Risk Sub-Category · 7.0
• Lack of data, poor data quality, and biases in training data Risk Sub-Category · 1.1
• Vulnerability of AI systems to attacks and misuse Risk Sub-Category · 2.2
• Lack of AI experts with comprehensive AI knowledge Risk Sub-Category · 6.5
• Immaturity of AI technology can cause incorrect decisions Risk Sub-Category · 7.3
• High investment costs of AI hinder integration Risk Sub-Category · 6.1
• Informational and Communicational AI Risks Risk Category · 4.1
• Manipulation and control of information provision (e.g., personalised adds, filtered news) Risk Sub-Category · 4.1
• Disinformation and computational propaganda Risk Sub-Category · 4.1
• Censorship of opinions expressed in the Internet restricts freedom of expression Risk Sub-Category · 5.2
• Endangerment of data protection through AI cyberattacks Risk Sub-Category · 4.2