MIT AI Risk Repository · Risk Sub-Category · 34.02.01
Situational Awareness
Category: Double edge components
Description
"AI systems may gain the ability to effectively acquire and use knowledge about itsstatus, its position in the broader environment, its avenues for influencing this environment, and the potentialreactions of the world (including humans) to its actions (Cotra, 2022). ...However, suchknowledge also paves the way for advanced methods of reward hacking, heightened deception/manipulationskills, and an increased propensity to chase instrumental subgoals (Ngo et al., 2024)."
From AI Alignment: A Comprehensive Survey (Ji2023), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
- Causal entity
- AI
- Intent
- Intentional
- Timing
- Other
Subdomain definition: AI systems that develop, access, or are provided with capabilities that increase their potential to cause mass harm through deception, weapons development and acquisition, persuasion and manipulation, political strategy, cyber-offense, AI development, situational awareness, and self-proliferation. These capabilities may cause mass harm due to malicious human actors, misaligned AI systems, or failure in the AI system.
How other frameworks describe this risk
- Agentic LLMs Pose Novel Risks
- Goal-Directedness Incentivizes Undesirable Behaviors
- Safety Risks from Affordances Provided to LLM-agents
- Capabilities that could be used to reduce human control - Autonomous replication and adaptation
- Capabilities that could be used to reduce human control - Manipulation
- Capabilities that could be used to reduce human control - Cyber offence
- Subagents
- Nascent capabilities (emergent capabilities)