MIT AI Risk Repository · Risk Sub-Category · 53.03.06
Failures in or misuse of intermediary (non-AGI) AI systems, resulting in catastrophe
Category: Direct catastrophe from AI
Description
"Deployment of “prepotent” AI systems that are non-general but capable of outperforming human collective efforts on various key dimensions;170 → Militarization of AI enabling mass attacks using swarms of lethal autonomous weapons systems;171 → Military use of AI leading to (intentional or unintentional) nuclear escalation, either because machine learning systems are directly integrated in nuclear command and control systems in ways that result in escalation172 or because conventional AI-enabled systems (e.g., autonomous ships) are deployed in ways that result in provocation and escalation;173
From Advancing AI Governance: A Literature Review of Problems, Options, and Proposals (Maas2023), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
- Domain
- 4. Malicious actors
- Causal entity
- Other
- Intent
- Other
- Timing
- Post-deployment
Subdomain definition: Using AI systems to develop cyber weapons (e.g., coding cheaper, more effective malware), develop new or enhance existing weapons (e.g., Lethal Autonomous Weapons or CBRNE), or use weapons to cause mass harm.
Real-world incidents in this subdomain
- Anthropic's Claude Was Reportedly Jailbroken To Allegedly Help Steal Sensitive Mexican Government Data
- OpenAI ChatGPT Models Reportedly Jailbroken to Provide Chemical, Biological, and Nuclear Weapons Instructions
- Anthropic Reportedly Identifies AI Misuse in Extortion Campaigns, North Korean IT Schemes, and Ransomware Sales
- LAMEHUG Malware Reportedly Integrates Large Language Model for Real-Time Command Generation in a Purported APT28-Linked Cyberattack
- Reported AI-Aided Development of Explosive Devices by Long Island Resident Michael Gann
- AI Chatbot Allegedly Used to Research Explosive Materials in Palm Springs Fertility Clinic Bombing
How other frameworks describe this risk
Other entries from Maas2023
- Alignment failures in existing ML systems
- Faulty reward functions in the wild
- Specification gaming
- Reward model overoptimization
- Instrumental convergence
- Goal misgeneralization
- Inner misalignment
- Language model misalignment
- Harms from increasingly agentic algorithmic systems
- Dangerous capabilities in AI systems
- Situational awareness
- Acquisition of a goal to harm society