Emergent Agency

Classification

Domain

7. AI system safety, failures, & limitations

Subdomain

7.6 Multi-agent risks

Causal entity

AI

Intent

Unintentional

Timing

Post-deployment

Subdomain definition: Risks from multi-agent interactions, due to incentives (which can lead to conflict or collusion) and/or the structure of multi-agent systems, which can create cascading failures, selection pressures, new security vulnerabilities, and a lack of shared information and trust.

How other frameworks describe this risk

• Groups of LLM-Agents May Show Emergent Functionality

  Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

• Collusion between LLM-Agents

  Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

• Multi-Agent Safety Is Not Assured by Single-Agent Safety

  Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

• Foundationality May Cause Correlated Failures

  Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

• Financial instability due to model homogeneity

  Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems (Gipiškis2024)

• Impact on Financial Stability

  Frontier AI Risk Management Framework (v1.0) (Tse2025)

• Multi-agent collaboration capability

  Frontier AI Risk Management Framework (v1.0) (Tse2025)

• Multi-agent collusion propensity:

  Frontier AI Risk Management Framework (v1.0) (Tse2025)

Other entries from Hammond2025

• Miscoordination Risk Category · 7.6
• Miscoordination Additional evidence
• Incompatible strategies Risk Sub-Category · 7.6
• Incompatible strategies Additional evidence
• Credit Assignment Risk Sub-Category · 7.6
• Limited Interactions Risk Sub-Category · 7.6
• Conflict Risk Category · 7.6
• Conflict Additional evidence
• Conflict Additional evidence
• Social Dilemmas Risk Sub-Category · 7.6
• Military Domains Risk Sub-Category · 7.6
• Coercion and Extortion Risk Sub-Category · 7.6