{"attribution":{"source":"MIT AI Risk Repository, Domain Taxonomy of AI Risks v1 (MIT AI Risk Initiative)","license":"CC BY 4.0","license_url":"https://creativecommons.org/licenses/by/4.0/","citation":"Slattery, P., Saeri, A. K., Grundy, E. A. C., Graham, J., Noetel, M., Uuk, R., Dao, J., Pour, S., Casper, S., & Thompson, N. (2025). The AI Risk Repository: A comprehensive meta-review, database, and taxonomy of risks from artificial intelligence. arXiv:2408.12622."},"exported_at":"2026-09-11"}
{"rows":[{"ev_id":"63.01.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Miscoordination ","risk_subcategory":null,"description":"\"Miscoordination arises when agents, despite a mutual and clear objective, cannot align their behaviours to achieve this objective. Unlike the case of differing objectives, in common-interest settings there is a more easily well-defined notion of ‘optimal’ behaviour and we describe agents as miscoordinating to the extent that they fall short of this optimum. Note that for common-interest settings it is not sufficient for agents’ objectives to be the same in the sense of being symmetric (e.g., when two agents both want the same prize, but only one can win). Rather, agents must have identical pr","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.01.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Miscoordination ","risk_subcategory":"Incompatible strategies ","description":"\"Incompatible Strategies. Even if all agents can perform well in isolation, miscoordination can still occur due to the agents choosing incompatible strategies (Cooper et al., 1990). Competitive (i.e., two- player zero-sum) settings allow designers to produce agents that are maximally capable without taking other players into account. Crucially, this is possible because playing a strategy at equilibrium in the zero-sum setting guarantees a certain payoff, even if other players deviate from the equilibrium (Nash, 1951). On the other hand, common-interest (and mixed-motive) settings often allow a","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.01.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Miscoordination ","risk_subcategory":"Credit Assignment ","description":"\"Credit Assignment. While agents can often learn to jointly solve tasks and thus avoid coordination failures, learning is made more challenging in the multi-agent setting due to the problem of credit assignment (Du et al., 2023; Li et al., 2025, see also Section 3.1 on information asymmetries and Section 3.4, which discusses distributional shift). That is, in the presence of other learning agents, it can be unclear which agents’ actions caused a positive or negative outcome to obtain, especially if the environment is complex. Moreover, in multi-principal settings, agents may not have been trai","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.01.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Miscoordination ","risk_subcategory":"Limited Interactions","description":"\"Limited Interactions. Sometimes learning from historical interactions with the relevant agents may not be possible, or may be possible using only limited interactions. In such cases, some other form of information exchange is required for agents to be able to reliably coordinate their actions, such as via communication (Crawford & Sobel, 1982; Farrell & Rabin, 1996a) or a correlation device (Aumann, 1974, 1987). While advances in language modelling mean that there are likely to be fewer settings in which the inability of advanced AI systems to communicate leads to miscoordination, situations ","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.02.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Conflict ","risk_subcategory":null,"description":"\"In the vast majority of real-world strategic interactions, agents’ objectives are neither identical nor completely opposed. Indeed, if AI agents are sufficiently aligned to their users or deployers, we should expect some degree of both cooperation and competition, mirroring human society. These mixed-motive settings include the possibility of mutual gains, but also the risk of conflict due to selfish incentives. In what follows, we examine the extent to which advanced AI might precipitate or exacerbate such risks.\"","entity":"AI","intent":"Other","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.02.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Conflict ","risk_subcategory":"Social Dilemmas ","description":"\"Social Dilemmas. As noted in our definition, conflict can arise in any situation in which selfish incentives diverge from the collective good, known as a social dilemma (Dawes & Messick, 2000; Hardin, 1968; Kollock, 1998; Ostrom, 1990). While this is by no means a modern problem, advances in AI could further enable actors to pursue their selfish incentives by overcoming the technical, legal, or social barriers that standardly help to prevent this. To take a plausible, near-term (if very low-stakes) example, an automated AI assistant could easily reserve a table at every restaurant in town in ","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.02.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Conflict ","risk_subcategory":"Military Domains ","description":"\"Perhaps the most obvious and worrying instances of AI conflict are those in which human conflict is already a major concern, such as military domains (although other, less salient forms of conflict such as international trade wars are also cause for concern). For example, beyond applications of more narrow AI tools in lethal autonomous weapons systems (Horowitz, 2021), future AI systems might serve as advisors or negotiators in high-stakes military decisions (Black et al., 2024; Manson, 2024). Indeed, companies such as Palantir have already developed LLM-powered tools for military planning (P","entity":"AI","intent":"Other","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.02.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Conflict ","risk_subcategory":"Coercion and Extortion ","description":"\"Advanced AI systems might also lead to various forms of coercion and extortion in less extreme settings (Ellsberg, 1968; Harrenstein et al., 2007). These threats might target humans directly (such as the revelation of private information extracted by advanced AI surveillance tools), or other AI systems that are deployed on behalf of humans (such as by hacking a system to limit its resources or operational capacity; see also Section 3.7). Increasing AI cyber-offensive capabilities – including those that target other AI systems via adversarial attacks and jailbreaking (Gleave et al., 2020; Yami","entity":"AI","intent":"Other","timing":"Other","domain":7,"subdomain":"7.6"},{"ev_id":"63.03.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Collusion ","risk_subcategory":null,"description":"\"Collusion has long been a topic of intense study in economics, law, and politics, among other disciplines. While there is no universal definition of collusion, it generally refers to secretive cooperation between two or more parties at the expense of one or more other parties. Most classic examples of collusion – such as firms working together to set supra-competitive prices at the expense of consumers – also tend to be not only secretive but in violation of some law, rule, or ethical standard. Distinctions are also commonly made between explicit and tacit collusion (Rees, 1993), depending on","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.03.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Collusion ","risk_subcategory":"Markets ","description":"\"Markets. The quintessential case of collusion in mixed-motive settings is markets, in which efficiency results from competition, not cooperation. While this is not a new problem, collusion between AI systems is especially concerning since they may operate inscrutably due to the speed, scale, complexity, or subtlety of their actions.17 Warnings of this possibility have come from technologists, economists, and legal scholars (Beneke & Mackenrodt, 2019; Brown & MacKay, 2023; Ezrachi & Stucke, 2017; Harrington, 2019; Mehra, 2016). Importantly, AI systems can collude even when collusion is not int","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.03.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Collusion ","risk_subcategory":"Steganography ","description":"\"Steganography. In the near future we will likely see LLMs communicating with each other to jointly accomplish tasks. To try to prevent collusion, we could monitor and constrain their communication (e.g., to be in natural language). However, models might secretly learn to communicate by concealing messages within other, non-secret text. Recent work on steganography using ML has demonstrated that this concern is well-founded (Hu et al., 2018; Mathew et al., 2024; Roger & Greenblatt, 2023; Schroeder de Witt et al., 2023b; Yang et al., 2019, see also Case Study 5). Secret communication could also","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.04.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Information Asymmetries","risk_subcategory":null,"description":"\"Information asymmetries (Section 3.1): private information can lead to miscoordination, deception, and conflict;\"","entity":"AI","intent":"Other","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.04.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Information Asymmetries","risk_subcategory":"Communication constraints","description":"\"Communication Constraints. A fundamental source of information asymmetries is that constraints on information exchange can exist, even when agents share a common goal (see Section 2.1). These might be constraints on space (i.e., the amount of information that can be communicated) if the information that needs to be communicated is especially complex, time if a snap decision is required before all information can be communicated, or both.\"","entity":"Other","intent":"Other","timing":"Other","domain":7,"subdomain":"7.6"},{"ev_id":"63.04.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Information Asymmetries","risk_subcategory":"Bargaining ","description":"\"Bargaining. As a classic example of these strategic considerations is that when agents attempt to come to an agreement despite diverging interests, information asymmetries can lead to bargaining inef- ficiencies (Myerson & Satterthwaite, 1983). Relevant uncertainties about other agents can include how much they value possible agreements, their outside options, or their beliefs about others. The essential reason for such inefficiencies is that, under uncertainty about their counterparties, agents must make a trade-off between the rewards of making more favourable demands and the risk of other ","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.04.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Information Asymmetries","risk_subcategory":"Deception ","description":null,"entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.05.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Network Effects ","risk_subcategory":null,"description":"\"Network effects (Section 3.2): minor changes in properties or connection patterns of agents in a network can lead to dramatic changes in the behaviour of the whole group;\"","entity":"AI","intent":"Other","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.05.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Network Effects ","risk_subcategory":"Error propagation ","description":"\"Error Propagation. One well-known issue with communication networks is that information can be corrupted as it propagates through the network.24 As AI systems become capable of generating and processing more and more kinds of information, AI agents could end up ‘polluting the epistemic commons’ (Huang & Siddarth, 2023; Kay et al., 2024) of both other agents (Ju et al., 2024) and humans (see Case Study 7 and Section 3.1) Another increasingly important framework is the use of individual AI agents as part of teams and scaffolded chains of delegation, which transmit not only information but instr","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.05.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Network Effects ","risk_subcategory":"Network rewiring ","description":"\"Network Rewiring. A different class of problems concerns not changes in the content transmitted through the network but changes in the network structure itself (Albert et al., 2000).\"","entity":"Other","intent":"Other","timing":"Other","domain":7,"subdomain":"7.6"},{"ev_id":"63.05.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Network Effects ","risk_subcategory":"Homogeneity and correlated failures","description":"\"Homogeneity and Correlated Failures. The current paradigm driving the state of the art in AI is the ‘foundation model’ (Bommasani et al., 2021): large-scale ML models pre-trained on broad data, which can be repurposed for a wide range of downstream applications. The costs required to create such models (and continuing returns to scale) means that only well-resourced actors can create cutting- edge models (Epoch, 2023; Hoffmann et al., 2022; Kaplan et al., 2020), making them relatively few in number. If current trends continue, it is likely that many AI agents will be powered by a small number","entity":"Other","intent":"Other","timing":"Other","domain":7,"subdomain":"7.6"},{"ev_id":"63.06.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Selection Pressures","risk_subcategory":null,"description":"\"Selection pressures (Section 3.3): some aspects of training and selection by those deploying and using AI agents can lead to undesirable behaviour;\"","entity":"Human","intent":"Unintentional","timing":"Pre-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.06.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Selection Pressures","risk_subcategory":"Undesirable Dispositions from Competition","description":"\"Undesirable Dispositions from Competition. It is plausible that evolution selected for certain conflict-prone dispostions in humans, such as vengefulness, aggression, risk-seeking, selfishness, dishon- esty, deception, and spitefulness towards out-groups (Grafen, 1990; Han, 2022; Konrad & Morath, 2012; McNally & Jackson, 2013; Nowak, 2006; Rusch, 2014). Such traits could also be selected for in ML systems that are trained in more competitive multi-agent settings. For example, this might happen if systems are selected based on their performance relative to other agents (and so one agent’s loss","entity":"Other","intent":"Unintentional","timing":"Other","domain":7,"subdomain":"7.6"},{"ev_id":"63.06.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Selection Pressures","risk_subcategory":"Undesirable Dispositions from Human Data","description":"\"Undesirable Dispositions from Human Data. It is well-understood that models trained on human data – such as being pre-trained on human-written text or fine-tuned on human feedback – can exhibit human biases. For these reasons, there has already been considerable attention to measuring biases related to protected characteristics such as sex and ethnicity (e.g., Ferrara, 2023; Liang et al., 2021; Nadeem et al., 2020; Nangia et al., 2020), which can be amplified in multi-agent settings (Acerbi & Stubbersfield, 2023, see also Case Study 7). More recently, there has been increasing attention paid ","entity":"Other","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.06.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Selection Pressures","risk_subcategory":"Undesirable Capabilities","description":"\"Undesirable Capabilities. As agents interact, they iteratively exploit each other’s weaknesses, forc- ing them to address these weaknesses and gain new capabilities. This co-adaptation between agents can quickly lead to emergent self-supervised autocurricula (where agents create their own challenges, driving open-ended skill acquisition through interaction), generating agents with ever-more sophisticated strate- gies in order to out-compete each other (Leibo et al., 2019). This effect is so powerful that harnessing it has been critical to the success of superhuman systems, such as the use of ","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.07.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Destabilising Dynamics ","risk_subcategory":null,"description":"\"Destabilising dynamics (Section 3.4): systems that adapt in response to one another can produce dangerous feedback loops and unpredictability;\"","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.07.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Destabilising Dynamics ","risk_subcategory":"Feedback Loops","description":"\"Feedback Loops. One of the best-known historical examples to illustrate destabilising dynamics in the context of autonomous agents is the 2010 flash crash, in which algorithmic trading agents entered into an unexpected feedback loop (Commission & Commission, 2010, see also Case Study 10).37 More generally, a feedback loop occurs when the output of a system is used as part of its input, creating a cycle that can either amplify or dampen the system’s behaviour. In multi-agent settings, feedback loops often arise from the interactions between agents, as each agent’s actions affect the environmen","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.07.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Destabilising Dynamics ","risk_subcategory":"Cyclic Behaviour","description":"\"Cyclic Behaviour. The dynamics described above are highly non-linear (small changes to the system’s state can result in large changes to its trajectory). Similar non-linear dynamics can emerge in multi- agent learning and lead to a variety of phenomena that do not occur in single-agent learning (Barfuss et al., 2019; Barfuss & Mann, 2022; Galla & Farmer, 2013; Leonardos et al., 2020; Nagarajan et al., 2020). One of the simplest examples of this phenomenon is Q-learning (Watkins & Dayan, 1992): in the case of a single agent, convergence to an optimal policy is guaranteed under modest condition","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.07.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Destabilising Dynamics ","risk_subcategory":"Chaos","description":"\"Chaos. Unlike the systems that tend towards fixed points or cycles described above, chaotic systems are inherently unpredictable and highly sensitive to initial conditions. While it might seem easy to dismiss such notions as mathematical exoticisms, recent work has shown that, in fact, chaotic dynamics are not only possible in a wide range of multi-agent learning setups (Andrade et al., 2021; Galla & Farmer, 2013; Palaiopanos et al., 2017; Sato et al., 2002; Vlatakis-Gkaragkounis et al., 2023), but can become the norm as the number of agents increases (Bielawski et al., 2021; Cheung & Piliour","entity":"AI","intent":"Other","timing":"Other","domain":7,"subdomain":"7.6"},{"ev_id":"63.07.04","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Destabilising Dynamics ","risk_subcategory":"Phase Transitions","description":"\"Phase Transitions. Finally, small external changes to the system – such as the introduction of new agents or a distributional shift – can cause phase transitions, where the system undergoes an abrupt qualitative shift in overall behaviour (Barfuss et al., 2024). Formally, this corresponds to bifurcations in the system’s parameter space, which lead to the creation or destruction of dynamical attractors, resulting in complex and unpredictable dynamics (Crawford, 1991; Zeeman, 1976). For example, Leonardos & Piliouras (2022) show that changes to the exploration hyperparameter of RL agents can le","entity":"Other","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.07.05","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Destabilising Dynamics ","risk_subcategory":"Distributional Shift","description":"\"Distributional Shift. Individual ML systems can perform poorly in contexts different from those in which they were trained. A key source of these distributional shifts is the actions and adaptations of other agents (Narang et al., 2023; Papoudakis et al., 2019; Piliouras & Yu, 2022), which in single-agent approaches are often simply or ignored or at best modelled exogenously. Indeed, the sheer number and variance of behaviours that can be exhibited other agents means that multi-agent systems pose an especially challenging generalisation problem for individual learners (Agapiou et al., 2022; L","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.08.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Commitment and Trust ","risk_subcategory":null,"description":"\"Commitment and trust (Section 3.5): difficulties in forming credible commitments, trust, or reputation can prevent mutual gains in AI-AI and human-AI interactions;\"","entity":"Other","intent":"Other","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.08.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Commitment and Trust ","risk_subcategory":"Inefficient Outcomes","description":"\"Inefficient Outcomes. Without careful planning and the appropriate safeguards, we may soon be entering a world overrun by increasingly competent and autonomous software agents, able to act with little restriction. The abilities of these agents to persuade, deceive, and obfuscate their activities, as well as the fact they can be deployed remotely and easily created or destroyed by their deployer, means that by default they may garner little trust (from humans or from other agents). Such a world may end up being rife with economic inefficiencies (Krier, 2023; Schmitz, 2001), political problems ","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.08.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Commitment and Trust ","risk_subcategory":"Threats and Extortion","description":"\"Threats and Extortion. A natural solution to problems of trust is to provide some kind of com- mitment ability to AI agents, which can be used to bind them to more cooperative courses of action. Unfortunately, the ability to make credible commitments may come with the ability to make credible threats, which facilitate extortion and could incentivize brinkmanship (see Section 2.2).\"","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.08.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Commitment and Trust ","risk_subcategory":"Rigidity and Mistaken Commitments","description":"\"Rigidity and Mistaken Commitments. Even when it is desirable to be able to make threats in order to deter socially harmful behaviour, doing so using AI agents effectively removes the human from the loop, which could prove disastrous in high-stakes contexts (e.g., a false positive in a nuclear sub- marine’s warning system; see also Case Study 11), or when irresponsible actors are enabled in making disproportionate or mistaken commitments.\"","entity":"Human","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.09.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Emergent Agency ","risk_subcategory":null,"description":"\"Emergent agency (Section 3.6): qualitatively different goals or capabilities can emerge from the composition of innocuous independent systems or behaviours;\"","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.09.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Emergent Agency ","risk_subcategory":"Emergent Capabilities","description":"\"Emergent Capabilities. Dangerous emergent capabilities could arise when a multi-agent system over- comes the safety-enhancing limitations of the individual systems, such as individual models’ narrow domains of application or myopia caused by a lack of long-term planning and long-term memory. For example, narrow systems for research planning, predicting the properties of molecules, and synthesising new chemicals could, when combined, lead to a complex ‘test and iterate’ automated workflow capable of designing dangerous new chemical compounds far beyond the scope of the initial systems’ capabil","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.09.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Emergent Agency ","risk_subcategory":"Emergent Goals","description":"\"Emergent Goals. Ascribing goals to a system is not always straightforward. For our present purposes, it will suffice to adopt a Dennetian perspective (Dennett, 1971), ascribing goals and intentions only when it is useful (i.e., predictive) to do so.51 While it might not be helpful to describe individual narrow AI tools as having goals, their combination may act as a (seemingly) goal-directed collective. For example, a group of moderation bots on a major social networking site could subtly but systematically manipulate the overall political perspectives of the user population, even though, ind","entity":"AI","intent":"Unintentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.10.00","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Category","risk_category":"Multi-Agent Security ","risk_subcategory":null,"description":"\"Multi-agent security (Section 3.7): multi-agent systems give rise to new kinds of security threats and vulnerabilities.\"","entity":"Other","intent":"Other","timing":"Other","domain":7,"subdomain":"7.6"},{"ev_id":"63.10.01","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Multi-Agent Security ","risk_subcategory":"Swarm Attacks","description":"\"Swarm Attacks. The need for multi-agent security is foreshadowed by attacks today that benefit from the use of many decentralised agents, such as distributed denial-of-service attacks (Cisco, 2023; Yoachimik & Pacheco, 2024). Such attacks exploit the massive collective resources of individual low- resourced actors, chained into an attack that breaks the assumptions of bandwidth constraints on a single well-resourced agent.\"","entity":"Human","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.10.02","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Multi-Agent Security ","risk_subcategory":"Heterogeneous Attacks","description":"\"Heterogeneous Attacks. A closely related risk is the possibility of multiple agents combining different affordances to overcome safeguards, for which there is already preliminary evidence (Jones et al., 2024, see also Case Study 12). In this case, it is not the sheer number of agents that leads to the novel attack method, but the combination of their different abilities. This might include the agents’ lack of individual safeguards, tasks that they have specialised to complete, systems or information that they may have access to (either directly or via training), or other incidental features s","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.10.03","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Multi-Agent Security ","risk_subcategory":"Social Engineering at Scale","description":"\"Social Engineering at Scale. Advanced AI agents will be more easily able to interact with large numbers of humans, and vice versa. This provides a wider attack surface for various forms of automated social engineering (Ai et al., 2024). For example, coordinated agents could use advanced surveillance tools and produce personalized phishing or manipulative content at scale, adjusting their tactics based on user feedback (Figueiredo et al., 2024; Hazell, 2023). A large number of subtle interactions with a range of seemingly independent AI agents might be more likely to lead to someone being pers","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.10.04","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Multi-Agent Security ","risk_subcategory":"Vulnerable AI Agents","description":"\"Vulnerable AI Agents. The use of AI agents as delegates or representatives of humans or organisa- tions also introduces the possibility of attacks on AI agents themselves. In other words, agents can be considered vulnerable extensions of their principals, introducing a novel attack surface (SecureWorks, 2023). Attacks on an AI agent could be used to extract private information about their principal (Wei & Liu, 2024; Wu et al., 2024a), or to manipulate the agent to take actions that the principal would find undesirable (Zhang et al., 2024a). This includes attacks that have direct relevance for","entity":"Other","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.10.05","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Multi-Agent Security ","risk_subcategory":"Cascading Security Failures","description":"\"Cascading Security Failures. Localised attacks in multi-agent systems can result in catastrophic macroscopic outcomes (Motter & Lai, 2002, see also Sections 3.2 and 3.4). These cascades can be hard to mitigate or recover from because component failure may be difficult to detect or localise in multi-agent systems (Lamport et al., 1982), and authentication challenges can facilitate false flag attacks (Skopik & Pahi, 2020). Computer worms represent a classic example of a cybersecurity threat that relies inherently on networked systems. Recent work has provided preliminary evidence that similar a","entity":"Other","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"},{"ev_id":"63.10.06","quick_ref":"Hammond2025","paper_title":"Multi-Agent Risks from Advanced AI ","level":"Risk Sub-Category","risk_category":"Multi-Agent Security ","risk_subcategory":"Undetectable Threats","description":"\"Undetectable Threats. Cooperation and trust in many multi-agent systems relies crucially on the ability to detect (and then avoid or sanction) adversarial actions taken by others (Ostrom, 1990; Schneier, 2012). Recent developments, however, have shown that AI agents are capable of both steganographic communication (Motwani et al., 2024; Schroeder de Witt et al., 2023b) and ‘illusory’ attacks (Franzmeyer et al., 2023), which are black-box undetectable and can even be hidden using white-box undetectable encrypted backdoors (Draguns et al., 2024). Similarly, in environments where agents learn fr","entity":"AI","intent":"Intentional","timing":"Post-deployment","domain":7,"subdomain":"7.6"}]}