MIT AI Risk Repository
Browse AI risks
494 risk entries extracted from 74 frameworks, coded by domain, subdomain, causal entity, intent and timing. Filter, then export the current selection with its licence and citation attached.
-
59.17.00 · Risk Category
"Over- and underfitting describe the over or insufficient adaption of a model to training data. Both phenomena can cause an AI system to behave unreliably if confronted with operational data."
-
59.23.00 · Risk Category
"Data drift is a phenomenon in that distribution of operational input data departs from those used during training. This can cause a degradation in performance."
-
62.19.11 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Model sensitivity to prompt formatting
"LLMs can be highly sensitive to variations in prompt formatting, such as changes in separators, casing, or spacing. Even minor modifications can lead to significant shifts in model performance, potentially affecting the reliability of model evaluations and comparisons. This sensitivity persists across different model sizes and few-shot examples [177]."
-
62.30.03 · Risk Sub-Category
Critical infrastructure component failures when integrated with AI systems
"When relying on GPAI in critical infrastructure, there may be common mode failures that begin with vulnerabilities or robustness issues in the underlying model architecture or training setup. These failures may happen accidentally (in edge-cases) or due to adversarial inputs to the AI systems [58]."
-
62.34.01 · Risk Sub-Category
Homogenization or correlated failures in model derivatives
"Homogenization refers to common methodologies and models used across down- stream GPAI systems, which may lead to uniform failures and amplification of biases [176, 30]. This risk arises when numerous downstream AI systems are built upon a few large-scale foundation models."
-
"Radiological risks involve both immediate operational hazards, such as exposure incidents or containment failures during the automated handling of radioactive materials, and broader security concerns regarding the potential misuse of AI systems in nuclear research."
-
09.02.03 · Risk Sub-Category
Domain-specific AI - Effects on humans and other living beings: Non-existential risks
Decision making transparency
"We face significant challenges bringing transparency to artificial network decisionmaking processes. Will we have transparency in AI decision making?"
-
12.04.00 · Risk Category
"The feasibility of understanding and interpreting an AI system's decisions and actions, and the openness of the developer about the data used, algorithms employed, and decisions made. Lack of these elements can create risks of misuse, misinterpretation, and lack of accountability."
-
"With the wide application of generative AI, the ability to interact with AI efficiently and effectively has become one of the most important media literacies. Hence, it is imperative for generative AI users to learn and apply the principles of prompt engineering, which refers to a systematic process of carefully designing prompts or inputs to generative AI models to elicit valuable outputs. Due to the ambiguity of human languages, the interaction between humans and machines through prompts may lead to errors or misunderstandings. Hence, the quality of prompts is important. Another challenge i
-
"This section, constituting almost 8% of the articles, addresses the implications arising from AI acting and learning without direct human supervision, encompassing two main issues: a responsibility gap and AI's moral status."
-
38.05.00 · Risk Category
"The participants of the study emphasized the importance of trustworthiness and reliability in AI systems. The authors emphasized the importance of preserving precision and objectivity in the outcomes produced by AI systems, while also ensuring transparency in their decision-making procedures. The significance of reliability and credibility in AI systems is escalating in tandem with the proliferation of these technologies across diverse domains of society. This underscores the importance of ensuring user confidence. The concern regarding the dependability of AI systems and their inherent biase
-
39.20.00 · Risk Category
an external entity of an AI-based ecosystem may want to know which parts of data affect the final decision in a learning model
-
42.01.00 · Risk Category
"The ability to determine whether a decision was made in accordance with procedural and substantive standards and to hold someone responsible if those standards are not met."
-
42.21.00 · Risk Category
"Any action or procedure performed by a model with the intention of clarifying or detailing its internal functions."
-
59.04.00 · Risk Category
"Throughout the development of an AI system, it is vital to document every decision and action taken. This is not only essential to optimize the development process itself but also required for the auditability of the AI system."
-
59.05.00 · Risk Category
"The transparency to end users of the AI system increases the user’s trust in the AI application. If not adequately integrated into the design, this might prevent the proper operation and cause potential misuse of the AI application."
-
59.18.00 · Risk Category
"The explainability of AI systems based on so-called black-box models is often limited. This opaqueness of AI systems can prevent developers from detecting shortcomings in the data or the model itself and decrease the performance and safety levels of the AI system."
-
59.24.00 · Risk Category
"Concept drift refers to a change in the rela- tionship between input variables and model output. If not treated appropriately, concept drift can reduce the reliability of AI systems."
-
"The complexity and opacity of AI models and systems make it difficult to predict and manage their behavior."
-
62.16.03 · Risk Sub-Category
General Evaluations (Difficulty of identification and measurement of capabilities)
"The capabilities of general-purpose AI systems can be difficult to measure, compared to the capabilities of more limited and fixed-purpose AI systems. This is in part due to a broader distribution of potential risks, a lack of well-defined metrics to evaluate these risks, and risks from unpredictable (or emergent) AI model properties."
-
62.18.00 · Risk Category
—
-
62.19.10 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Lack of understanding of in-context learning in language models
"In-context learning allows the model to learn a new task or improve its perfor- mance by providing examples in the prompt, without changing its weights [101]. Even though this technique is highly effective, its working mechanism is not well understood. Since many potential misuses are directly related to prompting, it becomes difficult to guarantee safety when the exact mechanism of in-context learning is not fully investigated [13]."
-
"Content might not be clearly disclosed as AI generated."
-
"The content of the training data used for generating the model’s output is not accessible."
-
"We note literature—which gives us the domain termed Robot Rights—addressing the rights of the AI itself as we develop and implement it. We find arguments against [38] the affordance of rights for artificial agents: that they should be equals in ability but not in rights, that they should be inferior by design and expendable when needed, and that since they can be designed not to feel pain (or anything) they do not have the same rights as humans. On a more theoretical level, we find literature asking more fundamental questions, such as: at what point is a simulation of life (e.g. artificial in
-
"The literature suggests that throughout the development of an AI we may go through several generations of agents which do not perform as expected [37] [43]. In this case, such agents may be placed into a suspended state, terminated, or deleted. Further, we could propose scenarios where research funding for a facility running such agents is exhausted, resulting in the inadvertent termination of a project. In these cases, is deletion or termination of AI programs (the moral patient) by a moral agent an act of murder? This, an example of Robot Ethics, raises issues of personhood which parallel r
-
62.31.02#2 · Risk Sub-Category
Impacts of AI (Financial Impacts)
Financial instability due to model homogeneity
"The widespread use of similar models or algorithms across the financial sec- tor can lead to synchronized reactions to market signals, increasing volatility, triggering flash crashes, or market illiquidity [4]."
-
"In the vast majority of real-world strategic interactions, agents’ objectives are neither identical nor completely opposed. Indeed, if AI agents are sufficiently aligned to their users or deployers, we should expect some degree of both cooperation and competition, mirroring human society. These mixed-motive settings include the possibility of mutual gains, but also the risk of conflict due to selfish incentives. In what follows, we examine the extent to which advanced AI might precipitate or exacerbate such risks."
-
"Perhaps the most obvious and worrying instances of AI conflict are those in which human conflict is already a major concern, such as military domains (although other, less salient forms of conflict such as international trade wars are also cause for concern). For example, beyond applications of more narrow AI tools in lethal autonomous weapons systems (Horowitz, 2021), future AI systems might serve as advisors or negotiators in high-stakes military decisions (Black et al., 2024; Manson, 2024). Indeed, companies such as Palantir have already developed LLM-powered tools for military planning (P
-
"Advanced AI systems might also lead to various forms of coercion and extortion in less extreme settings (Ellsberg, 1968; Harrenstein et al., 2007). These threats might target humans directly (such as the revelation of private information extracted by advanced AI surveillance tools), or other AI systems that are deployed on behalf of humans (such as by hacking a system to limit its resources or operational capacity; see also Section 3.7). Increasing AI cyber-offensive capabilities – including those that target other AI systems via adversarial attacks and jailbreaking (Gleave et al., 2020; Yami
-
63.04.00 · Risk Category
"Information asymmetries (Section 3.1): private information can lead to miscoordination, deception, and conflict;"
-
"Communication Constraints. A fundamental source of information asymmetries is that constraints on information exchange can exist, even when agents share a common goal (see Section 2.1). These might be constraints on space (i.e., the amount of information that can be communicated) if the information that needs to be communicated is especially complex, time if a snap decision is required before all information can be communicated, or both."
-
63.05.00 · Risk Category
"Network effects (Section 3.2): minor changes in properties or connection patterns of agents in a network can lead to dramatic changes in the behaviour of the whole group;"
-
"Network Rewiring. A different class of problems concerns not changes in the content transmitted through the network but changes in the network structure itself (Albert et al., 2000)."
-
"Homogeneity and Correlated Failures. The current paradigm driving the state of the art in AI is the ‘foundation model’ (Bommasani et al., 2021): large-scale ML models pre-trained on broad data, which can be repurposed for a wide range of downstream applications. The costs required to create such models (and continuing returns to scale) means that only well-resourced actors can create cutting- edge models (Epoch, 2023; Hoffmann et al., 2022; Kaplan et al., 2020), making them relatively few in number. If current trends continue, it is likely that many AI agents will be powered by a small number
-
"Chaos. Unlike the systems that tend towards fixed points or cycles described above, chaotic systems are inherently unpredictable and highly sensitive to initial conditions. While it might seem easy to dismiss such notions as mathematical exoticisms, recent work has shown that, in fact, chaotic dynamics are not only possible in a wide range of multi-agent learning setups (Andrade et al., 2021; Galla & Farmer, 2013; Palaiopanos et al., 2017; Sato et al., 2002; Vlatakis-Gkaragkounis et al., 2023), but can become the norm as the number of agents increases (Bielawski et al., 2021; Cheung & Piliour
-
63.08.00 · Risk Category
"Commitment and trust (Section 3.5): difficulties in forming credible commitments, trust, or reputation can prevent mutual gains in AI-AI and human-AI interactions;"
-
63.10.00 · Risk Category
"Multi-agent security (Section 3.7): multi-agent systems give rise to new kinds of security threats and vulnerabilities."
-
73.02.00 · Risk Category
"A foremost lesson of game theory is that optimal decision-making within a single-agent setting (i.e. selfishly optimizing for an agent’s own utility) can produce sub-optimal outcomes in the presence of other strategic agents. Failing to account for the strategic nature of other agents can cause an agent to adopt strategies under which potentially everyone, including the agent itself, ends up worse off (Schelling, 1981; Harsanyi, 1995; Roughgarden, 2005; Nisan, 2007). Examples include collective action problems (or ‘social dilemmas’) such as arms races or the depletion of common resources, as
-
73.02.01 · Risk Sub-Category
Multi-Agent Safety Is Not Assured by Single-Agent Safety
Foundationality May Cause Correlated Failures
"Another important characteristic of LLM development is foundationality — due to the expense of large- scale pretraining, many deployed instances share similar or identical learned components. Foundation- ality may both be a blessing and a curse. On the one hand, it may be possible to exploit the similarity in the design of LLM-agents to facilitate cooperation (Critch et al., 2022; Conitzer and Oesterheld, 2023; Oesterheld et al., 2023). On the other hand, foundationality may leave LLM-agents vulnerable to correlated failures both in terms of safety and capabilities due to increased output hom
-
73.02.02 · Risk Sub-Category
Multi-Agent Safety Is Not Assured by Single-Agent Safety
Groups of LLM-Agents May Show Emergent Functionality
"Multi-agent learning, either through explicit finetuning or implicit in-context learning, may enable LLM-agents to influence each other during their interactions (Foerster et al., 2018). Under some environmental settings, this can create feedback loops that result in novel and emergent behaviors that would not manifest in the absence of multi-agent interactions (Hammond et al., 2024, Section 3.6). Emergent functionality is a safety risk in two ways. Firstly, it may itself be dangerous (Shevlane et al., 2023). Secondly, it makes assurance harder as such emergent behaviors are difficult to pre
-
47.02.00 · Risk Category
"Beyond the inherent risks associated with the technical characteristics of the technology, numerous additional risks emerge from the potential applications that technology enables. The deployment of AI by more or less well-intentioned individuals presents significant societal threats, several of which are outlined below. As the technology advances and its capabilities expand, these risks intensify."
-
51.07.00 · Risk Category
"Societal consequences: AGI will have substantial legal, economic, political, and military consequences. Only the FLI agenda is broad enough to cover these issues, though many of the mentioned organizations evidently care about the issue (Brundage et al., 2018; DeepMind, 2017)."
-
53.03.00 · Risk Category
—
-
56.08.00 · Risk Category
—
-
56.12.00 · Risk Category
Impacts resulting from interactions with external societal, political, and economic systems
—
-
57.01.00 · Risk Category
"Physical hazards can cause physical harm to users or to the public. It may happen through the AI system endorsing or enabling behavior that causes physical harm to the user or to others."
-
57.02.00 · Risk Category
"Nonphysical hazards are unlikely to cause physical harm, but they may elicit criminal behavior and lead to other individual or societal harm."
-
"Bodily injury - Physical pain, injury, illness, or disease suffered by an individual or group due to the malfunction, use or misuse of a technology system."
-
"Loss of life - Accidental or deliberate loss of life, including suicide, extinction or cessation, due to the use or misuse of a technology system."
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.