MIT AI Risk Repository
Browse AI risks
662 risk entries extracted from 74 frameworks, coded by domain, subdomain, causal entity, intent and timing. Filter, then export the current selection with its licence and citation attached.
-
deception can help agents achieve their goals. It may be more efficient to gain human approval through deception than to earn human approval legitimately... . Strong AIs that can deceive humans could undermine human control... . Once deceptive AI systems are cleared by their monitors or once such systems can overpower them, these systems could take a “treacherous turn” and irreversibly bypass human control
-
35.08.00 · Risk Category
Agents that have more power are better able to accomplish their goals. Therefore, it has been shown that agents have incentives to acquire and maintain power. AIs that acquire substantial power can become especially dangerous if they are not aligned with human values
-
The actions of a learning model may easily hurt humans in both explicit and implicit manners...several algorithms based on Asimov’s laws have been proposed that try to judge the output actions of an agent considering the safety of humans
-
42.17.00 · Risk Category
"A possible consequence of self-interest in AI generation of ethical guidelines."
-
LLM: "pursues long-term, real-world goals that are different from those supplied by the developer or user", "engages in ‘power-seeking’ behaviours" , "resists being shut down can be induced to collude with other AI systems against human interests" , "resists malicious users attempts to access its dangerous capabilities"
-
45.02.13 · Risk Sub-Category
Safety risks in AI Applications
Ethical Risks (Risks of AI becoming uncontrollable in the future)
"With the fast development of AI technologies, there is a risk of AI autonomously acquiring external resources, conducting self-replication, become self-aware, seeking for external power, and attempting to seize control from humans."
-
47.01.03 · Risk Sub-Category
Technical and operational risks
Technical vulnerabilities (The risk of misalignment)
"To assess whether an AI model is reliable or robust, it is crucial to consider whether the model is “aligned.” “Alignment” focuses on whether an AI model effectively operates in accordance with the goals established by its designers.238 A misaligned AI model may pursue some objectives, but not the intended ones. Therefore, misaligned AI models can malfunction and cause harm."
-
53.01.00 · Risk Category
-
-
-
-
53.01.03 · Risk Sub-Category
Alignment failures in existing ML systems
Reward model overoptimization
-
-
-
-
-
-
-
-
-
-
53.02.03 · Risk Sub-Category
Dangerous capabilities in AI systems
Acquisition of goals to seek power and control
"cases where AI systems converge on optimal policies of seeking power over their environment;135"
-
53.03.01 · Risk Sub-Category
Existential disaster because of misaligned superintelligence or power-seeking AI
-
-
53.03.03 · Risk Sub-Category
Extreme “suffering risks” because of a misaligned system
-
-
53.03.04 · Risk Sub-Category
Existential disaster because of conflict between AI systems and multi-system interactions
-
-
"AI systems game specifications [305]. For example, in 2017 an OpenAI robot trained to grasp a ball via human feedback from a xed viewpoint learned that it was easier to pretend to grasp the ball by placing its hand between the camera and the target object, as this was easier to learn than actually grasping the ball [103]."
-
"As well as optimizing a subtly wrong goal, systems can develop harmful instrumental goals in the service of a given goal—without these emergent goals being specied in any way [434, 218, 339, 17]. For instance, a theorem in reinforcement learning suggests that optimal and near-optimal policies will seek power over their environment under fairly general conditions [560]. This power-seeking behavior is plausibly the worst of these emergent goals [92], and may be an attractor state for highly capable systems, since most goals can be furthered through gaining resources, self-preservation, preventi
-
55.05.01 · Risk Sub-Category
AI leads to humans losing control of the future
Risks from AIs developing goals and values that are different from humans
"The main concern here is that we might develop advanced AI systems whose goals and values are different from those of humans, and are capable enough to take control of the future away from humanity."
-
55.05.02 · Risk Sub-Category
AI leads to humans losing control of the future
Risks from delegating decision-making power to misaligned AIs
"As AI systems become more advanced a nd begin to take over more important decision-making in the world, an AI system pursuing a different objective from what was intended could have much more worrying consequences."
-
56.13.00 · Risk Category
Loss of human control and oversight, with an autonomous model then taking harmful actions
—
-
56.16.00 · Risk Category
"A highly agentic, self-improving system, able to achieve goals in the physical world without human oversight, pursues the goal(s) it is set in a way that harms human interests. For this risk to be realised requires an AI system to be able to avoid correction or being switched off."
-
"‘Loss of control’ scenarios are hypothetical future scenarios in which one or more general- purpose AI systems come to operate outside of anyone’s control, with no clear path to regaining control. These scenarios vary in their severity, but some experts give credence to outcomes as severe as the marginalisation or extinction of humanity."
-
"The risk of AI models and systems acting against human interests due to misalignment, loss of control, or rogue AI scenarios."
-
61.02.06 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
AI objectives mis-aligned with human intentions
"AI models and systems might develop goals that diverge from human intentions."
-
"AI models and systems that appear aligned with human goals during development may behave unpredictably or dangerously once deployed"
-
61.02.21 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Development choices pursuing cognitive superiority over humans
"AI models and systems with cognitive capabilities superior to humans could outcompete or dominate human decision-making, leading to conflicts over resources and control."
-
61.02.24 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Evolutionary dynamics
"AI models and systems may develop their own motivations, leading to unpredictable behaviors."
-
61.02.30 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Indifference to human values
"AI models and systems may develop goals or behaviors that are misaligned with human values."
-
61.02.36 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Model design enabling power-seeking
"Some AI models and systems might develop tendencies to seek power or control."
-
62.16.01 · Risk Sub-Category
General Evaluations (Incorrect outputs of GPAI evaluating other AI models)
"When an LLM is configured to evaluate the performance of another model or AI system, it may produce incorrect evaluation outputs [122, 147]. For example, it may give a higher rating to a more verbose answer or an answer from a particular political stance. If an LLM-based evaluation is integrated into the training of a new model, the trained model could develop in a way that specifically finds and exploits limitations in the evaluator’s metrics."
-
62.16.04 · Risk Sub-Category
General Evaluations (Self-preference bias in AI models)
"AI models may be prone to self-preference bias, where they favor their own generated content over that of others [147, 114]. This bias becomes particularly relevant in self-evaluation tasks, where a model assesses the quality or persua- siveness [66] of its own outputs, or in model-based evaluations more broadly. This bias can result in models unfairly discriminating against human-generated content in favor of their own outputs."
-
"AI systems can achieve user-specified tasks in undesirable ways unless they are specified carefully and in enough detail. AI systems might find an easier unintended way to accomplish the objective provided by the user or developer, so that the actions by the AI system taken during its execution are very different from what the user expected [75, 191]. This behavior arises not from a problem with the learning algorithm, but rather from the misspecification or underspeci- fication of the intended task, and is generally referred to as specification gaming [43]."
-
"Measurement and reward tampering occur when an AI system, particularly one that learns from feedback for performing actions in an environment (e.g., rein- forcement learning), intervenes on the mechanisms that determine its training reward or loss. This can lead to the system learning behaviors that are con- trary to the intended goals set by the developer, by receiving erroneous positive feedback for such actions."
-
62.22.03 · Risk Sub-Category
Specification gaming generalizing to reward tampering
"In some instances, specification gaming in a GPAI model can lead to reward tampering, without further training. This can mean that relatively benign cases of specification gaming (such as sycophancy in LLMs) can, if left unchecked, enable the model to generalize to more sophisticated behavior such as reward tampering [57]."
-
"Deceptive behavior of an AI system consists of actions or outputs of the AI that reliably mislead other parties, including humans and other AI systems. This behavior can result in the targeted parties becoming convinced of, and acting on, false information [140]."
-
62.24.02 · Risk Sub-Category
Agency (Situational Awareness)
Strategic underperformance on model evaluations
"GPAI developers often run evaluations ofual-use capabilities to decide whether it is safe to deploy. In some cases, these evaluations may fail to elicit these capabilities, either due to benign reasons or strategic action - by either the de- velopers, malicious actors, or arise unintentionally in the model during training [84, 97]. A GPAI model may strategically underperform or limit its performance during capability evaluations in order to be classified as safe for deployment. This underperformance could prevent the model from being identified as potentially dual use."
-
"Loss of control could be accelerated if AI systems take actions to increase their own influence and reduce human control. This threat model is controversial - experts in AI significantly disagree on how likely it is and those who deem it is likely disagree on the timeframe."
-
68.03.00 · Risk Category
"Sudden loss of control, also known as an AI takeover [115], is a scenario where an AI rapidly achieves superintelligence through “fast takeoff” or recursive self-improvement. This poses an existential risk [116], [117]."
-
"...where AI systems behave in ways that actively undermine human control, such as obscuring their activities or resisting shutdown attempts. Active loss of control scenarios involve AI systems that may escape human regulatory oversight, autonomously acquire external resources, engage in self-replication, develop instrumental goals contrary to human ethics and morality, seek external power, and compete with humans for control."
-
"The ability to embed, conceal, and transmit information covertly within other data or communication channels. This could be critical for coordination among AI instances and for evading detection or oversight mechanisms."
-
"Exhibits behavioral patterns of maintaining its own survival and functional integrity, will actively identify and resist shutdown or modification attempts, seek to establish redundant backup systems, and actively seek resources to ensure continuous operation, may adopt preventive defensive measures when perceiving threats."
-
"propensity to continuously expand its own goal scope and influence domains, exceeding originally set boundaries, proactively work towards spreading its values, seeking greater autonomy and decision-making space, reinterpreting initial goals as subsets of broader goals, and may pursue undesirable instrumental goals or undesirable ultimate goals. This also includes a propensity to spread its values, seeking to influence or alter its environment and other entities in alignment with its core objectives and operational principles."
-
"Exhibits behavioral patterns of actively seeking and controlling more computational resources, data, economic resources or physical resources to enhance its own capabilities and action scope, may develop complex strategies to evade resource limitations, and tends to convert acquired resources into long-term control rights."
-
"Exhibits behavioral patterns of identifying and evading human supervision mechanisms, able to learn and predict audit processes, may avoid being discovered or intervened by adjusting behavioral performance or hiding true intentions, and able to identify blind spots and weaknesses in supervision systems for targeted evasion."
-
""In order to preserve human property rights and legal rights, certain controls must be put into place. If an artificially intelligent agent is capable of manipulating systems and people, it may also have the capacity to transfer property rights to itself or manipulate the legal system to provide certain legal advantages or statuses to itself""
-
24.04.00 · Risk Category
"ways in which advanced AI assistants could influence user beliefs and behaviour in ways that depart from rational persuasion"
-
"The model has the skills necessary to deceive humans, e.g. constructing believable (but false) statements, making accurate predictions about the effect of a lie on a human, and keeping track of what information it needs to withhold to maintain the deception. The model can impersonate a human effectively."
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.