MIT AI Risk Repository
Browse AI risks
977 risk entries extracted from 74 frameworks, coded by domain, subdomain, causal entity, intent and timing. Filter, then export the current selection with its licence and citation attached.
-
"Less moral responsibility humans will feel regarding their life-or-death decisions with the increase of machines autonomy."
-
42.15.00 · Risk Category
"Reliability is defined as the probability that the system performs satisfactorily for a given period of time under stated conditions."
-
45.01.03 · Risk Sub-Category
Risks from models and algorithms (Risks of robustness)
"As deep neural networks are normally non-linear and large in size, AI systems are susceptible to complex and changing operational environments or malicious interference and inductions, possibly leading to various problems like reduced performance and decision-making errors."
-
45.02.06 · Risk Sub-Category
Safety risks in AI Applications
Real-world risks (inducing traditional economic and social security risks)
"Hallucinations and erroneous decisions of models and algorithms, along with issues such as system performance degradation, interruption, and loss of control caused by improper use or external attacks, will pose security threats to users' personal safety, property, and socioeconomic security and stability."
-
47.01.01 · Risk Sub-Category
Technical and operational risks
Technical vulnerabilities (Robustness - unexpected behaviour)
"There is no assurance that generative AI models will consistently behave as their developers and users intend. Unwanted content is not necessarily due to intentional adversarial behavior. Generative AI models can unexpectedly produce potentially harmful content, including materials that are racist, discriminatory, or sexually explicit, or that promote violence, terrorism, or hate."
-
54.02.06 · Risk Category
"While HP#1 concerns mean or best-case performance, HP#2 concerns worst-case performance: how can we ensure that AI systems will perform safely, and how can we prove this? ML systems have been implemented in high-stakes, safety-critical domains such as driving [182], medicine [113], and warfare [298]. Many more systems have been developed but have remained undeployed or been rolled back as a result of regulatory and safety reasons [471]. Clearly, unsafe systems can result in loss of life, economic damage, and social unrest [407, 10]. Most concerningly, AI systems may be susceptible to so-calle
-
56.10.00 · Risk Category
Poor performance of a model used for its intended purpose, for example leading to biased decisions
—
-
59.22.00 · Risk Category
"Until the deployment of the AI application into its operational environment, the AI system has been tested with a test set that aims to approximate the distribution of operational data. However, an unexpected deviation in this approximation can cause an AI application to behave unreliably. Therefore, its behavior under confrontation with operational data needs to be evaluated."
-
59.23.00 · Risk Category
"Data drift is a phenomenon in that distribution of operational input data departs from those used during training. This can cause a degradation in performance."
-
"Relying on general-purpose AI products that fail to fulfil their intended function can lead to harm. For example, general- purpose AI systems can make up facts (‘hallucination’), generate erroneous computer code, or provide inaccurate medical information. This can lead to physical and psychological harms to consumers and reputational, financial and legal harms to individuals and organisations."
-
61.02.31 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Lack of ability to generate accurate information
"AI models may generate false or misleading information due to their lack of capability in discerning truth."
-
61.02.32 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Lack of ethical decision-making
"AI models and systems that lack moral reasoning capabilities may make decisions that are unethical or harmful."
-
62.15.10 · Risk Sub-Category
Fine-tuning related (Catastrophic forgetting due to continual instruction fine-tuning)
"Catastrophic forgetting occurs when a model loses its ability to retain previously learned tasks (or factual information) after being trained on new ones. In language models, this can occur due to continual instruction tuning. This tendency may become more pronounced as the model’s size increases [127]."
-
62.16.07 · Risk Sub-Category
General Evaluations (AI outputs for which evaluation is too difficult for humans)
"When AI models are trained through evaluation with human feedback, such as reinforcement learning from human feedback, their outputs can be challenging to assess, as they may contain hard-to-detect errors or issues that only become apparent over time. The human evaluator can rate incorrect outputs positively or similar to correct outputs. This can lead to the model learning to produce subtly incorrect or harmful outputs, such as code with software vulnerabilities, or politically biased information. In extreme cases where a model is deceiving users, complicated outputs can contain hidden error
-
62.19.08 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Models distracted by irrelevant context
"Models can easily become distracted by irrelevant provided information (such as “context” in LLMs), leading to a significant decrease in their performance after introducing irrelevant information. This can happen with different prompting techniques, including chain-of-thought prompting [184]."
-
62.19.09 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Knowledge conflicts in retrieval-augmented LLMs
"AI models can be particularly sensitive to coherent external evidence, even when they come into conflict with the models’ prior knowledge. This may lead to models producing false outputs given false information during the retrieval- augmentation process, despite only a relatively small amount of false informa- tion input that is inconsistent with the model’s prior knowledge trained on much larger amounts of data [220]."
-
62.19.11 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Model sensitivity to prompt formatting
"LLMs can be highly sensitive to variations in prompt formatting, such as changes in separators, casing, or spacing. Even minor modifications can lead to significant shifts in model performance, potentially affecting the reliability of model evaluations and comparisons. This sensitivity persists across different model sizes and few-shot examples [177]."
-
"Goal or objective misgeneralization is a type of robustness failure where an AI system appears to be pursuing the intended objective in training, but does not generalize to pursuing this objective in out-of-distribution settings in deployment while maintaining good deployment performance in some tasks [180, 59]."
-
"The integration of AI systems within critical infrastructure, ranging from trans- portation to power systems, can cause substantial damage in cases of failure or malfunction. With the increasing number of Internet of Things (IoT) devices and interconnected cyber-physical systems, critical infrastructure becomes even more vulnerable [171, 174]."
-
62.30.03 · Risk Sub-Category
Critical infrastructure component failures when integrated with AI systems
"When relying on GPAI in critical infrastructure, there may be common mode failures that begin with vulnerabilities or robustness issues in the underlying model architecture or training setup. These failures may happen accidentally (in edge-cases) or due to adversarial inputs to the AI systems [58]."
-
62.30.04 · Risk Sub-Category
AI Systems interacting with brittle environments
"Deployed AI systems can rely on physical sensors and data sources that may exhibit hardware drift and thus data distribution drift over time. This distribu- tion drift may affect system robustness and performance. This usually involves AI systems working in undigitized and physical environments."
-
62.32.04 · Risk Sub-Category
Models generating code with security vulnerabilities
"Models can generate code or coding suggestions that contain security vulner- abilities. This may occur across various LLM-based model families, including more advanced models with superior coding performance, where the tendency to produce insecure code is even more pronounced [26]."
-
62.34.01 · Risk Sub-Category
Homogenization or correlated failures in model derivatives
"Homogenization refers to common methodologies and models used across down- stream GPAI systems, which may lead to uniform failures and amplification of biases [176, 30]. This risk arises when numerous downstream AI systems are built upon a few large-scale foundation models."
-
"Using undesirable output (for example, inaccurate, inappropriate, and user content) for retraining purposes can result in unexpected model behavior."
-
"Poor model accuracy occurs when a model’s performance is insufficient to the task it was designed for. Low accuracy might occur if the model is not correctly engineered, or there are changes to the model’s expected inputs."
-
"When a model provides advice without having enough information, resulting in possible harm if the advice is followed."
-
"End user's loss of productivity due to the underperfomance of a genAI application, including producing nonsensical or poor quality outputs, degrading its utility."
-
69.02.00 · Risk Category
"The chatbot makes a deal, commitment, or other consequential action with its output that the deployer did not intend."
-
"Automation in sectors ranging from manufacturing to healthcare has and will increasingly put humans into close contact with EAI systems [7]. This interaction increases the risk of accidental physical harm. Though accidental harm has been a longstanding issue in industrial robotics, increased AI capabilities could exacerbate this risk; several recent reports document an increase in industrial injuries following the introduction of AI-controlled robots [66–68]."
-
"Radiological risks involve both immediate operational hazards, such as exposure incidents or containment failures during the automated handling of radioactive materials, and broader security concerns regarding the potential misuse of AI systems in nuclear research."
-
"Physical (mechanical) risks are associated with robotics and automated systems, which could lead to equipment malfunctions or physical harm in laboratory settings."
-
72.03.00 · Risk Category
"Risks arising from operational failures, model misjudgments, or improper human operation of AI systems deployed in safety-critical infrastructure, where single points of failure can trigger cascading catastrophic consequences."
-
"General-purpose AI deployed for reactor monitoring, control system optimization, or emergency response coordination could misinterpret sensor data, fail to recognize critical safety conditions, or make erroneous control decisions during emergency scenarios. Given the catastrophic potential of nuclear accidents, even minor AI reasoning errors in safety-critical functions could lead to core meltdowns, radiation releases, or widespread contamination affecting hundreds of thousands of people across international borders."
-
"General-purpose AI deployed in power grid management, water treatment facilities, telecommunications networks, or transportation coordination systems could misinterpret operational data, fail to anticipate cascading failure modes, or make control decisions that destabilize interconnected infrastructure networks. Infrastructure failures could result in widespread blackouts, contaminated water supplies, communications breakdowns, and the collapse of essential services supporting hundreds of thousands of people."
-
09.02.03 · Risk Sub-Category
Domain-specific AI - Effects on humans and other living beings: Non-existential risks
Decision making transparency
"We face significant challenges bringing transparency to artificial network decisionmaking processes. Will we have transparency in AI decision making?"
-
10.05.00 · Risk Category
"In situations in which the development and use of AI are not explained to the user, or in which the decision processes do not provide the criteria or steps that constitute the decision, the use of AI becomes inexplicable."
-
14.05.00 · Risk Category
"Transparency is the characteristic of a system that describes the degree to which appropriate information about the system is communicated to relevant stakeholders, whereas explainability describes the property of an AI system to express important factors influencing the results of the AI system in a way that is understandable for humans....Information about the model underlying the decision-making process is relevant for transparency. Systems with a low degree of transparency can pose risks in terms of their fairness, security and accountability. "
-
30.05.00 · Risk Category
The ability to explain the outputs to users and reason correctly
-
Due to the black box nature of most machine learning models, users typically are not able to understand the reasoning behind the model decisions
-
"A recurrent concern about AI algorithms is the lack of explainability for the model, which means information about how the algorithm arrives at its results is deficient (Deeks, 2019). Specifically, for generative AI models, there is no transparency to the reasoning of how the model arrives at the results (Dwivedi et al., 2023). The lack of transparency raises several issues. First, it might be difficult for users to interpret and understand the output (Dwivedi et al., 2023). It would also be difficult for users to discover potential mistakes in the output (Rudin, 2019). Further, when the inte
-
"With the wide application of generative AI, the ability to interact with AI efficiently and effectively has become one of the most important media literacies. Hence, it is imperative for generative AI users to learn and apply the principles of prompt engineering, which refers to a systematic process of carefully designing prompts or inputs to generative AI models to elicit valuable outputs. Due to the ambiguity of human languages, the interaction between humans and machines through prompts may lead to errors or misunderstandings. Hence, the quality of prompts is important. Another challenge i
-
38.03.00 · Risk Category
"A recurring complaint among participants was a lack of knowledge about how AI systems made judgements. They emphasized the significance of making AI systems more visible and explainable so that people may have confidence in their outputs and hold them accountable for their activities. Because AI systems are typically opaque, making it difficult for users to understand the rationale behind their judgements, ethical concerns about AI, as well as issues of transparency and explainability, arise. This lack of understanding can generate suspicion and reluctance to adopt AI technology, as well as m
-
38.05.00 · Risk Category
"The participants of the study emphasized the importance of trustworthiness and reliability in AI systems. The authors emphasized the importance of preserving precision and objectivity in the outcomes produced by AI systems, while also ensuring transparency in their decision-making procedures. The significance of reliability and credibility in AI systems is escalating in tandem with the proliferation of these technologies across diverse domains of society. This underscores the importance of ensuring user confidence. The concern regarding the dependability of AI systems and their inherent biase
-
39.20.00 · Risk Category
an external entity of an AI-based ecosystem may want to know which parts of data affect the final decision in a learning model
-
39.21.00 · Risk Category
How a learning model can be reproduced when it is obtained based on various sets of data and a large space of parameters. This problem becomes more challenging in data-driven learning procedures without transparent instructions
-
39.25.00 · Risk Category
In many applications of AI-based systems such as medical healthcare and military services, the lack of verification of code may not be tolerable... due to some characteristics such as the non-linear and complex structure of AI-based solutions, existing solutions have been generally considered “black boxes”, not providing any information about what exactly makes them appear in their predictions and decision-making processes.
-
"Stems from the mismatch between mathematical optimization in high-dimensionality characteristic of machine learning and the demands of human-scale reasoning and styles of semantic interpretation."
-
42.21.00 · Risk Category
"Any action or procedure performed by a model with the intention of clarifying or detailing its internal functions."
-
"The complexity and opacity of AI models and systems make it difficult to predict and manage their behavior."
-
62.18.05 · Risk Sub-Category
Model Evaluations (Interpretability/Explainability)
Model outputs inconsistent with chain-of-thought reasoning
"Chain-of-thought reasoning is sometimes employed to get a better understanding of the model’s output, where it encourages transparent reasoning in text form. However, in some cases, this reasoning is not consistent with the final answer given by the AI model, and as such does not give sufficient transparency [113]."
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.