MIT AI Risk Repository
Browse AI risks
554 risk entries extracted from 74 frameworks, coded by domain, subdomain, causal entity, intent and timing. Filter, then export the current selection with its licence and citation attached.
-
56.10.00 · Risk Category
Poor performance of a model used for its intended purpose, for example leading to biased decisions
—
-
56.11.00 · Risk Category
—
-
59.01.00 · Risk Category
"The operational design domain (ODD) is a technical description of the application’s operational environment, initially conceptualized for autonomous driving systems. An inadequate specification of the ODD limits essential functions such as testing the learned functionality and out-of-distribution detection."
-
59.03.00 · Risk Category
"The expected performance of the AI system should be planned adequately. Hereby, an important aspect is that chosen performance metrics are meaningful for presenting the intended functionality. Otherwise, expectations and safety requirements can be unfulfillable at later life cycle stages."
-
59.11.00 · Risk Category
"Data labels are essential for any supervised learning algorithm since they preset the result of the learning process. If the correctness of the data labels is not given, the AI system is prevented from learning the ground truth and therefore the intended functionality."
-
59.16.00 · Risk Category
"The model specifications have significant impact on the functionality of an AI system. The developer mak- ing wrong decisions might cause the AI system to behave biased and unreliable."
-
59.19.00 · Risk Category
"AI systems tend to show unreliable behavior when confronted with rare or ambiguous input data, also called corner cases. Therefore, the controlled behavior is required whenever the AI system is faces a corner case."
-
59.20.00 · Risk Category
"Robustness characterizes the resilience of an AI system’s output against minor changes in the input domain. A great variation in an AI system’s response to small input changes indicates unreliable outputs."
-
59.22.00 · Risk Category
"Until the deployment of the AI application into its operational environment, the AI system has been tested with a test set that aims to approximate the distribution of operational data. However, an unexpected deviation in this approximation can cause an AI application to behave unreliably. Therefore, its behavior under confrontation with operational data needs to be evaluated."
-
"Technical AI hazards are the root causes of technical deficiencies in the AI system. An example of such an AI hazard is overfitting, which describes a model’s excessive adaptation to the training dataset. Quantitative methods to assess (metrics) and treat (mitigation means) exist for technical AI hazards, which might be performed automatically. In case of overfitting, metrics are based on the comparison of performance between the training and validation datasets, and mitigation means may include regularization techniques, among others."
-
"The third class encompasses procedural AI hazards. These pertain to issues arising from processes and actions made by individuals involved in the develop- ment process. Such hazards are not readily quantifiable and necessitate alter- native mitigation strategies. An example of such an AI hazard would be ”poor model design choices,” which could be expressed, for instance, through a devel- oper’s decision to select an unsuitable AI model for a given problem. Due to the challenges in quantifying and mitigating these issues, qualitative approaches must be employed. In the case of the aforemention
-
"Relying on general-purpose AI products that fail to fulfil their intended function can lead to harm. For example, general- purpose AI systems can make up facts (‘hallucination’), generate erroneous computer code, or provide inaccurate medical information. This can lead to physical and psychological harms to consumers and reputational, financial and legal harms to individuals and organisations."
-
61.02.31 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Lack of ability to generate accurate information
"AI models may generate false or misleading information due to their lack of capability in discerning truth."
-
61.02.32 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Lack of ethical decision-making
"AI models and systems that lack moral reasoning capabilities may make decisions that are unethical or harmful."
-
61.02.46 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Unclear attribution from AI component interactions
"Interactions between different AI components can cause harm, but it may be difficult to pinpoint which components are the cause."
-
62.14.02 · Risk Sub-Category
Data-related (Lack of cross-organizational documentation)
"When sharing data between multiple organizations, documentation may be missing or inadequate, making it difficult for other organizations to understand it. For example, a lack of metadata or a change in schema by a collaborating party can result in an unusable dataset and wasted data collection efforts, or it can lead to misunderstandings about the dataset’s limitations, resulting in downstream risks related to its use [173]."
-
62.14.03 · Risk Sub-Category
Data-related (Manipulation of data by non-domain experts)
"Manipulating data (e.g., training data) carries a set of assumptions on how the data should appear and be used by those performing the manipulation. Common manipulations applied on data in the context of AI models include defining the ground truth label and merging different data formats or sources. People who have little or no expertise in the domain of the data performing such manipulations may render the data unusable or harmful to the development of the AI system [173]."
-
62.15.00 · Risk Sub-Category
Training-related (Robust overfitting in adversarial training)
"Adversarial training can be affected by robust overfitting, where the model’s robustness on test data decreases during further training, particularly after the learning rate decay. This issue has been consistently observed across various datasets and algorithms in adversarial training settings [163, 230]. Robust over- fitting can affect the model’s ability to generalize effectively and reduce its resilience to adversarial attacks."
-
"Models can be affected by poor confidence calibration [85], where the predicted probabilities do not accurately reflect the true likelihood of ground truth cor- rectness. This miscalibration makes it difficult to interpret the model’s predic- tions reliably, as high accuracy does not guarantee that the confidence levels are meaningful. This can cause overconfidence in incorrect predictions or un- derconfidence in correct ones."
-
62.15.10 · Risk Sub-Category
Fine-tuning related (Catastrophic forgetting due to continual instruction fine-tuning)
"Catastrophic forgetting occurs when a model loses its ability to retain previously learned tasks (or factual information) after being trained on new ones. In language models, this can occur due to continual instruction tuning. This tendency may become more pronounced as the model’s size increases [127]."
-
62.16.07 · Risk Sub-Category
General Evaluations (AI outputs for which evaluation is too difficult for humans)
"When AI models are trained through evaluation with human feedback, such as reinforcement learning from human feedback, their outputs can be challenging to assess, as they may contain hard-to-detect errors or issues that only become apparent over time. The human evaluator can rate incorrect outputs positively or similar to correct outputs. This can lead to the model learning to produce subtly incorrect or harmful outputs, such as code with software vulnerabilities, or politically biased information. In extreme cases where a model is deceiving users, complicated outputs can contain hidden error
-
62.19.08 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Models distracted by irrelevant context
"Models can easily become distracted by irrelevant provided information (such as “context” in LLMs), leading to a significant decrease in their performance after introducing irrelevant information. This can happen with different prompting techniques, including chain-of-thought prompting [184]."
-
62.19.09 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Knowledge conflicts in retrieval-augmented LLMs
"AI models can be particularly sensitive to coherent external evidence, even when they come into conflict with the models’ prior knowledge. This may lead to models producing false outputs given false information during the retrieval- augmentation process, despite only a relatively small amount of false informa- tion input that is inconsistent with the model’s prior knowledge trained on much larger amounts of data [220]."
-
"The integration of AI systems within critical infrastructure, ranging from trans- portation to power systems, can cause substantial damage in cases of failure or malfunction. With the increasing number of Internet of Things (IoT) devices and interconnected cyber-physical systems, critical infrastructure becomes even more vulnerable [171, 174]."
-
62.30.04 · Risk Sub-Category
AI Systems interacting with brittle environments
"Deployed AI systems can rely on physical sensors and data sources that may exhibit hardware drift and thus data distribution drift over time. This distribu- tion drift may affect system robustness and performance. This usually involves AI systems working in undigitized and physical environments."
-
62.32.04 · Risk Sub-Category
Models generating code with security vulnerabilities
"Models can generate code or coding suggestions that contain security vulner- abilities. This may occur across various LLM-based model families, including more advanced models with superior coding performance, where the tendency to produce insecure code is even more pronounced [26]."
-
"Laws and other restrictions can limit or prohibit the use of some data for specific AI use cases."
-
"Laws and other regulations might limit the collection of certain types of data for specific AI use cases."
-
"Laws and other restrictions can limit or prohibit transferring data."
-
"Data contamination occurs when incorrect data is used for training. For example, data that is not aligned with model’s purpose or data that is already set aside for other development tasks such as testing and evaluation."
-
"Unrepresentative data occurs when the training or fine-tuning data is not sufficiently representative of the underlying population or does not measure the phenomenon of interest."
-
"Using undesirable output (for example, inaccurate, inappropriate, and user content) for retraining purposes can result in unexpected model behavior."
-
"Improper collection and preparation of training or tuning data includes data label errors and by using data with conflicting information or misinformation."
-
"Poor model accuracy occurs when a model’s performance is insufficient to the task it was designed for. Low accuracy might occur if the model is not correctly engineered, or there are changes to the model’s expected inputs."
-
"When a model provides advice without having enough information, resulting in possible harm if the advice is followed."
-
"End user's loss of productivity due to the underperfomance of a genAI application, including producing nonsensical or poor quality outputs, degrading its utility."
-
69.02.00 · Risk Category
"The chatbot makes a deal, commitment, or other consequential action with its output that the deployer did not intend."
-
69.04.00 · Risk Category
"The chatbot gives guidance that ranges from simply unhelpful to harmful if acted on."
-
—
-
69.04.03 · Risk Sub-Category
Bad advice/failure to generate helpful content
Bad links and references
—
-
—
-
"Automation in sectors ranging from manufacturing to healthcare has and will increasingly put humans into close contact with EAI systems [7]. This interaction increases the risk of accidental physical harm. Though accidental harm has been a longstanding issue in industrial robotics, increased AI capabilities could exacerbate this risk; several recent reports document an increase in industrial injuries following the introduction of AI-controlled robots [66–68]."
-
"Physical (mechanical) risks are associated with robotics and automated systems, which could lead to equipment malfunctions or physical harm in laboratory settings."
-
72.03.00 · Risk Category
"Risks arising from operational failures, model misjudgments, or improper human operation of AI systems deployed in safety-critical infrastructure, where single points of failure can trigger cascading catastrophic consequences."
-
"General-purpose AI deployed for reactor monitoring, control system optimization, or emergency response coordination could misinterpret sensor data, fail to recognize critical safety conditions, or make erroneous control decisions during emergency scenarios. Given the catastrophic potential of nuclear accidents, even minor AI reasoning errors in safety-critical functions could lead to core meltdowns, radiation releases, or widespread contamination affecting hundreds of thousands of people across international borders."
-
"General-purpose AI deployed in power grid management, water treatment facilities, telecommunications networks, or transportation coordination systems could misinterpret operational data, fail to anticipate cascading failure modes, or make control decisions that destabilize interconnected infrastructure networks. Infrastructure failures could result in widespread blackouts, contaminated water supplies, communications breakdowns, and the collapse of essential services supporting hundreds of thousands of people."
-
06.06.00 · Risk Category
"The idea of a "black box" making decisions without any explanation, without offering insight in the process, has a couple of disadvantages: it may fail to gain the trust of its users and it may fail to meet regulatory standards such as the ability to audit."
-
10.05.00 · Risk Category
"In situations in which the development and use of AI are not explained to the user, or in which the decision processes do not provide the criteria or steps that constitute the decision, the use of AI becomes inexplicable."
-
14.05.00 · Risk Category
"Transparency is the characteristic of a system that describes the degree to which appropriate information about the system is communicated to relevant stakeholders, whereas explainability describes the property of an AI system to express important factors influencing the results of the AI system in a way that is understandable for humans....Information about the model underlying the decision-making process is relevant for transparency. Systems with a low degree of transparency can pose risks in terms of their fairness, security and accountability. "
-
30.05.00 · Risk Category
The ability to explain the outputs to users and reason correctly
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.