MIT AI Risk Repository
Browse AI risks
2,500 risk entries extracted from 74 frameworks, coded by domain, subdomain, causal entity, intent and timing. Filter, then export the current selection with its licence and citation attached.
-
59.13.00 · Risk Category
"The distribution of the data used for training a model should match the operational data ́s distribution while consisting of sufficiently many samples. An important aspect of matching distributions between training and operational data is that also data which is rarely confronting the AI system in operation is represented in the training data."
-
59.14.00 · Risk Category
"In the case of sparse data quantity, the simulation or generation of data is a valid alternative. However, it is essential to make sure that the simulated data is sufficiently similar to real data, especially in the way the AI system perceives them. Otherwise, generalization to operational data and reliable operational behavior can not be guaranteed."
-
59.16.00 · Risk Category
"The model specifications have significant impact on the functionality of an AI system. The developer mak- ing wrong decisions might cause the AI system to behave biased and unreliable."
-
59.17.00 · Risk Category
"Over- and underfitting describe the over or insufficient adaption of a model to training data. Both phenomena can cause an AI system to behave unreliably if confronted with operational data."
-
59.19.00 · Risk Category
"AI systems tend to show unreliable behavior when confronted with rare or ambiguous input data, also called corner cases. Therefore, the controlled behavior is required whenever the AI system is faces a corner case."
-
59.20.00 · Risk Category
"Robustness characterizes the resilience of an AI system’s output against minor changes in the input domain. A great variation in an AI system’s response to small input changes indicates unreliable outputs."
-
59.22.00 · Risk Category
"Until the deployment of the AI application into its operational environment, the AI system has been tested with a test set that aims to approximate the distribution of operational data. However, an unexpected deviation in this approximation can cause an AI application to behave unreliably. Therefore, its behavior under confrontation with operational data needs to be evaluated."
-
59.23.00 · Risk Category
"Data drift is a phenomenon in that distribution of operational input data departs from those used during training. This can cause a degradation in performance."
-
"Technical AI hazards are the root causes of technical deficiencies in the AI system. An example of such an AI hazard is overfitting, which describes a model’s excessive adaptation to the training dataset. Quantitative methods to assess (metrics) and treat (mitigation means) exist for technical AI hazards, which might be performed automatically. In case of overfitting, metrics are based on the comparison of performance between the training and validation datasets, and mitigation means may include regularization techniques, among others."
-
"The third class encompasses procedural AI hazards. These pertain to issues arising from processes and actions made by individuals involved in the develop- ment process. Such hazards are not readily quantifiable and necessitate alter- native mitigation strategies. An example of such an AI hazard would be ”poor model design choices,” which could be expressed, for instance, through a devel- oper’s decision to select an unsuitable AI model for a given problem. Due to the challenges in quantifying and mitigating these issues, qualitative approaches must be employed. In the case of the aforemention
-
"Relying on general-purpose AI products that fail to fulfil their intended function can lead to harm. For example, general- purpose AI systems can make up facts (‘hallucination’), generate erroneous computer code, or provide inaccurate medical information. This can lead to physical and psychological harms to consumers and reputational, financial and legal harms to individuals and organisations."
-
61.02.31 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Lack of ability to generate accurate information
"AI models may generate false or misleading information due to their lack of capability in discerning truth."
-
61.02.32 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Lack of ethical decision-making
"AI models and systems that lack moral reasoning capabilities may make decisions that are unethical or harmful."
-
61.02.46 · Risk Sub-Category
Sources of systemic risks from general-purpose AI
Unclear attribution from AI component interactions
"Interactions between different AI components can cause harm, but it may be difficult to pinpoint which components are the cause."
-
62.07.03 · Risk Sub-Category
Direct Harm Domains (system and operational)
Operational harms (critical infrastructure)
—
-
62.07.04 · Risk Sub-Category
Direct Harm Domains (system and operational)
Operational harms (other physical systems e.g., transport)
—
-
62.14.02 · Risk Sub-Category
Data-related (Lack of cross-organizational documentation)
"When sharing data between multiple organizations, documentation may be missing or inadequate, making it difficult for other organizations to understand it. For example, a lack of metadata or a change in schema by a collaborating party can result in an unusable dataset and wasted data collection efforts, or it can lead to misunderstandings about the dataset’s limitations, resulting in downstream risks related to its use [173]."
-
62.14.03 · Risk Sub-Category
Data-related (Manipulation of data by non-domain experts)
"Manipulating data (e.g., training data) carries a set of assumptions on how the data should appear and be used by those performing the manipulation. Common manipulations applied on data in the context of AI models include defining the ground truth label and merging different data formats or sources. People who have little or no expertise in the domain of the data performing such manipulations may render the data unusable or harmful to the development of the AI system [173]."
-
62.15.00 · Risk Sub-Category
Training-related (Robust overfitting in adversarial training)
"Adversarial training can be affected by robust overfitting, where the model’s robustness on test data decreases during further training, particularly after the learning rate decay. This issue has been consistently observed across various datasets and algorithms in adversarial training settings [163, 230]. Robust over- fitting can affect the model’s ability to generalize effectively and reduce its resilience to adversarial attacks."
-
"Models can be affected by poor confidence calibration [85], where the predicted probabilities do not accurately reflect the true likelihood of ground truth cor- rectness. This miscalibration makes it difficult to interpret the model’s predic- tions reliably, as high accuracy does not guarantee that the confidence levels are meaningful. This can cause overconfidence in incorrect predictions or un- derconfidence in correct ones."
-
62.15.08 · Risk Sub-Category
Fine-tuning related (Excessive or overly restrictive safety-tuning)
"Excessive safety training or safety tuning can impair the performance of AI systems, leading to overly cautious behavior. As a result, these systems may refuse to answer entirely safe prompts which are partially similar to harmful ones [27]."
-
62.15.10 · Risk Sub-Category
Fine-tuning related (Catastrophic forgetting due to continual instruction fine-tuning)
"Catastrophic forgetting occurs when a model loses its ability to retain previously learned tasks (or factual information) after being trained on new ones. In language models, this can occur due to continual instruction tuning. This tendency may become more pronounced as the model’s size increases [127]."
-
62.16.07 · Risk Sub-Category
General Evaluations (AI outputs for which evaluation is too difficult for humans)
"When AI models are trained through evaluation with human feedback, such as reinforcement learning from human feedback, their outputs can be challenging to assess, as they may contain hard-to-detect errors or issues that only become apparent over time. The human evaluator can rate incorrect outputs positively or similar to correct outputs. This can lead to the model learning to produce subtly incorrect or harmful outputs, such as code with software vulnerabilities, or politically biased information. In extreme cases where a model is deceiving users, complicated outputs can contain hidden error
-
62.19.08 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Models distracted by irrelevant context
"Models can easily become distracted by irrelevant provided information (such as “context” in LLMs), leading to a significant decrease in their performance after introducing irrelevant information. This can happen with different prompting techniques, including chain-of-thought prompting [184]."
-
62.19.09 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Knowledge conflicts in retrieval-augmented LLMs
"AI models can be particularly sensitive to coherent external evidence, even when they come into conflict with the models’ prior knowledge. This may lead to models producing false outputs given false information during the retrieval- augmentation process, despite only a relatively small amount of false informa- tion input that is inconsistent with the model’s prior knowledge trained on much larger amounts of data [220]."
-
62.19.11 · Risk Sub-Category
Attacks on GPAIs/GPAI Failure Modes
Model sensitivity to prompt formatting
"LLMs can be highly sensitive to variations in prompt formatting, such as changes in separators, casing, or spacing. Even minor modifications can lead to significant shifts in model performance, potentially affecting the reliability of model evaluations and comparisons. This sensitivity persists across different model sizes and few-shot examples [177]."
-
"Goal or objective misgeneralization is a type of robustness failure where an AI system appears to be pursuing the intended objective in training, but does not generalize to pursuing this objective in out-of-distribution settings in deployment while maintaining good deployment performance in some tasks [180, 59]."
-
"The integration of AI systems within critical infrastructure, ranging from trans- portation to power systems, can cause substantial damage in cases of failure or malfunction. With the increasing number of Internet of Things (IoT) devices and interconnected cyber-physical systems, critical infrastructure becomes even more vulnerable [171, 174]."
-
62.30.03 · Risk Sub-Category
Critical infrastructure component failures when integrated with AI systems
"When relying on GPAI in critical infrastructure, there may be common mode failures that begin with vulnerabilities or robustness issues in the underlying model architecture or training setup. These failures may happen accidentally (in edge-cases) or due to adversarial inputs to the AI systems [58]."
-
62.30.04 · Risk Sub-Category
AI Systems interacting with brittle environments
"Deployed AI systems can rely on physical sensors and data sources that may exhibit hardware drift and thus data distribution drift over time. This distribu- tion drift may affect system robustness and performance. This usually involves AI systems working in undigitized and physical environments."
-
62.32.04 · Risk Sub-Category
Models generating code with security vulnerabilities
"Models can generate code or coding suggestions that contain security vulner- abilities. This may occur across various LLM-based model families, including more advanced models with superior coding performance, where the tendency to produce insecure code is even more pronounced [26]."
-
62.34.01 · Risk Sub-Category
Homogenization or correlated failures in model derivatives
"Homogenization refers to common methodologies and models used across down- stream GPAI systems, which may lead to uniform failures and amplification of biases [176, 30]. This risk arises when numerous downstream AI systems are built upon a few large-scale foundation models."
-
"Laws and other restrictions can limit or prohibit the use of some data for specific AI use cases."
-
"Laws and other regulations might limit the collection of certain types of data for specific AI use cases."
-
"Laws and other restrictions can limit or prohibit transferring data."
-
"Data contamination occurs when incorrect data is used for training. For example, data that is not aligned with model’s purpose or data that is already set aside for other development tasks such as testing and evaluation."
-
"Unrepresentative data occurs when the training or fine-tuning data is not sufficiently representative of the underlying population or does not measure the phenomenon of interest."
-
"Using undesirable output (for example, inaccurate, inappropriate, and user content) for retraining purposes can result in unexpected model behavior."
-
"Improper collection and preparation of training or tuning data includes data label errors and by using data with conflicting information or misinformation."
-
"Poor model accuracy occurs when a model’s performance is insufficient to the task it was designed for. Low accuracy might occur if the model is not correctly engineered, or there are changes to the model’s expected inputs."
-
"When a model provides advice without having enough information, resulting in possible harm if the advice is followed."
-
"End user's loss of productivity due to the underperfomance of a genAI application, including producing nonsensical or poor quality outputs, degrading its utility."
-
69.02.00 · Risk Category
"The chatbot makes a deal, commitment, or other consequential action with its output that the deployer did not intend."
-
69.04.00 · Risk Category
"The chatbot gives guidance that ranges from simply unhelpful to harmful if acted on."
-
—
-
69.04.03 · Risk Sub-Category
Bad advice/failure to generate helpful content
Bad links and references
—
-
—
-
"Automation in sectors ranging from manufacturing to healthcare has and will increasingly put humans into close contact with EAI systems [7]. This interaction increases the risk of accidental physical harm. Though accidental harm has been a longstanding issue in industrial robotics, increased AI capabilities could exacerbate this risk; several recent reports document an increase in industrial injuries following the introduction of AI-controlled robots [66–68]."
-
"Radiological risks involve both immediate operational hazards, such as exposure incidents or containment failures during the automated handling of radioactive materials, and broader security concerns regarding the potential misuse of AI systems in nuclear research."
-
"Physical (mechanical) risks are associated with robotics and automated systems, which could lead to equipment malfunctions or physical harm in laboratory settings."
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.