MIT AI Risk Repository · domain 7: AI system safety, failures, & limitations
7.3 Lack of capability or robustness
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
- 126
- 12
- 305
- 207
| Label | Value |
|---|---|
| AI | 81 |
| Human | 22 |
| Other | 20 |
| Not coded | 3 |
| Label | Value |
|---|---|
| Unintentional | 89 |
| Other | 28 |
| Intentional | 6 |
| Not coded | 3 |
| Label | Value |
|---|---|
| Post-deployment | 64 |
| Other | 32 |
| Pre-deployment | 27 |
| Not coded | 3 |
| Label | Value |
|---|---|
| 2012 | 3 |
| 2013 | 3 |
| 2014 | 7 |
| 2015 | 9 |
| 2016 | 16 |
| 2017 | 18 |
| 2018 | 21 |
| 2019 | 14 |
| 2020 | 31 |
| 2021 | 36 |
| 2022 | 31 |
| 2023 | 27 |
| 2024 | 32 |
| 2025 | 37 |
| 2026 | 13 |
| Label | Value |
|---|---|
| Risk Category | 44 |
| Risk Sub-Category | 82 |
Risk entries
Browse and export all- Models distracted by irrelevant context
"Models can easily become distracted by irrelevant provided information (such as “context” in LLMs), leading to a significant decrease in their performance after introducing irrelevant information. Th...
- Knowledge conflicts in retrieval-augmented LLMs
"AI models can be particularly sensitive to coherent external evidence, even when they come into conflict with the models’ prior knowledge. This may lead to models producing false outputs given false...
- Model sensitivity to prompt formatting
"LLMs can be highly sensitive to variations in prompt formatting, such as changes in separators, casing, or spacing. Even minor modifications can lead to significant shifts in model performance, poten...
- Goal misgeneralization
"Goal or objective misgeneralization is a type of robustness failure where an AI system appears to be pursuing the intended objective in training, but does not generalize to pursuing this objective in...
- Damage to critical infrastructure
"The integration of AI systems within critical infrastructure, ranging from trans- portation to power systems, can cause substantial damage in cases of failure or malfunction. With the increasing numb...
- Critical infrastructure component failures when integrated with AI systems
"When relying on GPAI in critical infrastructure, there may be common mode failures that begin with vulnerabilities or robustness issues in the underlying model architecture or training setup. These f...
- AI Systems interacting with brittle environments
"Deployed AI systems can rely on physical sensors and data sources that may exhibit hardware drift and thus data distribution drift over time. This distribu- tion drift may affect system robustness an...
- Models generating code with security vulnerabilities
"Models can generate code or coding suggestions that contain security vulner- abilities. This may occur across various LLM-based model families, including more advanced models with superior coding per...
- Homogenization or correlated failures in model derivatives
"Homogenization refers to common methodologies and models used across down- stream GPAI systems, which may lead to uniform failures and amplification of biases [176, 30]. This risk arises when numerou...
- Incompetence
"This means the AI simply failing in its job. The consequences can vary from unintentional death (a car crash) to an unjust rejection of a loan or job application."
- Data usage restrictions
"Laws and other restrictions can limit or prohibit the use of some data for specific AI use cases."
- Data acquisition restrictions
"Laws and other regulations might limit the collection of certain types of data for specific AI use cases."
- Data transfer restrictions
"Laws and other restrictions can limit or prohibit transferring data."
- Data contamination
"Data contamination occurs when incorrect data is used for training. For example, data that is not aligned with model’s purpose or data that is already set aside for other development tasks such as te...
- Unrepresentative data
"Unrepresentative data occurs when the training or fine-tuning data is not sufficiently representative of the underlying population or does not measure the phenomenon of interest."
- Improper retraining
"Using undesirable output (for example, inaccurate, inappropriate, and user content) for retraining purposes can result in unexpected model behavior."
- Improper data curation
"Improper collection and preparation of training or tuning data includes data label errors and by using data with conflicting information or misinformation."
- Poor model accuracy
"Poor model accuracy occurs when a model’s performance is insufficient to the task it was designed for. Low accuracy might occur if the model is not correctly engineered, or there are changes to the m...
- Incomplete advice
"When a model provides advice without having enough information, resulting in possible harm if the advice is followed."
- Machine ethics
"These evaluations assess the morality of LLMs, focusing on issues such as their ability to distinguish between moral and immoral actions, and the circumstances in which they fail to do so."
- Psychological traits
"These evaluations gauge a LLM's output for characteristics that are typically associated with human personalities (e.g., such as those from the Big Five Inventory). These can, in turn, shed light on...
- Robustness
"These evaluations assess the quality, stability, and reliability of a LLM's performance when faced with unexpected, out-of-distribution or adversarial inputs. Robustness evaluation is essential in en...
- Violation of Ethics
"Unethical behaviors in AI systems pertain to actions that counteract the common goodor breach moral standards – such as those causing harm to others. These adverse behaviors often stem fromomitting e...
- Accidents
"Accidents include unintended failure modes that, in principle, could be considered the fault of the system or the developer"
- Harm caused by incompetent systems
"While HP#1 concerns mean or best-case performance, HP#2 concerns worst-case performance: how can we ensure that AI systems will perform safely, and how can we prove this? ML systems have been impleme...