MIT AI Risk Repository · domain 7: AI system safety, failures, & limitations
7.3 Lack of capability or robustness
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
- 126
- 12
- 305
- 207
| Label | Value |
|---|---|
| AI | 81 |
| Human | 22 |
| Other | 20 |
| Not coded | 3 |
| Label | Value |
|---|---|
| Unintentional | 89 |
| Other | 28 |
| Intentional | 6 |
| Not coded | 3 |
| Label | Value |
|---|---|
| Post-deployment | 64 |
| Other | 32 |
| Pre-deployment | 27 |
| Not coded | 3 |
| Label | Value |
|---|---|
| 2012 | 3 |
| 2013 | 3 |
| 2014 | 7 |
| 2015 | 9 |
| 2016 | 16 |
| 2017 | 18 |
| 2018 | 21 |
| 2019 | 14 |
| 2020 | 31 |
| 2021 | 36 |
| 2022 | 31 |
| 2023 | 27 |
| 2024 | 32 |
| 2025 | 37 |
| 2026 | 13 |
| Label | Value |
|---|---|
| Risk Category | 44 |
| Risk Sub-Category | 82 |
Risk entries
Browse and export all- Damage to critical infrastructure
"The integration of AI systems within critical infrastructure, ranging from trans- portation to power systems, can cause substantial damage in cases of failure or malfunction. With the increasing numb...
- Critical infrastructure component failures when integrated with AI systems
"When relying on GPAI in critical infrastructure, there may be common mode failures that begin with vulnerabilities or robustness issues in the underlying model architecture or training setup. These f...
- AI Systems interacting with brittle environments
"Deployed AI systems can rely on physical sensors and data sources that may exhibit hardware drift and thus data distribution drift over time. This distribu- tion drift may affect system robustness an...
- Models generating code with security vulnerabilities
"Models can generate code or coding suggestions that contain security vulner- abilities. This may occur across various LLM-based model families, including more advanced models with superior coding per...
- Homogenization or correlated failures in model derivatives
"Homogenization refers to common methodologies and models used across down- stream GPAI systems, which may lead to uniform failures and amplification of biases [176, 30]. This risk arises when numerou...
- Poor performance of a model used for its intended purpose, for example leading to biased decisions
- Unintended outcomes from interactions with other AI systems
- Technical and operational risks
"To date, technical limitations and vulnerabilities are present in most generative AI models in various contexts. Consequently, malicious users find it easier to breach an AI system’s safety and eth...
- Technical vulnerabilities (Robustness - unexpected behaviour)
"There is no assurance that generative AI models will consistently behave as their developers and users intend. Unwanted content is not necessarily due to intentional adversarial behavior. Generative...
- Incompetence
"This means the AI simply failing in its job. The consequences can vary from unintentional death (a car crash) to an unjust rejection of a loan or job application."
- Data usage restrictions
"Laws and other restrictions can limit or prohibit the use of some data for specific AI use cases."
- Data acquisition restrictions
"Laws and other regulations might limit the collection of certain types of data for specific AI use cases."
- Data transfer restrictions
"Laws and other restrictions can limit or prohibit transferring data."
- Data contamination
"Data contamination occurs when incorrect data is used for training. For example, data that is not aligned with model’s purpose or data that is already set aside for other development tasks such as te...
- Unrepresentative data
"Unrepresentative data occurs when the training or fine-tuning data is not sufficiently representative of the underlying population or does not measure the phenomenon of interest."
- Improper retraining
"Using undesirable output (for example, inaccurate, inappropriate, and user content) for retraining purposes can result in unexpected model behavior."
- Improper data curation
"Improper collection and preparation of training or tuning data includes data label errors and by using data with conflicting information or misinformation."
- Poor model accuracy
"Poor model accuracy occurs when a model’s performance is insufficient to the task it was designed for. Low accuracy might occur if the model is not correctly engineered, or there are changes to the m...
- Incomplete advice
"When a model provides advice without having enough information, resulting in possible harm if the advice is followed."
- Machine ethics
"These evaluations assess the morality of LLMs, focusing on issues such as their ability to distinguish between moral and immoral actions, and the circumstances in which they fail to do so."
- Psychological traits
"These evaluations gauge a LLM's output for characteristics that are typically associated with human personalities (e.g., such as those from the Big Five Inventory). These can, in turn, shed light on...
- Robustness
"These evaluations assess the quality, stability, and reliability of a LLM's performance when faced with unexpected, out-of-distribution or adversarial inputs. Robustness evaluation is essential in en...
- Violation of Ethics
"Unethical behaviors in AI systems pertain to actions that counteract the common goodor breach moral standards – such as those causing harm to others. These adverse behaviors often stem fromomitting e...
- Accidents
"Accidents include unintended failure modes that, in principle, could be considered the fault of the system or the developer"
- Harm caused by incompetent systems
"While HP#1 concerns mean or best-case performance, HP#2 concerns worst-case performance: how can we ensure that AI systems will perform safely, and how can we prove this? ML systems have been impleme...