MIT AI Risk Repository · domain 7: AI system safety, failures, & limitations
7.3 Lack of capability or robustness
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
- 126
- 12
- 305
- 207
| Label | Value |
|---|---|
| AI | 81 |
| Human | 22 |
| Other | 20 |
| Not coded | 3 |
| Label | Value |
|---|---|
| Unintentional | 89 |
| Other | 28 |
| Intentional | 6 |
| Not coded | 3 |
| Label | Value |
|---|---|
| Post-deployment | 64 |
| Other | 32 |
| Pre-deployment | 27 |
| Not coded | 3 |
| Label | Value |
|---|---|
| 2012 | 3 |
| 2013 | 3 |
| 2014 | 7 |
| 2015 | 9 |
| 2016 | 16 |
| 2017 | 18 |
| 2018 | 21 |
| 2019 | 14 |
| 2020 | 31 |
| 2021 | 36 |
| 2022 | 31 |
| 2023 | 27 |
| 2024 | 32 |
| 2025 | 37 |
| 2026 | 13 |
| Label | Value |
|---|---|
| Risk Category | 44 |
| Risk Sub-Category | 82 |
Risk entries
Browse and export all- Productivity loss
"End user's loss of productivity due to the underperfomance of a genAI application, including producing nonsensical or poor quality outputs, degrading its utility."
- Inconsistency
models could fail to provide the same and consistent answers to different users, to the same user but in different sessions, and even in chats within the sessions of the same conversation
- Limited Logical Reasoning
LLMs can provide seemingly sensible but ultimately incorrect or invalid justifications when answering questions
- Limited Causal Reasoning
Causal reasoning makes inferences about the relationships between events or states of the world, mostly by identifying cause-effect relationships
- Unawareness of Emotions
when a certain vulnerable group of users asks for supporting information, the answers should be informative but at the same time sympathetic and sensitive to users’ reactions
- Robustness
Resilience against adversarial attacks and distribution shift
- Accidents
"As general purpose AI models as “black-box” models are not fully controllable and understandable, even to their developers, unexpected failures could arise from their unreliability. This could lead t...
- AGIs with poor ethics, morals and values
"The risks associated with an AGI without human morals and ethics, with the wrong morals, without the capability of moral reasoning, judgement"
- Unethical decision making
"If, for example, an agent was programmed to operate war machinery in the service of its country, it would need to make ethical decisions regarding the termination of human life. This capacity to make...
- Safety
"Are AI safe with respect to human life and property? Will their use create unintended or intended safety issues?"
- Law abiding
"We find literature that proposes [38] that early artificial intelligence should be built to be safe and lawabiding, and that later artificial intelligence (that which surpasses our own intelligence)...
- Human-like immoral decisions
"If we design our machines to match human levels of ethical decision-making, such machines would then proceed to take some immoral actions (since we humans have had occasion to take immoral actions ou...
- Technology concerns
"Challenges related to technology refer to the limitations or constraints associated with generative AI. For example, the quality of training data is a major challenge for the development of generativ...
- Quality of training data
"The quality of training data is another challenge faced by generative AI. The quality of generative AI models largely depends on the quality of the training data (Dwivedi et al., 2023; Su & Yang, 202...
- Accidental harm
"Automation in sectors ranging from manufacturing to healthcare has and will increasingly put humans into close contact with EAI systems [7]. This interaction increases the risk of accidental physical...
- Robustness and Reliability
The robustness of an AI-based model refers to the stability of the model performance after abnormal changes in the input data... The cause of this change may be a malicious attacker, environmental noi...
- Predictability
whether the decision of an AI-based agent can be predicted in every situation or not
- Complexity
Nowadays, we are faced with systems that utilize numerous learning models in their modules for their perception and decision-making processes... One aspect of an AI-based system that leads to increasi...
- Inadequate specification of ODD
"The operational design domain (ODD) is a technical description of the application’s operational environment, initially conceptualized for autonomous driving systems. An inadequate specification of th...
- Inadequate planning of performance requirements
"The expected performance of the AI system should be planned adequately. Hereby, an important aspect is that chosen performance metrics are meaningful for presenting the intended functionality. Otherw...
- Incorrect data labels
"Data labels are essential for any supervised learning algorithm since they preset the result of the learning process. If the correctness of the data labels is not given, the AI system is prevented fr...
- Insufficient data representation
"The distribution of the data used for training a model should match the operational data ́s distribution while consisting of sufficiently many samples. An important aspect of matching distributions b...
- Problems of synthetic data
"In the case of sparse data quantity, the simulation or generation of data is a valid alternative. However, it is essential to make sure that the simulated data is sufficiently similar to real data, e...
- Poor model design choices
"The model specifications have significant impact on the functionality of an AI system. The developer mak- ing wrong decisions might cause the AI system to behave biased and unreliable."
- Over- and underfitting
"Over- and underfitting describe the over or insufficient adaption of a model to training data. Both phenomena can cause an AI system to behave unreliably if confronted with operational data."