MIT AI Risk Repository · Risk Category · 59.18.00
Lack of explainability
Description
"The explainability of AI systems based on so-called black-box models is often limited. This opaqueness of AI systems can prevent developers from detecting shortcomings in the data or the model itself and decrease the performance and safety levels of the AI system."
From AI Hazard Management: A Framework for the Systematic Management of Root Causes for AI Risks (Schnitzer2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
Subdomain definition: Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
Real-world incidents in this subdomain
- Uber Launched Opaque Algorithm That Changes Drivers' Payments in the US
- Israeli Tax Authority Reportedly Used an Opaque Automated System to Issue a Fine, Declining to Explain or Disclose the Underlying Calculation
- Uber Allegedly Wrongfully Accused Drivers of Fraud via Automated Systems
- Houston ISD's EVAAS Teacher-Evaluation System Reportedly Put Teachers' Jobs at Risk Through Unverifiable Scores
- Dutch City Court Defended Home Value Generated by Black-Box Algorithm
How other frameworks describe this risk
- Intelligibility
- Attributing the responsibility for AI's failures
- Model Evaluations (Interpretability/Explainability)
- Model outputs inconsistent with chain-of-thought reasoning
- Lack of understanding of in-context learning in language models
- General Evaluations (Difficulty of identification and measurement of capabilities)
- Lack of transparency and interpretability
- Opacity (the black box problem)
Other entries from Schnitzer2024
- Inadequate specification of ODD
- Inappropriate degree of automation
- Inadequate planning of performance requirements
- Insufficient AI development documentation
- Inappropriate degree of transparency to end users
- Missing requirements for the implemented hardware
- Choice of untrustworthy data source
- Lack of data understanding
- Discriminative data bias
- Harming users’ data privacy
- Incorrect data labels
- Data poisoning