MIT AI Risk Repository · Risk Sub-Category · 62.16.03
General Evaluations (Difficulty of identification and measurement of capabilities)
Category: Model Evaluations
Description
"The capabilities of general-purpose AI systems can be difficult to measure, compared to the capabilities of more limited and fixed-purpose AI systems. This is in part due to a broader distribution of potential risks, a lack of well-defined metrics to evaluate these risks, and risks from unpredictable (or emergent) AI model properties."
From Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems (Gipiškis2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
Subdomain definition: Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
Real-world incidents in this subdomain
- Uber Launched Opaque Algorithm That Changes Drivers' Payments in the US
- Israeli Tax Authority Reportedly Used an Opaque Automated System to Issue a Fine, Declining to Explain or Disclose the Underlying Calculation
- Uber Allegedly Wrongfully Accused Drivers of Fraud via Automated Systems
- Houston ISD's EVAAS Teacher-Evaluation System Reportedly Put Teachers' Jobs at Risk Through Unverifiable Scores
- Dutch City Court Defended Home Value Generated by Black-Box Algorithm