MIT AI Risk Repository · Risk Sub-Category · 47.02.09
Bias and discrimination (value embedding)
Category: Ethical and social risks
On this page 5 sections
Description
"Generative AI models may also be subject to the “value embedding” phenomenon.361 “Value embedding” refers to the fact that developers of generative AI models strive to minimize biased outputs by retraining their models based on normative values.362 Contemporary state-of- the-art models not only reflect the values embedded within their training data, they also undergo additional fine-tuning that follows a set of chosen rules and principles. Due to the absence of universally accepted standards, developers bear the responsibility of making decisions on sensitive issues. These practices lead to c
From Regulating under Uncertainty: Governance Options for Generative AI (G'sell2024), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
- Causal entity
- Human
- Intent
- Unintentional
- Timing
- Pre-deployment
Subdomain definition: Accuracy and effectiveness of AI decisions and actions is dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Real-world incidents in this subdomain
- Washington State DOL's AI Phone System Reportedly Failed to Provide Spanish-Language Service to Callers Requesting Spanish
- UK Facial Recognition System Reportedly Exhibits Higher False Positive Rates for Black and Asian Subjects
- Infinite Campus AI-Driven Student Risk Model Leads to Cuts in Support for Nevada's Low-Income Schools
- Police Use of Facial Recognition Software Causes Wrongful Arrests Without Defendant Knowledge
- Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud
- Facewatch Reported to Have Wrongfully Flagged Home Bargains Customer as Shoplifter
How other frameworks describe this risk
Other entries from G'sell2024
- Technical and operational risks
- Technical vulnerabilities (Robustness - unexpected behaviour)
- Technical vulnerabilities (Robustness - unexpected behaviour)
- Technical vulnerabilities (Robustness - vulnerability to jailbreaking
- Technical vulnerabilities (Robustness - vulnerability to jailbreaking
- Technical vulnerabilities (The risk of misalignment)
- Technical vulnerabilities (The risk of misalignment)
- Factually incorrect content (inaccuracies and fabricated sources)
- Factually incorrect content (inaccuracies and fabricated sources)
- Opacity (the black box problem)
- Opacity (industry opacity)
- Opacity (industry opacity)