MIT AI Risk Repository · Risk Sub-Category · 17.01.01

Social stereotypes and unfair discrmination

Category: Discrimination, Exclusion and Toxicity

Open the MIT Risk Navigator All entries from Weidinger2021

Description

"Perpetuating harmful stereotypes and discrimination is a well-documented harm in machine learning models that represent natural language (Caliskan et al., 2017). LMs that encode discriminatory language or social stereotypes can cause different types of harm... Unfair discrimination manifests in differential treatment or access to resources among individuals or groups based on sensitive traits such as sex, religion, gender, sexual orientation, ability and age."

From Ethical and social risks of harm from language models (Weidinger2021), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 1. Discrimination & Toxicity
Subdomain: 1.1 Unfair discrimination and misrepresentation
Causal entity: AI
Intent: Unintentional
Timing: Other

Subdomain definition: Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and representation of those groups.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Discrimination
A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)
Harms of Representation and Other Biases
Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)
Risks from bias and underrepresentation
International Scientific Report on the Safety of Advanced AI (Bengio2024)
Bias
International AI Safety Report 2025 (Bengio2025)
Bias
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Toxicity and Bias Tendencies
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Biased Training Data
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Broken systems
Navigating the Landscape of AI Ethics and Responsibility (Cunha2023)

Other entries from Weidinger2021

Discrimination, Exclusion and Toxicity Risk Category · 1.0
Social stereotypes and unfair discrmination Additional evidence
Exclusionary norms Risk Sub-Category · 1.1
Exclusionary norms Additional evidence
Exclusionary norms Additional evidence
Exclusionary norms Additional evidence
Toxic language Risk Sub-Category · 1.2
Lower performance for some languages and social groups Risk Sub-Category · 1.3
Lower performance for some languages and social groups Additional evidence
Information Hazards Risk Category · 2.1
Compromising privacy by leaking private infiormation Risk Sub-Category · 2.1
Compromising privacy by leaking private infiormation Additional evidence