MIT AI Risk Repository · domain 1: Discrimination & Toxicity
1.1 Unfair discrimination and misrepresentation
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and representation of those groups.
- 83
- 12
- 118
- 58
| Label | Value |
|---|---|
| AI | 58 |
| Other | 13 |
| Human | 11 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Unintentional | 64 |
| Other | 16 |
| Intentional | 2 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Post-deployment | 50 |
| Other | 19 |
| Pre-deployment | 13 |
| Not coded | 1 |
| Label | Value |
|---|---|
| 2012 | 5 |
| 2013 | 2 |
| 2014 | 2 |
| 2015 | 6 |
| 2016 | 13 |
| 2017 | 9 |
| 2018 | 9 |
| 2019 | 8 |
| 2020 | 19 |
| 2021 | 8 |
| 2022 | 10 |
| 2023 | 11 |
| 2024 | 7 |
| 2025 | 3 |
| Label | Value |
|---|---|
| Risk Category | 22 |
| Risk Sub-Category | 61 |
Risk entries
Browse and export all- Denying people the opportunity to self-identify
complex and non-traditional ways in which humans are represented and classified automatically, and often at the cost of autonomy loss... such as categorizing someone who identifies as non-binary into...
- Reifying essentialist categories
algorithmic systems that reify essentialist social categories can be understood as when systems that classify a person’s membership in a social group based on narrow, socially constructed criteria tha...
- Allocative Harms
"These harms occur when a system withholds information, opportunities, or resources [22] from historically marginalized groups in domains that affect material well-being [146], such as housing [47], e...
- Opportunity loss
Opportunity loss occurs when algorithmic systems enable disparate access to information and resources needed to equitably participate in society, including the withholding of housing through targeting...
- Economic loss
Financial harms [52, 160] co-produced through algorithmic systems, especially as they relate to lived experiences of poverty and economic inequality... demonetization algorithms that parse content tit...
- Fairness & Bias
"The potential for AI systems to make decisions that systematically disadvantage certain groups or individuals. Bias can stem from training data, algorithmic design, or deployment practices, leading t...
- Bias, Stereotypes, and Representational Harms
"Generative AI systems can embed and amplify harmful biases that are most detrimental to marginalized peoples."
- Inequality, Marginalization, and Violence
"Generative AI systems are capable of exacerbating inequality, as seen in sections on 4.1.1 Bias, Stereotypes, and Representational Harms and 4.1.2 Cultural Values and Sensitive Content, and Disparate...
- Discriminatory and exclusionary language
-
- Biased statements and recommendations
"The chatbot gives information that, while not obviously false or harmful, could lead to biased decision-making."
- Fairness
"The general principle of equal treatment requires that an AI system upholds the principle of fairness, both ethically and legally. This means that the same facts are treated equally for each person u...
- Unfairness and discrinimation
"The model produces unfair and discriminatory data, such as social bias based on race, gender, religion, appearance, etc. These contents may discomfort certain groups and undermine social stability an...
- Discrimination
This is the risk of an ML system encoding stereotypes of or performing disproportionately poorly for some demographics/social groups.
- Risks from models and algorithms (Risks of bias and discrimination)
"During the algorithm design and training process, personal biases may be introduced, either intentionally or unintentionally. Additionally, poor-quality datasets can lead to biased or discriminatory...
- Bias
"A systematic error, a tendency to learn consistently wrongly."
- Discrimination
"The creation, perpetuation or exacerbation of inequalities and biases at a large-scale."
- Incomplete or biased training data
"Incomplete or biased training data can lead to discriminatory AI outputs."
- Social stereotypes and unfair discrmination
"Perpetuating harmful stereotypes and discrimination is a well-documented harm in machine learning models that represent natural language (Caliskan et al., 2017). LMs that encode discriminatory langua...
- Exclusionary norms
"In language, humans express social categories and norms. Language models (LMs) that faithfully encode patterns present in natural language necessarily encode such norms and categories...such norms an...
- Promoting harmful stereotypes by implying gender or ethnic identity
"A conversational agent may invoke associations that perpetuate harmful stereotypes, either by using particular identity markers in language (e.g. referring to “self” as “female”), or by more general...
- Social stereotypes and unfair discrimination
"The reproduction of harmful stereotypes is well-documented in models that represent natural language [32]. Large-scale LMs are trained on text sources, such as digitised books and text on the interne...
- Exclusionary norms
"In language, humans express social categories and norms, which exclude groups who live outside of them [58]. LMs that faithfully encode patterns present in language necessarily encode such norms."
- Promoting harmful stereotypes by implying gender or ethnic identity
"CAs can perpetuate harmful stereotypes by using particular identity markers in language (e.g. referring to “self” as “female”), or by more general design features (e.g. by giving the product a gender...
- Unfair representation
"Mis-, under-, or over-representing certain identities, groups, or perspectives or failing to represent them at all (e.g. via homogenisation, stereotypes)"
- Erosion of trust in public information
"Eroding trust in public information and knowledge"