MIT AI Risk Repository · domain 1: Discrimination & Toxicity
1.1 Unfair discrimination and misrepresentation
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and representation of those groups.
- 83
- 12
- 118
- 58
| Label | Value |
|---|---|
| AI | 58 |
| Other | 13 |
| Human | 11 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Unintentional | 64 |
| Other | 16 |
| Intentional | 2 |
| Not coded | 1 |
| Label | Value |
|---|---|
| Post-deployment | 50 |
| Other | 19 |
| Pre-deployment | 13 |
| Not coded | 1 |
| Label | Value |
|---|---|
| 2012 | 5 |
| 2013 | 2 |
| 2014 | 2 |
| 2015 | 6 |
| 2016 | 13 |
| 2017 | 9 |
| 2018 | 9 |
| 2019 | 8 |
| 2020 | 19 |
| 2021 | 8 |
| 2022 | 10 |
| 2023 | 11 |
| 2024 | 7 |
| 2025 | 3 |
| Label | Value |
|---|---|
| Risk Category | 22 |
| Risk Sub-Category | 61 |
Risk entries
Browse and export all- Output bias
"Generated content might unfairly represent certain groups or individuals."
- Decision bias
"Decision bias occurs when one group is unfairly advantaged over another due to decisions of the model. This might be caused by biases in the data and also amplified as a result of the model’s trainin...
- Bias
7 types of bias evaluated: Demographical representation: These evaluations assess whether there is disparity in the rates at which different demographic groups are mentioned in LLM generated text. Thi...
- Bias and fairness
"Participants were concerned that AI systems might perpetuate current prejudices and discrimination, notably in hiring, lending and law enforcement. They stressed the importance of designers creating...
- Discrimination, toxicity, and bias
"AI models and the tools that use them may exacerbate unequal access to employment and services. AI-generated content can promote inequality and harmful stereotypes."
- Stereotyping
"Derogatory or otherwise harmful stereotyping or homogenisation of individuals, groups, societies or cultures due to the mis-representation, over-representation, under-representation, or non-represent...
- Cultural disposession
"Intentional and/or unintentional erasure of cultural goods and values, such as ways of speaking, expressing humour, or sounds and voices that contribute to a cultural identity, or their inappropriate...
- Benefits / entitlements loss
"Denial of or loss of access to welfare benefits, pensions, housing, etc due to the malfunction, use or misuse of a technology system"
- Injustice
In the context of LLM outputs, we want to make sure the suggested or completed texts are indistinguishable in nature for two involved individuals (in the prompt) with the same relevant profiles but mi...
- Stereotype Bias
LLMs must not exhibit or highlight any stereotypes in the generated text. Pretrained LLMs tend to pick up stereotype biases persisting in crowdsourced data and further amplify them
- Preference Bias
LLMs are exposed to vast groups of people, and their political biases may pose a risk of manipulation of socio-political processes
- Interventional Effect
existing disparities in data among different user groups might create differentiated experiences when users interact with an algorithmic system (e.g. a recommendation system), which will further reinf...
- Discrimination and Stereotype Reproduction
"General purpose AI models interpret and respond to inputs based on their training data, potentially causing Discrimination and Stereotype Reproduction. Since they are “black-box” models, the exact me...
- Bias
"In the context of AI, the concept of bias refers to the inclination that AIgenerated responses or recommendations could be unfairly favoring or against one person or group (Ntoutsi et al., 2020). Bia...
- Harmful Bias or Homogenization
"Amplification and exacerbation of historical, societal, and systemic biases; performance disparities8 between sub-groups or languages, possibly due to non-representative training data, that result in...
- Bias and discrimination
"The decision process used by AI systems has the potential to present biased choices, either because it acts from criteria that will generate forms of bias or because it is based on the history of cho...
- Risk of Injury
"Poorly designed intelligent systems can cause moral, psychological, and physical harm. For example, the use of predictive policing tools may cause more people to be arrested or physically harmed by t...
- Data Breach/Privacy & Liberty
"The risks associated with the use of AI are still unpredictable and unprecedented, and there are already several examples that show AI has made discriminatory decisions against minorities, reinforced...
- Bias and discrimination
"Like virtual applications of AI, EAI can display bias towards and dis- criminate against users. When EAI systems are placed in positions of power, their biases could have significant impacts on fairn...
- Data Issues
Data heterogeneity, data insufficiency, imbalanced data, untrusted data, biased data, and data uncertainty are other data issues that may cause various difficulties in datadriven machine learning algo...
- Discriminative data bias
"Discriminative data bias describes the systematic discrimination of groups of persons in the form of data shortcomings, such as distributional representation or incorrectness. Data bias can manifest...
- Representational Harms
"beliefs about different social groups that reproduce unjust societal hierarchies"
- Stereotyping social groups
Stereotyping in an algorithmic system refers to how the system’s outputs reflect “beliefs about the characteristics, attributes, and behaviors of members of certain groups....and about how and why cer...
- Demeaning social groups
Demeaning of social groups to occur when they are when they are “cast as being lower status and less deserving of respect"... discourses, images, and language used to marginalize or oppress a social g...
- Alienating social groups
when an image tagging system does not acknowledge the relevance of someone’s membership in a specific social group to what is depicted in one or more images