{"attribution":{"source":"AI Incident Database (Responsible AI Collaborative)","license":"CC BY-SA 4.0","license_url":"https://creativecommons.org/licenses/by-sa/4.0/","citation":"McGregor, S. (2021). Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. Proceedings of the AAAI Conference on Artificial Intelligence (IAAI-21).","snapshot_date":"2026-09-07"},"exported_at":"2026-10-28"}
{"rows":[{"incident_id":1402,"slug":"doge-reportedly-relied-on-unvetted-chatgpt-outputs-in-canceling","occurred_on":"2025-04-02","title":"DOGE Reportedly Relied on Unvetted ChatGPT Outputs in Canceling National Endowment for the Humanities Grants","description":"Department of Government Efficiency (DOGE) staff reportedly fed National Endowment for the Humanities grant descriptions into ChatGPT to determine whether projects were \"DEI,\" then allegedly used those outputs to help compile a list of grants to terminate. Beginning in April 2025, NEH canceled awards and clawed back funding, reportedly disrupting humanities organizations and projects nationwide after officials allegedly relied on a purportedly flawed, insufficiently vetted AI screening process.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Nate Cavanaugh (DOGE)","Justin Fox (DOGE)","Department of Government Efficiency (DOGE)"],"developers":["OpenAI","Large language model developers","Chatbot developers"],"harmed":["Scholars","National Endowment for the Humanities grantees","Humanities organizations","Epistemic integrity"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/doge-reportedly-relied-on-unvetted-chatgpt-outputs-in-canceling"},{"incident_id":1000,"slug":"sora-video-generator-has-reportedly-been-creating-biased-human","occurred_on":"2025-03-23","title":"Sora Video Generator Has Reportedly Been Creating Biased Human Representations Across Race, Gender, and Disability","description":"A WIRED investigation found that OpenAI's video generation model, Sora, exhibits representational bias across race, gender, body type, and disability. In tests using 250 prompts, Sora was more likely to depict CEOs and professors as men, flight attendants and childcare workers as women, and showed limited or stereotypical portrayals of disabled individuals and people with larger bodies.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["OpenAI"],"developers":["Synthetic video generation technology developers","Synthetic media generation technology developers","OpenAI"],"harmed":["Women and girls","Women","People with larger bodies","People with disabilities","People of color","Marginalized groups","LGBTQ+ people","General public"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/sora-video-generator-has-reportedly-been-creating-biased-human"},{"incident_id":885,"slug":"meta-ai-characters-allegedly-exhibited-racism-fabricated-identities","occurred_on":"2025-01-03","title":"Meta AI Characters Allegedly Exhibited Racism, Fabricated Identities, and Exploited User Trust","description":"Meta deployed AI-generated profiles on its platforms, including Instagram and Facebook, as part of an experiment. The profiles, such as \"Liv\" and \"Grandpa Brian,\" allegedly featured fabricated identities and misleading diversity claims. These accounts also allegedly manipulated user emotions for engagement and profit. Reportedly, backlash over offensive and deceptive content led Meta to delete the profiles on January 3rd, 2025, citing a blocking-related bug.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Meta"],"developers":["Meta"],"harmed":["Meta Users, Instagram Users, Facebook Users"],"report_count":5,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/meta-ai-characters-allegedly-exhibited-racism-fabricated-identities"},{"incident_id":820,"slug":"alleged-ai-generated-photo-alteration-leads-to-inappropriate","occurred_on":"2024-10-15","title":"Alleged AI-Generated Photo Alteration Leads to Inappropriate Modifications in Speaker's Conference Picture","description":"An AI image expansion tool used by a conference organizer unintentionally altered Elizabeth Laraki’s profile picture for a marketing ad, making her blouse appear unbuttoned and showing a fabricated hint of undergarments. The AI generated the lower part of the image when expanding it for vertical formatting. The conference organizers quickly apologized and removed the altered content.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Unknown Conference Employee"],"developers":["Unknown Developer"],"harmed":["Elizabeth Laraki"],"report_count":5,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/alleged-ai-generated-photo-alteration-leads-to-inappropriate"},{"incident_id":822,"slug":"algorithmic-bias-in-french-welfare-system-allegedly-discriminates","occurred_on":"2024-10-15","title":"Algorithmic Bias in French Welfare System Allegedly Discriminates Against Marginalized Groups","description":"A coalition of 15 human rights groups has launched legal action against the French government alleging that an algorithm used to detect welfare fraud discriminates against single mothers and disabled people. The algorithm assigns risk scores based on personal data. The process allegedly subjects vulnerable recipients to invasive investigations, violates privacy and anti-discrimination laws, and disproportionately affects marginalized groups.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Government of France","Caisse Nationale des Allocations Familiales (CNAF)"],"developers":["Government of France"],"harmed":["Women and girls","Women","Single parents","Single mothers in France","Single mothers","Privacy","People with disabilities in France","People with disabilities"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/algorithmic-bias-in-french-welfare-system-allegedly-discriminates"},{"incident_id":740,"slug":"department-for-work-and-pensions-dwp-ai-systems-allegedly-discriminate","occurred_on":"2024-07-10","title":"Department for Work and Pensions (DWP) AI Systems Allegedly Discriminate Against Single Mothers","description":"Researchers have argued that the Department for Work and Pensions' Universal Credit system disproportionately impacts single mothers. Automated processes in the system, designed to determine eligibility and detect fraud, are reported to have introduced biases, leading to financial instability and hardship. The algorithms allegedly miscalculate earnings and delay childcare reimbursements, in turn exacerbating income volatility and debt among single mothers.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Department For Work And Pensions"],"developers":["Department For Work And Pensions"],"harmed":["Single Mothers, British Single Mothers"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/department-for-work-and-pensions-dwp-ai-systems-allegedly-discriminate"},{"incident_id":699,"slug":"va-suicide-prevention-algorithm-reach-vet-reportedly-prioritizes-men","occurred_on":"2024-05-23","title":"VA Suicide Prevention Algorithm REACH VET Reportedly Prioritizes Men Over Women Veterans","description":"An AI program named REACH VET, designed and used by the Department of Veterans Affairs (VA) to prevent veteran suicides, was reportedly found to prioritize white men while neglecting female veterans and survivors of military sexual trauma. This oversight persists despite rising suicide rates among these groups. The incident is an example of algorithmic bias and the exclusion of critical risk factors for female veterans.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"Human","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Department of Veterans Affairs (VA)"],"developers":["Department of Veterans Affairs (VA)"],"harmed":["Women and girls","Women","Veterans","Survivors of military sexual trauma","Emotionally vulnerable individuals"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/va-suicide-prevention-algorithm-reach-vet-reportedly-prioritizes-men"},{"incident_id":686,"slug":"meta-ai-image-generator-reportedly-fails-to-accurately-represent","occurred_on":"2024-04-03","title":"Meta AI Image Generator Reportedly Fails to Accurately Represent Interracial Relationships","description":"Meta's AI image generator is alleged to produce inaccurate and biased images, consistently failing to depict interracial relationships involving Asian individuals and Caucasian or Black individuals. Instead, it generates images featuring two Asian people or stereotypes, erasing the diversity and representation of Asian people.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Meta"],"developers":["Meta"],"harmed":["Interracial Couples, General Public, Asian People"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/meta-ai-image-generator-reportedly-fails-to-accurately-represent"},{"incident_id":665,"slug":"facial-recognition-misidentification-at-new-world-westend-in-new-zealand","occurred_on":"2024-04-02","title":"Facial Recognition Misidentification at New World Westend in New Zealand","description":"A facial recognition system at New World Westend supermarket misidentified a Māori woman as a known offender during its trial. The woman was wrongfully accused of trespassing and experienced public embarrassment, raising concerns about racial bias and the technology's accuracy. The supermarket acknowledged its error and apologized.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Foodstuffs, New World Westend"],"developers":["Foodstuffs"],"harmed":["Te Ani Solomon, Maori Community"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/facial-recognition-misidentification-at-new-world-westend-in-new-zealand"},{"incident_id":633,"slug":"nine-networks-ai-alters-lawmaker-georgie-purcells-image-inappropriately","occurred_on":"2024-01-28","title":"Nine Network's AI Alters Lawmaker Georgie Purcell's Image Inappropriately","description":"The Nine Network used Photoshop's Generative Expand AI tool to resize an image of lawmaker Georgie Purcell, inadvertently altering her attire to appear more revealing. This error, claimed to result from the AI's automation, led to public criticism and an apology from the network.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Nine Network"],"developers":["Adobe"],"harmed":["Georgie Purcell"],"report_count":17,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/nine-networks-ai-alters-lawmaker-georgie-purcells-image-inappropriately"},{"incident_id":619,"slug":"rite-aid-facial-recognition-disproportionately-misidentified-minority","occurred_on":"2023-12-20","title":"Rite Aid Facial Recognition Disproportionately Misidentified Minority Shoppers as Shoplifters","description":"Rite Aid used facial recognition technology from October 2012 to July 2020, allegedly leading to disproportionate misidentifications of women, Black, Latino, and Asian shoppers as \"likely\" shoplifters. The FTC settlement prohibits Rite Aid from using this technology in stores for five years.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm issue","sectors":["wholesale and retail trade"],"countries":["US"],"deployers":["Rite Aid","Facial recognition system deployers"],"developers":["Facial recognition system developers"],"harmed":["Women and girls","Women","Rite Aid customers","People of color","Latino people","Consumers","Black people","Asian people"],"report_count":14,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/rite-aid-facial-recognition-disproportionately-misidentified-minority"},{"incident_id":618,"slug":"navy-federal-credit-union-faces-allegations-of-racial-bias-in-mortgage","occurred_on":"2023-12-14","title":"Navy Federal Credit Union Faces Allegations of Racial Bias in Mortgage Approvals","description":"Navy Federal Credit Union, serving military members and veterans, faced allegations of racial bias in its mortgage approval process, which relies on automated underwriting technology. In 2022, data revealed significant disparities in loan approvals, with over 50% of Black applicants denied, compared to higher approval rates for white applicants.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Federal Navy Credit Union"],"developers":["Unknown Developer Of Automated Underwriting Technology"],"harmed":["Federal Navy Credit Union Customers"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/navy-federal-credit-union-faces-allegations-of-racial-bias-in-mortgage"},{"incident_id":1311,"slug":"peppermill-casino-facial-recognition-system-reportedly-misidentified","occurred_on":"2023-09-17","title":"Peppermill Casino Facial Recognition System Reportedly Misidentified Individual, Leading to Wrongful Arrest in Reno","description":"Purported facial recognition technology deployed at Reno's Peppermill Casino reportedly misidentified Jason Killinger as an individual previously banned from the venue. Casino security reportedly detained Killinger and contacted police, leading to his arrest by the Reno PD. He was reportedly held for 11 hours, including several hours in handcuffs, before fingerprinting reportedly confirmed the misidentification. The incident resulted in alleged physical injury and civil liberties harms.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Reno Police Department","Peppermill Casino","Law enforcement","Facial recognition system deployers"],"developers":["Facial recognition system developers"],"harmed":["People misidentified by facial recognition systems","Jason Killinger"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/peppermill-casino-facial-recognition-system-reportedly-misidentified"},{"incident_id":593,"slug":"ai-photo-filter-lightens-skin-changes-eye-color-in-students","occurred_on":"2023-07-21","title":"AI Photo Filter Lightens Skin, Changes Eye Color in Student's 'Professional' Image","description":"An AI application modified an MIT student's photo to appear 'professional' by lightening her skin and changing her eye color to blue, highlighting the racial bias in the training data of the program.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Playground Ai"],"developers":["Playground Ai"],"harmed":["Rona Wang, Minority Groups"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/ai-photo-filter-lightens-skin-changes-eye-color-in-students"},{"incident_id":579,"slug":"harmful-stereotyping-of-non-cisgendered-people-via-text-to-image-systems","occurred_on":"2023-07-03","title":"Harmful Stereotyping of Non-Cisgendered People via Text-to-Image Systems","description":"Text-to-image systems such as DALL-E are allegedly generating biased and often insulting representations of non-cisgender identities. The systems tend to generate stereotypical and sexualized images when prompted with gender identity terms like \"trans,\" \"nonbinary,\" or \"queer,\" highlighting systemic issues of bias.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Dall E"],"developers":["Openai"],"harmed":["Non Cisgender Individuals, Lgbtq+ Community"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/harmful-stereotyping-of-non-cisgendered-people-via-text-to-image-systems"},{"incident_id":580,"slug":"alleged-gender-discrimination-in-facebook-job-ads-algorithm","occurred_on":"2023-06-12","title":"Alleged Gender Discrimination in Facebook Job Ads Algorithm","description":"Facebook's ad delivery algorithm allegedly disproportionately showed job advertisements to one gender. Despite claims of non-discrimination, the algorithm's actions seem to perpetuate societal biases, which in turn could potentially limit opportunities for certain groups and hinder gender equity in the workplace.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Meta"],"developers":["Meta"],"harmed":["Women and girls","Women","Underrepresented genders","Social media users","General public","Facebook users","Advertisers"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/alleged-gender-discrimination-in-facebook-job-ads-algorithm"},{"incident_id":582,"slug":"racial-bias-in-lung-function-diagnostic-algorithm-leads","occurred_on":"2023-06-01","title":"Racial Bias in Lung Function Diagnostic Algorithm Leads to Underdiagnosis in Black Men","description":"A study published in JAMA Network Open reveals that racial bias built into a commonly used medical diagnostic algorithm for lung function may be leading to underdiagnoses of breathing problems in Black men. The study suggests that as many as 40% more Black male patients might have been accurately diagnosed if the software were not racially biased. The software algorithm adjusts diagnostic thresholds based on race, affecting medical treatments and interventions.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["University of Pennsylvania Health System"],"developers":["Unknown"],"harmed":["Black men who underwent lung function tests between 2010 and 2020 and potentially received inaccurate or delayed diagnos"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/racial-bias-in-lung-function-diagnostic-algorithm-leads"},{"incident_id":587,"slug":"apparent-failure-to-accurately-label-primates-in-image-recognition","occurred_on":"2023-05-22","title":"Apparent Failure to Accurately Label Primates in Image Recognition Software Due to Alleged Fear of Racial Bias","description":"Eight years after Google Photos mislabeled images of Black individuals as \"gorillas,\" image recognition software by Google, Apple, Amazon, and Microsoft still shows signs of either avoiding or inaccurately categorizing primates. Tests reveal that Google and Apple Photos refrain from labeling primates altogether, possibly to avoid the risk of perpetuating racial stereotypes. Microsoft OneDrive fails to identify any animals, while Amazon Photos overgeneralizes in its labeling.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Google, Apple, Amazon, Microsoft"],"developers":["Google, Apple, Amazon, Microsoft"],"harmed":["Consumers Relying On Accurate Image Categorization, Minority Groups"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/apparent-failure-to-accurately-label-primates-in-image-recognition"},{"incident_id":588,"slug":"australian-terrorism-prediction-tool-reportedly-disparately-impacts","occurred_on":"2023-05-12","title":"Australian Terrorism Prediction Tool Reportedly Disparately Impacts Persons with Autism","description":"An independent report found that the Vera-2R tool, designed to predict the risk of future terrorist activities, considered autism as a risk factor despite lacking empirical evidence to support this claim. The report called into question the tool's overall validity and reliability, stating it was \"extremely poor\" at accurately predicting risk. The inclusion of autism as a risk factor had potentially serious implications for the tool's use and credibility.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["New South Wales Government","Australian Federal Government"],"developers":["Unspecified"],"harmed":["people with autism","lawyers and other experts who were not informed of the tool's limitations","Individuals assessed as high-risk based on the flawed criteria","General public of Australia","General public"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/australian-terrorism-prediction-tool-reportedly-disparately-impacts"},{"incident_id":483,"slug":"indian-police-allegedly-tortured-and-killed-innocent-man-following","occurred_on":"2023-02-02","title":"Indian Police Allegedly Tortured and Killed Innocent Man Following Facial Misidentification","description":"A resident in Medak, India died allegedly due to custodial torture by the local police, who misidentified him as a suspect in a theft case using facial recognition.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Telangana Police","Medak Police"],"developers":["Unknown"],"harmed":["Mohammed Khadeer"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/indian-police-allegedly-tortured-and-killed-innocent-man-following"},{"incident_id":484,"slug":"us-cbp-apps-failure-to-detect-black-faces-reportedly-blocked-asylum","occurred_on":"2023-01-18","title":"US CBP App's Failure to Detect Black Faces Reportedly Blocked Asylum Applications","description":"CBP One's facial recognition feature was reportedly disproportionately failing to detect faces of Black asylum seekers from Haiti and African countries, effectively blocking their asylum applications.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Us Customs And Border Protection"],"developers":["Us Customs And Border Protection"],"harmed":["Haitian Asylum Seekers, African Asylum Seekers, Black Asylum Seekers"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/us-cbp-apps-failure-to-detect-black-faces-reportedly-blocked-asylum"},{"incident_id":416,"slug":"facebooks-job-ad-algorithm-allegedly-biased-against-older-and-female","occurred_on":"2022-12-01","title":"Facebook's Job Ad Algorithm Allegedly Biased against Older and Female Workers","description":"Facebook's algorithm was alleged in a complaint by Real Women in Trucking to have selectively shown job advertisements disproportionately against older and female workers in favor of younger men for blue-collar positions.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Meta"],"developers":["Meta"],"harmed":["Women and girls","Women","Social media users","Real Women in Trucking","Facebook users"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/facebooks-job-ad-algorithm-allegedly-biased-against-older-and-female"},{"incident_id":385,"slug":"canadian-polices-release-of-suspects-ai-generated-facial-photo","occurred_on":"2022-10-04","title":"Canadian Police's Release of Suspect's AI-Generated Facial Photo Reportedly Reinforced Racial Profiling","description":"The Edmonton Police Service (EPS) in Canada released a facial image of a Black male suspect generated by an algorithm using DNA phenotyping, which was denounced by the local community as racial profiling.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Edmonton Police Service"],"developers":["Parabon Nanolabs"],"harmed":["Black Residents In Edmonton"],"report_count":6,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/canadian-polices-release-of-suspects-ai-generated-facial-photo"},{"incident_id":351,"slug":"the-little-mermaid-clip-doctored-using-generative-ai-to-replace-black","occurred_on":"2022-09-13","title":"\"The Little Mermaid\" Clip Doctored Using Generative AI to Replace Black Actress with White Character","description":"A Twitter user reportedly modified using generative AI a short clip of Disney's 2022 version of \"The Little Mermaid,\" replacing a Black actress with a white digital character.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"Human","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["@Tengazillioiniq"],"developers":["Unknown"],"harmed":["Halle Bailey, Black Actresses"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/the-little-mermaid-clip-doctored-using-generative-ai-to-replace-black"},{"incident_id":529,"slug":"stable-diffusion-reportedly-exhibited-biases-for-prompts-featuring","occurred_on":"2022-08-22","title":"Stable Diffusion Reportedly Exhibited Biases for Prompts Featuring Professions","description":"Stable Diffusion reportedly posed risks of bias and stereotyping along gender and cultural lines for prompts containing descriptors and professions.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Stability AI"],"developers":["Stability AI","Runway","LAION","EleutherAI","CompVis LMU"],"harmed":["Women and girls","Women","Minority groups"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/stable-diffusion-reportedly-exhibited-biases-for-prompts-featuring"},{"incident_id":262,"slug":"dall-e-mini-reportedly-reinforced-or-exacerbated-societal-biases-in-its","occurred_on":"2022-06-11","title":"DALL-E Mini Reportedly Reinforced or Exacerbated Societal Biases in Its Outputs as Gender and Racial Stereotypes","description":"Publicly deployed open-source model DALL-E Mini was acknowledged by its developers and found by its users to have produced images which reinforced racial and gender biases.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Boris Dayma"],"developers":["Boris Dayma, Suraj Patil, Pedro Cuenca, Khalid Saifullah, Tanishq Abraham, Phuc Le Khac, Luke Melas, Ritobrata Ghosh"],"harmed":["Minority Groups, Underrepresented Groups"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/dall-e-mini-reportedly-reinforced-or-exacerbated-societal-biases-in-its"},{"incident_id":844,"slug":"saferent-ai-screening-tool-allegedly-discriminated-against-housing","occurred_on":"2022-05-25","title":"SafeRent AI Screening Tool Allegedly Discriminated Against Housing Voucher Applicants","description":"SafeRent’s AI-powered tenant screening tool used credit history and non-rental-related debts to assign scores, disproportionately penalizing Black and Hispanic renters and those using housing vouchers. The reported discriminatory housing outcomes violated the Fair Housing Act and Massachusetts law. A class action lawsuit (Louis, et al. v. SafeRent Solutions, et al.) resulted in a $2.275 million settlement and changes to SafeRent’s practices.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Landlords"],"developers":["Saferent Solutions"],"harmed":["Renters, Monica Douglas, Massachusetts Renters, Mary Louis, Hispanic Renters, Black Renters"],"report_count":5,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/saferent-ai-screening-tool-allegedly-discriminated-against-housing"},{"incident_id":179,"slug":"dall-e-2-reported-for-gender-and-racially-biased-outputs","occurred_on":"2022-04-01","title":"DALL-E 2 Reported for Gender and Racially Biased Outputs","description":"Developers of OpenAI's DALL-E 2 cited risks of the model, varying from misuse as disinformation and explicit content generation, to gender and racial bias.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Openai"],"developers":["Openai"],"harmed":["Underrepresented Groups, Minority Groups"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/dall-e-2-reported-for-gender-and-racially-biased-outputs"},{"incident_id":192,"slug":"three-make-up-artists-lost-jobs-following-black-box-automated-decision","occurred_on":"2022-03-17","title":"Three Make-Up Artists Lost Jobs Following Black-Box Automated Decision by HireVue","description":"Three make-up artists lost their positions following an algorithmically-assessed video interview by HireVue who reportedly failed to provide adequate explanation of the findings.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Estee Lauder"],"developers":["Hirevue"],"harmed":["Pseudonymous Estee Lauder'S Former Staff"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/three-make-up-artists-lost-jobs-following-black-box-automated-decision"},{"incident_id":301,"slug":"honorlocks-ai-proctoring-system-reportedly-led-broward-college","occurred_on":"2022-02-15","title":"Honorlock's AI Proctoring System Reportedly Led Broward College to Accuse a Teenager of Cheating","description":"During a February 2022 online biology exam, Honorlock reportedly flagged a Florida teenager taking a Broward College class as suspicious. The student denied cheating, but Broward College allegedly relied on the remote-proctoring system's flag in pursuing an academic-dishonesty accusation that the student said was unfounded.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Broward College"],"developers":["Honorlock"],"harmed":["Unnamed Broward College student","University students","Students","Epistemic integrity","Educational communities"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/honorlocks-ai-proctoring-system-reportedly-led-broward-college"},{"incident_id":154,"slug":"justice-departments-recidivism-risk-algorithm-pattern-allegedly-caused","occurred_on":"2022-01-26","title":"Justice Department’s Recidivism Risk Algorithm PATTERN Allegedly Caused Persistent Disparities Along Racial Lines","description":"Department of Justice’s inmate-recidivism risk assessment tool was reported to have produced racially uneven results, misclassifying risk levels for inmates of color.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm event","sectors":["law enforcement"],"countries":["US"],"deployers":["United States Department of Justice"],"developers":["United States Department of Justice"],"harmed":["inmates of color","Incarcerated people"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/justice-departments-recidivism-risk-algorithm-pattern-allegedly-caused"},{"incident_id":611,"slug":"uk-government-ai-allegedly-targets-disproportionate-numbers-of-certain","occurred_on":"2021-12-06","title":"UK Government AI Allegedly Targets Disproportionate Numbers of Certain Nationals for Fraud Review","description":"The UK's Department for Work and Pensions (DWP) faced scrutiny after many Bulgarian nationals reported unexplained suspensions of their Universal Credit benefits. The MP for Edmonton raised concerns about potential nationality-based targeting for benefit fraud investigations, leading to poverty and homelessness among affected individuals. The Home Office's own equality impact assessment found it was flagging a disproportionate number of marriages from Greece, Albania, Bulgaria and Romania.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Various British government offices","Home Office","Department for Work and Pensions","British government"],"developers":["Home Office","Department for Work and Pensions","British government"],"harmed":["Romanians in the United Kingdom","Greeks in the United Kingdom","Bulgarians in the United Kingdom","British public","Albanians in the United Kingdom"],"report_count":13,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/uk-government-ai-allegedly-targets-disproportionate-numbers-of-certain"},{"incident_id":312,"slug":"startups-accent-translation-ai-denounced-as-reinforcing-racial-bias","occurred_on":"2021-08-15","title":"Startup's Accent Translation AI Denounced as Reinforcing Racial Bias","description":"A startup’s use of AI voice technology to alter or remove accents for call center agents was scrutinized by critics as reaffirming bias, despite the company’s claim.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Sanas"],"developers":["Sanas"],"harmed":["Call Center Agents Having Non Midwestern American Accent, People Having Non Midwestern American Accent"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/startups-accent-translation-ai-denounced-as-reinforcing-racial-bias"},{"incident_id":425,"slug":"state-farm-allegedly-discriminated-against-black-customers-in-claim","occurred_on":"2021-06-12","title":"State Farm Allegedly Discriminated against Black Customers in Claim Payout","description":"State Farm's automated claims processing method was alleged in a class action lawsuit to have disproportionately against Black policyholders when paying out insurance claims.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["State Farm"],"developers":["State Farm"],"harmed":["Black State Farm Customers"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/state-farm-allegedly-discriminated-against-black-customers-in-claim"},{"incident_id":401,"slug":"kannada-insulted-by-googles-featured-answer-as-ugliest-language-in-india","occurred_on":"2021-06-03","title":"Kannada Insulted by Google's Featured Answer as \"Ugliest Language in India\"","description":"Google's knowledge-graph-powered algorithm showed Kannada in its featured Answer Box when prompted \"ugliest language in India,\" causing outrage from Kannada-speaking people and government.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Google"],"developers":["Google"],"harmed":["The Karnataka Government, Kannada Speakers"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/kannada-insulted-by-googles-featured-answer-as-ugliest-language-in-india"},{"incident_id":359,"slug":"facebook-instagram-and-twitter-cited-errors-in-automated-systems","occurred_on":"2021-05-23","title":"Facebook, Instagram, and Twitter Cited Errors in Automated Systems as Cause for Blocking pro-Palestinian Content on Israeli-Palestinian Conflict","description":"Facebook, Instagram, and Twitter wrongly blocked or restricted millions of pro-Palestinian posts and accounts related to the Israeli-Palestinian conflict, citing errors in their automated content moderation system.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Facebook, Instagram, Twitter"],"developers":["Facebook, Instagram, Twitter"],"harmed":["Palestinian Social Media Users, Facebook Users, Instagram Users, Facebook Employees Having Families Affected By The Conflict, X (Twitter) Users"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/facebook-instagram-and-twitter-cited-errors-in-automated-systems"},{"incident_id":265,"slug":"black-uber-eats-driver-allegedly-subjected-to-excessive-photo-checks","occurred_on":"2021-04-01","title":"Black Uber Eats Driver Allegedly Subjected to Excessive Photo Checks and Dismissed via FRT Results","description":"A lawsuit by a former Uber Eats delivery driver alleged the company to have wrongfully dismissed him due to frequent false mismatches of his verification selfies, and discriminated against him via excessive verification checks.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Uber Eats"],"developers":["Uber Eats"],"harmed":["Uber Eats Black delivery drivers","Pa Edrissa Manjang"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/black-uber-eats-driver-allegedly-subjected-to-excessive-photo-checks"},{"incident_id":104,"slug":"californias-algorithm-considered-zip-codes-in-vaccine-distribution","occurred_on":"2021-02-12","title":"California's Algorithm Considered ZIP Codes in Vaccine Distribution, Allegedly Excluding Low-Income Neighborhoods and Communities of Color","description":"California's vaccine-distribution algorithm used ZIP codes as opposed to census tracts in its decision-making, which critics said undermined equity and access for vulnerable communities who are largely low-income, underserved neighborhoods with low Healthy Places Index scores.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["public administration","human health and social work activities"],"countries":["US"],"deployers":["California Department Of Public Health"],"developers":["Blue Shield Of California"],"harmed":["California Low Income Neighborhoods, California Communities Of Color"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/californias-algorithm-considered-zip-codes-in-vaccine-distribution"},{"incident_id":142,"slug":"facebooks-advertisement-moderation-system-allegedly-routinely","occurred_on":"2021-02-11","title":"Facebook's Advertisement Moderation System Allegedly Routinely Misidentified Adaptive Fashion Products as Medical Equipment and Blocked Their Sellers","description":"Facebook platforms' automated ad moderation system falsely classified adaptive fashion products as medical and health care products and services, resulting in regular bans and appeals faced by their retailers.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm issue","sectors":["wholesale and retail trade","information and communication"],"countries":[],"deployers":["Meta"],"developers":["Meta"],"harmed":["Adaptive Fashion Retailers, Facebook Users, Instagram Users"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/facebooks-advertisement-moderation-system-allegedly-routinely"},{"incident_id":273,"slug":"faceapp-predicted-different-genders-for-similar-user-photos-with-slight","occurred_on":"2020-12-24","title":"FaceApp Predicted Different Genders for Similar User Photos with Slight Variations","description":"FaceApp’s algorithm was reported by a user to have predicted different genders for two mostly identical facial photos with only a slight difference in eyebrow thickness.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication","arts, entertainment and recreation"],"countries":["US"],"deployers":["Faceapp"],"developers":["Faceapp"],"harmed":["Faceapp Non Binary Presenting Users, Faceapp Transgender Users, Faceapp Users"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/faceapp-predicted-different-genders-for-similar-user-photos-with-slight"},{"incident_id":133,"slug":"online-trolls-allegedly-abused-tiktoks-automated-content-reporting","occurred_on":"2020-12-15","title":"Online Trolls Allegedly Abused TikTok’s Automated Content Reporting System to Discriminate against Marginalized Creators","description":"TikTok's automated content reporting system was allegedly abused by online trolls to intentionally misreport content created by users of marginalized groups.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["arts, entertainment and recreation","information and communication"],"countries":[],"deployers":["Tiktok"],"developers":["Tiktok"],"harmed":["Tiktok Content Creators Of Marginalized Groups"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/online-trolls-allegedly-abused-tiktoks-automated-content-reporting"},{"incident_id":94,"slug":"court-rules-deliveroo-used-discriminatory-algorithm","occurred_on":"2020-11-27","title":"Court Rules Deliveroo Used 'Discriminatory' Algorithm","description":"In December 2020, an Italian court ruled that Deliveroo’s employee ‘reliability’ algorithm illegally discriminated against workers with legitimate reasons for cancelling shifts.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm event","sectors":["accommodation and food service activities"],"countries":["IT"],"deployers":["Deliveroo"],"developers":["Deliveroo"],"harmed":["Deliveroo workers with legitimate reasons for cancelling shifts","Deliveroo workers"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in Italy.","related_policy_slugs":["italy-law-132-2025-ai","us-colorado-ai-act","india-dpdp-act","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/court-rules-deliveroo-used-discriminatory-algorithm"},{"incident_id":83,"slug":"ai-spam-filters-allegedly-block-legitimate-emails-based-on-biased","occurred_on":"2020-10-22","title":"AI Spam Filters Allegedly Block Legitimate Emails Based on Biased Keyword Detection","description":"AlgorithmWatch tested spam filtering algorithms across Gmail, Yahoo, Outlook, GMX, and LaPoste. Their findings reportedly showed that Microsoft Outlook’s spam filter flagged emails based on specific keywords that led to racial and content-based biases blocking legitimate communications. Emails mentioning Nigeria or containing certain financial and sexual health terms were found to be disproportionately marked as spam.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":[],"deployers":["Yahoo, Outlook, Laposte, Gmx, Gmail"],"developers":["Yahoo, Microsoft, Laposte, Google, Gmx"],"harmed":["Yahoo! Mail Users, Microsoft Outlook Users, Laposte Users, Gmx Users, Gmail Users"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/ai-spam-filters-allegedly-block-legitimate-emails-based-on-biased"},{"incident_id":103,"slug":"twitters-image-cropping-tool-allegedly-showed-gender-and-racial-bias","occurred_on":"2020-09-18","title":"Twitter’s Image Cropping Tool Allegedly Showed Gender and Racial Bias","description":"Twitter's photo cropping algorithm was revealed by researchers to favor white and women faces in photos containing multiple faces, prompting the company to stop its use on mobile platform.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":[],"deployers":["Twitter"],"developers":["Twitter"],"harmed":["Twitter Non White Users, Twitter Non Male Users, X (Twitter) Users"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/twitters-image-cropping-tool-allegedly-showed-gender-and-racial-bias"},{"incident_id":374,"slug":"uk-ofquals-algorithm-disproportionately-provided-lower-grades-than","occurred_on":"2020-08-13","title":"UK Ofqual's Algorithm Disproportionately Provided Lower Grades Than Teachers' Assessments","description":"UK Office of Qualifications and Examinations Regulation (Ofqual)'s grade-standardization algorithm providing predicted grades for A level and GCSE qualifications in the UK, Wales, Northern Ireland, and Scotland was reportedly giving grades lower than teachers' assessments, and disproportionately for state schools.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Uk Office Of Qualifications And Examinations Regulation"],"developers":["Uk Office Of Qualifications And Examinations Regulation"],"harmed":["Underprivileged Pupils, Students, Pupils In State Schools, Gcse Pupils, Epistemic Integrity, Educational Communities, A Level Pupils"],"report_count":8,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/uk-ofquals-algorithm-disproportionately-provided-lower-grades-than"},{"incident_id":558,"slug":"activists-allege-nypds-application-of-facial-recognition-interfered","occurred_on":"2020-08-07","title":"Activists Allege NYPD's Application of Facial Recognition Interfered with Right to Protest","description":"Black Lives Matter activists alleged being targeted for arrest by New York Police using facial recognition, interfering with their right to protest.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"Human","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["New York City Police Department"],"developers":["Surveillance technology developers","Facial recognition system developers","Clearview AI"],"harmed":["Privacy","General public of New York City","General public","Derrick Ingram","Democratic integrity","Black people","Black Lives Matter activists","Biometric data subjects"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/activists-allege-nypds-application-of-facial-recognition-interfered"},{"incident_id":118,"slug":"openais-gpt-3-associated-muslims-with-violence","occurred_on":"2020-08-06","title":"OpenAI's GPT-3 Associated Muslims with Violence","description":"Users and researchers revealed generative AI GPT-3 associating Muslims to violence in prompts, resulting in disturbingly racist and explicit outputs such as casting Muslim actor as a terrorist.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication","professional, scientific and technical activities"],"countries":["US"],"deployers":["Openai"],"developers":["Openai"],"harmed":["Muslims"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/openais-gpt-3-associated-muslims-with-violence"},{"incident_id":331,"slug":"bug-in-instagrams-related-hashtags-algorithm-allegedly-caused","occurred_on":"2020-08-05","title":"Bug in Instagram’s “Related Hashtags” Algorithm Allegedly Caused Disproportionate Treatment of Political Hashtags","description":"A bug was reported by Instagram’s spokesperson to have prevented an algorithm from populating related hashtags for thousands of hashtags, resulting in an allege preferential treatment for some politically partisan hashtags.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Instagram"],"developers":["Instagram"],"harmed":["Instagram Users"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/bug-in-instagrams-related-hashtags-algorithm-allegedly-caused"},{"incident_id":115,"slug":"genderifys-ai-to-predict-a-persons-gender-revealed-by-free-api-users","occurred_on":"2020-07-28","title":"Genderify’s AI to Predict a Person’s Gender Revealed by Free API Users to Exhibit Bias","description":"A company's AI predicting a person's gender based on their name, email address, or username was reported by its users to show biased and inaccurate results.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["US"],"deployers":["Genderify"],"developers":["Genderify"],"harmed":["Genderify Customers, Minority Groups"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/genderifys-ai-to-predict-a-persons-gender-revealed-by-free-api-users"},{"incident_id":222,"slug":"thoughts-app-allegedly-created-toxic-tweets","occurred_on":"2020-07-18","title":"Thoughts App Allegedly Created Toxic Tweets","description":"Tweets created by Thoughts, a tweet generation app that leverages OpenAI’s GPT-3, allegedly exhibited toxicity when given prompts related to minority groups.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Satria Technologies"],"developers":["Openai"],"harmed":["Thoughts Users, X (Twitter) Users"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/thoughts-app-allegedly-created-toxic-tweets"},{"incident_id":113,"slug":"facebooks-ai-put-primates-label-on-video-featuring-black-men","occurred_on":"2020-06-27","title":"Facebook's AI Put \"Primates\" Label on Video Featuring Black Men","description":"Facebook's AI mislabeled video featuring Black men as a video about \"primates,\" resulting in an offensive prompt message for users who watched the video.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":[],"deployers":["Facebook"],"developers":["Facebook"],"harmed":["Black People, Facebook Users"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/facebooks-ai-put-primates-label-on-video-featuring-black-men"},{"incident_id":165,"slug":"image-upscaling-algorithm-pulse-allegedly-produced-facial-images","occurred_on":"2020-06-20","title":"Image Upscaling Algorithm PULSE Allegedly Produced Facial Images with Caucasian Features More Often","description":"Image upscaling tool PULSE powered by NVIDIA's StyleGAN reportedly generated faces with Caucasian features more often, although AI academics, engineers, and researchers were not in agreement about where the source of bias was.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Duke Researchers"],"developers":["Duke Researchers"],"harmed":["People Having Non Caucasian Facial Features"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/image-upscaling-algorithm-pulse-allegedly-produced-facial-images"},{"incident_id":367,"slug":"igpt-simclr-learned-biased-associations-from-internet-training-data","occurred_on":"2020-06-17","title":"iGPT, SimCLR Learned Biased Associations from Internet Training Data","description":"Unsupervised image generation models trained using Internet images such as iGPT and SimCLR were shown to have embedded racial, gender, and intersectional biases, resulting in stereotypical depictions.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Pre-deployment","harm_level":"none","sectors":["information and communication"],"countries":[],"deployers":["Openai, Google"],"developers":["Openai, Google"],"harmed":["Underrepresented Groups In Training Data, Minority Groups"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/igpt-simclr-learned-biased-associations-from-internet-training-data"},{"incident_id":127,"slug":"microsofts-algorithm-allegedly-selected-photo-of-the-wrong-mixed-race","occurred_on":"2020-06-06","title":"Microsoft’s Algorithm Allegedly Selected Photo of the Wrong Mixed-Race Person Featured in a News Story","description":"A news story published on MSN.com featured a photo of the wrong mixed-race person that was allegedly selected by an algorithm, following Microsoft’s layoff and replacement of journalists and editorial workers at its organizations with AI systems.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["arts, entertainment and recreation","information and communication"],"countries":[],"deployers":["Microsoft, Msn.Com"],"developers":["Microsoft"],"harmed":["Jade Thirlwall, Leigh Anne Pinnock"],"report_count":12,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/microsofts-algorithm-allegedly-selected-photo-of-the-wrong-mixed-race"},{"incident_id":424,"slug":"canadian-universities-ai-proctoring-tools-reportedly-collected-student","occurred_on":"2020-03-09","title":"Canadian Universities' AI Proctoring Tools Reportedly Collected Student Biometric Data Without Adequate Consent","description":"During the COVID-19 pandemic, Canadian universities reportedly used AI-enabled remote-proctoring tools for online exams. A University of Ottawa project supported by the Office of the Privacy Commissioner of Canada found that tools such as Respondus Monitor, ProctorU, ProctorTrack, Proctorio, ProctorExam, and Examity collected student personal or biometric data under conditions not conducive to meaningful consent.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Universities, Professors, Educators, Canadian Universities"],"developers":["Respondus Monitor, Proctoru, Proctortrack, Proctorio, Proctorexam, Examity"],"harmed":["University Students, Universities, Students, Educational Communities, Canadian Students, Privacy, Biometric Data Subjects"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/canadian-universities-ai-proctoring-tools-reportedly-collected-student"},{"incident_id":117,"slug":"tiktoks-suggested-accounts-algorithm-allegedly-reinforced-racial-bias","occurred_on":"2020-02-24","title":"TikTok's \"Suggested Accounts\" Algorithm Allegedly Reinforced Racial Bias through Feedback Loops","description":"TikTok's \"Suggested Accounts\" recommendations allegedly reinforced racial bias despite not basing recommendations on race or creators' profile photo.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Tiktok"],"developers":["Tiktok"],"harmed":["Tiktok Users, Tiktok Content Creators"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/tiktoks-suggested-accounts-algorithm-allegedly-reinforced-racial-bias"},{"incident_id":180,"slug":"algorithm-used-by-the-malaysian-judiciary-reportedly-recommended","occurred_on":"2020-02-19","title":"Algorithm Used by the Malaysian Judiciary Reportedly Recommended Unusually High Sentencing to a Drug Possession Case","description":"The AI system used by the Malaysian judiciary which explicitly considered age, employment, and socio-economic data provided sentencing to a drug possession case that was alleged by lawyer to be disproportionately high for the crime committed.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Malaysian Judiciary, Malaysian Courts"],"developers":["Sarawak Information Systems"],"harmed":["Malaysian Convicted People"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/algorithm-used-by-the-malaysian-judiciary-reportedly-recommended"},{"incident_id":166,"slug":"networking-platform-giggle-employs-ai-to-determine-users-gender","occurred_on":"2020-02-07","title":"Networking Platform Giggle Employs AI to Determine Users’ Gender, Allegedly Excluding Transgender Women","description":"A social networking platform, Giggle, allegedly collected, shared to third-parties, and used sensitive information and biometric data to verify whether a person is a woman via facial recognition, which critics claimed to be discriminatory against women of color and harmful towards trans women.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Giggle"],"developers":["Kairos"],"harmed":["Women of color","Women and girls","Women","Trans women","Privacy","Biometric data subjects"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/networking-platform-giggle-employs-ai-to-determine-users-gender"},{"incident_id":92,"slug":"apple-cards-credit-assessment-algorithm-allegedly-discriminated-against","occurred_on":"2019-11-11","title":"Apple Card's Credit Assessment Algorithm Allegedly Discriminated against Women","description":"Apple Card's credit assessment algorithm was reported by Goldman-Sachs customers to have shown gender bias, in which men received significantly higher credit limits than women with equal credit qualifications.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm event","sectors":["financial and insurance activities"],"countries":["UNITED STATES"],"deployers":["Goldman-Sachs"],"developers":["Apple"],"harmed":["Women and girls","Women","Apple card users","Apple card credit applicants"],"report_count":6,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/apple-cards-credit-assessment-algorithm-allegedly-discriminated-against"},{"incident_id":95,"slug":"job-screening-service-halts-facial-analysis-of-applicants","occurred_on":"2019-11-06","title":"Job Screening Service Halts Facial Analysis of Applicants","description":"In January 2021, HireVue removed the controversial AI expression tracking tool from its virtual job interview software.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":"AI tangible harm issue","sectors":["administrative and support service activities","professional, scientific and technical activities"],"countries":["US"],"deployers":["Hirevue"],"developers":["Hirevue"],"harmed":["Job Applicants Using Hirevue, Hirevue Customers"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/job-screening-service-halts-facial-analysis-of-applicants"},{"incident_id":124,"slug":"optum-algorithmic-health-risk-scores-reportedly-underestimated-black","occurred_on":"2019-10-24","title":"Optum Algorithmic Health Risk Scores Reportedly Underestimated Black Patients' Needs","description":"Researchers analyzing an Optum algorithm used by a large academic hospital reportedly found evidence that it tended to under-predict the health care needs of Black patients, which in turn appeared to result in lower prioritization for extra care programs compared with white patients exhibiting similar health profiles.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm event","sectors":["human health and social work activities"],"countries":["US"],"deployers":["unnamed large academic hospital"],"developers":["Optum"],"harmed":["Black patients"],"report_count":7,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/optum-algorithmic-health-risk-scores-reportedly-underestimated-black"},{"incident_id":272,"slug":"grab-tweaked-matchmaking-algorithm-providing-preferential-treatment","occurred_on":"2019-10-08","title":"Grab Tweaked Matchmaking Algorithm, Providing Preferential Treatment to Drivers Registered with Affiliated Car Rental Service","description":"Grab Indonesia was fined by the Indonesian Competition Commission (KPPU) for unfairly favoring drivers who rented cars via the Grab-affiliated company Teknologi Pengangkutan Indonesia (TPI), including offering more rides via their matchmaking algorithm.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Grab"],"developers":["Grab"],"harmed":["non-TPI-registered Grab drivers","Grab drivers in Indonesia","Grab drivers"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/grab-tweaked-matchmaking-algorithm-providing-preferential-treatment"},{"incident_id":439,"slug":"detroit-police-wrongfully-arrested-black-man-due-to-faulty-facial","occurred_on":"2019-07-31","title":"Detroit Police Wrongfully Arrested Black Man Due To Faulty Facial Recognition","description":"A Black man was wrongfully detained by the Detroit Police Department as a result of a false facial recognition (FRT) result.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Detroit Police Department"],"developers":["DataWorks Plus"],"harmed":["Michael Oliver","Black people in Detroit","Black people"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/detroit-police-wrongfully-arrested-black-man-due-to-faulty-facial"},{"incident_id":489,"slug":"workdays-ai-tools-allegedly-enabled-employers-to-discriminate-against","occurred_on":"2019-06-03","title":"Workday's AI Tools Allegedly Enabled Employers to Discriminate against Applicants of Protected Groups","description":"Workday's algorithmic screening systems were alleged in a lawsuit allowing employers to discriminate against African-Americans, people over 40, and people with disabilities.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["professional, scientific and technical activities","information and communication"],"countries":["US"],"deployers":["Workday"],"developers":["Workday"],"harmed":["Derek Mobley, Applicants With Disabilities, Applicants Over 40, African American Applicants"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/workdays-ai-tools-allegedly-enabled-employers-to-discriminate-against"},{"incident_id":546,"slug":"algorithm-to-distribute-social-welfare-reported-for-oversimplifying","occurred_on":"2019-05-31","title":"Algorithm to Distribute Social Welfare Reported for Oversimplifying Economic Vulnerability","description":"Takaful cash transfer program's algorithm which ranks families by their economic vulnerability level to determine financial assistance reportedly oversimplified people's economic situation, fueling social tension and perceptions of unfairness.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["National Aid Fund"],"developers":["The World Bank, Unicef, World Food Programme"],"harmed":["Jordanians In Poverty"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/algorithm-to-distribute-social-welfare-reported-for-oversimplifying"},{"incident_id":161,"slug":"facebooks-ad-delivery-reportedly-excluded-audience-along-racial","occurred_on":"2019-04-03","title":"Facebook's Ad Delivery Reportedly Excluded Audience along Racial and Gender Lines","description":"Facebook's housing and employment ad delivery process allegedly resulted in skews in exposure for some users along demographic lines such as gender and racial identity.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Meta"],"developers":["Meta"],"harmed":["Women and girls","Women","Social media users","Facebook users","Black Facebook users"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/facebooks-ad-delivery-reportedly-excluded-audience-along-racial"},{"incident_id":388,"slug":"facial-recognition-pilot-in-bahia-reportedly-targeted-black-and-poor","occurred_on":"2018-12-01","title":"Facial Recognition Pilot in Bahia Reportedly Targeted Black and Poor People","description":"Facial recognition deployed in a pilot project by the local government of Bahia despite having minimal hit rate reportedly targeted Black and poor people disproportionately.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["The Government In Bahia, Bahia'S Secretary Of Public Security"],"developers":["Huawei"],"harmed":["Black People In Brazil, Black People In Bahia"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/facial-recognition-pilot-in-bahia-reportedly-targeted-black-and-poor"},{"incident_id":405,"slug":"schufa-credit-scoring-in-germany-reported-for-unreliable-and-imbalanced","occurred_on":"2018-11-28","title":"Schufa Credit Scoring in Germany Reported for Unreliable and Imbalanced Scores","description":"Creditworthiness Schufa scores in Germany reportedly privileged older and female consumers, and people who changed addresses less frequently, and were unreliable depending on scoring version.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Schufa Holding AG"],"developers":["Schufa Holding AG"],"harmed":["young men having credit scores","people scored on old scoring versions","people changing addresses frequently"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/schufa-credit-scoring-in-germany-reported-for-unreliable-and-imbalanced"},{"incident_id":454,"slug":"emotion-detection-models-showed-disparate-performance-along-racial-lines","occurred_on":"2018-11-09","title":"Emotion Detection Models Showed Disparate Performance along Racial Lines","description":"Emotion detection tools by Face++ and Microsoft's Face API allegedly scored smiling or defaulted ambiguous facial photos for Black faces as negative emotion more often than for white faces.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication","unclear"],"countries":["US"],"deployers":["Megvii, Microsoft"],"developers":["Megvii, Microsoft"],"harmed":["Black People"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/emotion-detection-models-showed-disparate-performance-along-racial-lines"},{"incident_id":238,"slug":"oregons-screening-tool-for-child-abuse-cases-discontinued-following","occurred_on":"2018-10-01","title":"Oregon’s Screening Tool for Child Abuse Cases Discontinued Following Concerns of Racial Bias","description":"Oregon’s Department of Human Services (DHS) stopped using its Safety at Screening Tool, that is aimed to predict the risk that children wind up in foster care or be investigated in the future, and opted for a new process allegedly to reduce disparities and improve racially equitable decision-making.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Oregon Department of Human Services"],"developers":["Oregon Department of Human Services"],"harmed":["families of minority groups","children of minority groups"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/oregons-screening-tool-for-child-abuse-cases-discontinued-following"},{"incident_id":101,"slug":"dutch-families-wrongfully-accused-of-tax-fraud-due-to-discriminatory","occurred_on":"2018-09-01","title":"Dutch Families Wrongfully Accused of Tax Fraud Due to Discriminatory Algorithm","description":"A childcare benefits system in the Netherlands falsely accused thousands of families of fraud, in part due to an algorithm that treated having a second nationality as a risk factor.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm event","sectors":["public administration"],"countries":["NL"],"deployers":["Dutch Tax Authority"],"developers":["Unknown"],"harmed":["Dutch Tax Authority, Dutch Families"],"report_count":6,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the Netherlands.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/dutch-families-wrongfully-accused-of-tax-fraud-due-to-discriminatory"},{"incident_id":93,"slug":"hud-charges-facebook-with-enabling-housing-discrimination","occurred_on":"2018-08-13","title":"HUD charges Facebook with enabling housing discrimination","description":"In March 2019 the U.S. Department of Housing and Urban Development charged Facebook with violating the Fair Housing Act by allowing real estate sellers to target advertisements in a discriminatory manner.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["US"],"deployers":["Facebook"],"developers":["Facebook"],"harmed":["Non American Born Facebook Users, Non Christian Facebook Users, Facebook Users Interested In Accessibility, Facebook Users Interested In Hispanic Culture, Minority Groups"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/hud-charges-facebook-with-enabling-housing-discrimination"},{"incident_id":107,"slug":"chinese-tech-firms-allegedly-developed-facial-recognition-to-identify","occurred_on":"2018-07-20","title":"Chinese Tech Firms Allegedly Developed Facial Recognition to Identify People by Race, Targeting Uyghur Muslims","description":"Various Chinese firms were revealed by patent applications to have developed facial recognition capable of detecting people by race, which critics feared would enable persecution and discrimination of Uyghur Muslims.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["None"],"developers":["Huawei, Megvii, Sensetime, Alibaba, Baibu"],"harmed":["Uyghur People"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/chinese-tech-firms-allegedly-developed-facial-recognition-to-identify"},{"incident_id":396,"slug":"transgender-uber-drivers-mistakenly-kicked-off-app-for-appearance","occurred_on":"2018-07-04","title":"Transgender Uber Drivers Mistakenly Kicked off App for Appearance Change during Gender Transitions","description":"Transgender Uber drivers reported being automatically deactivated from the app due to Real-Time ID Check failing to account for difference in appearance of people undergoing gender transitions.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Uber"],"developers":["Uber"],"harmed":["transgender Uber drivers"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/transgender-uber-drivers-mistakenly-kicked-off-app-for-appearance"},{"incident_id":188,"slug":"argentinian-city-government-deployed-teenage-pregnancy-predictive","occurred_on":"2018-04-11","title":"Argentinian City Government Deployed Teenage-Pregnancy Predictive Algorithm Using Invasive Demographic Data","description":"In 2018, during the abortion-decriminalization debate in Argentina, the Salta city government deployed a teenage-pregnancy predictive algorithm built by Microsoft that allegedly lacked a defined purpose, explicitly considered sensitive information such as disability and whether their home had access to hot water.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Salta city government","Government of Argentina"],"developers":["Microsoft"],"harmed":["Women and girls","Women","Salta teenage girls","Salta girls of minority groups","Privacy","Girls","Adolescent girls"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/argentinian-city-government-deployed-teenage-pregnancy-predictive"},{"incident_id":261,"slug":"robot-deployed-by-animal-shelter-to-patrol-sidewalks-outside-its-office","occurred_on":"2017-11-15","title":"Robot Deployed by Animal Shelter to Patrol Sidewalks outside Its Office, Warding off Homeless People in San Francisco","description":"Society for the Prevention of Cruelty to Animals (SPCA) deployed a Knightscope robot to autonomously patrol the area outside its office and ward off homeless people, which was criticized by residents as a tool of intimidation and ordered by the city of San Francisco to stop its use on a public right-of-way.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"Human","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Society For The Prevention Of Cruelty To Animals"],"developers":["Knightscope"],"harmed":["San Francisco Homeless People"],"report_count":8,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/robot-deployed-by-animal-shelter-to-patrol-sidewalks-outside-its-office"},{"incident_id":14,"slug":"biased-sentiment-analysis","occurred_on":"2017-10-26","title":"Biased Sentiment Analysis","description":"Google Cloud's Natural Language API provided racist, homophobic, amd antisemitic sentiment analyses.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":[],"deployers":["Google"],"developers":["Google"],"harmed":["Women and girls","Women","Minority groups"],"report_count":6,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/biased-sentiment-analysis"},{"incident_id":531,"slug":"ai-assisted-body-scanners-reportedly-subjected-transgender-travelers","occurred_on":"2017-09-15","title":"AI-Assisted Body Scanners Reportedly Subjected Transgender Travelers to Invasive Body Searches","description":"Transportation Security Administration (TSA)'s use of image-processing body scanners at airports led transgender and gender-nonconforming travelers to be subjected to allegedly discriminatory and invasive searches, such as being asked to remove undergarments in private rooms by officers not of their gender.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Transportation Security Administration"],"developers":["L3Harris Technologies"],"harmed":["Transgender Travelers, Gender Nonconforming Travelers"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/ai-assisted-body-scanners-reportedly-subjected-transgender-travelers"},{"incident_id":183,"slug":"airbnbs-trustworthiness-algorithm-allegedly-banned-users-without","occurred_on":"2017-07-01","title":"Airbnb's Trustworthiness Algorithm Allegedly Banned Users without Explanation, and Discriminated against Sex Workers","description":" Airbnb allegedly considered publicly available data on users to gauge their trustworthiness via algorithmic assessment of personality and behavioral traits, resulting in unexplained bans and discriminatory bans against sex workers.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Airbnb"],"developers":["Airbnb, Trooly"],"harmed":["Sex Workers, Airbnb Users"],"report_count":6,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/airbnbs-trustworthiness-algorithm-allegedly-banned-users-without"},{"incident_id":60,"slug":"faceapp-racial-filters","occurred_on":"2017-04-25","title":"FaceApp Racial Filters","description":"FaceApp is criticized for offering racist filters.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["arts, entertainment and recreation","information and communication"],"countries":["RU"],"deployers":["Faceapp"],"developers":["Faceapp"],"harmed":["Minority Groups"],"report_count":23,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in Russia.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/faceapp-racial-filters"},{"incident_id":59,"slug":"gender-biases-in-google-translate","occurred_on":"2017-04-13","title":"Gender Biases in Google Translate","description":"A Cornell University study in 2016 highlighted Google Translate's pattern of assigning gender to occupations in a way showing an implicit gender bias against women.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":[],"deployers":["Google"],"developers":["Google"],"harmed":["Women and girls","Women"],"report_count":10,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/gender-biases-in-google-translate"},{"incident_id":502,"slug":"pennsylvania-countys-family-screening-tool-allegedly-exhibited","occurred_on":"2017-04-10","title":"Pennsylvania County's Family Screening Tool Allegedly Exhibited Discriminatory Effects","description":"Data analysis by the American Civil Liberty Union (ACLU) on Allegheny County's decision-support Family Screening Tool to predict child abuse or neglect risk found the tool resulting in higher screen-in rates for Black families and higher risk scores for households with disabled residents.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Allegheny County"],"developers":["Rhema Vaithianathan","Emily Putnam-Hornstein","Centre for Social Data Analytics"],"harmed":["People with disabilities","Hackneys family","Black families in Allegheny"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/pennsylvania-countys-family-screening-tool-allegedly-exhibited"},{"incident_id":394,"slug":"social-medias-automated-word-flagging-without-context-shifted-content","occurred_on":"2017-03-15","title":"Social Media's Automated Word-Flagging without Context Shifted Content Creators' Language Use","description":"TikTok's, YouTube's, Instagram's, and Twitch's use of algorithms to flag certain words devoid of context changed content creators' use of everyday language or discussion about certain topics in fear of their content getting flagged or auto-demonetized by mistake.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Youtube, Twitch, Tiktok, Instagram"],"developers":["Youtube, Twitch, Tiktok, Instagram"],"harmed":["Youtube Content Creators, Twitch Content Creators, Tiktok Content Creators, Instagram Content Creators"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/social-medias-automated-word-flagging-without-context-shifted-content"},{"incident_id":13,"slug":"high-toxicity-assessed-on-text-involving-women-and-minority-groups","occurred_on":"2017-02-27","title":"High-Toxicity Assessed on Text Involving Women and Minority Groups","description":"Google's Perspective API, which assigns a toxicity score to online text, seems to award higher toxicity scores to content involving non-white, male, Christian, heterosexual phrases.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["US"],"deployers":["Google"],"developers":["Google"],"harmed":["Women and girls","Women","Minority groups"],"report_count":8,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/high-toxicity-assessed-on-text-involving-women-and-minority-groups"},{"incident_id":472,"slug":"nypds-deployment-of-facial-recognition-cameras-reportedly-reinforced","occurred_on":"2016-10-08","title":"NYPD's Deployment of Facial Recognition Cameras Reportedly Reinforced Biased Policing","description":"New York Police Department’s use of facial recognition deployment of surveillance cameras were shown using crowdsourced volunteer data reinforcing discriminatory policing against minority communities.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"Human","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["New York Police Department"],"developers":["Unknown"],"harmed":["racial minorities"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/nypds-deployment-of-facial-recognition-cameras-reportedly-reinforced"},{"incident_id":47,"slug":"linkedin-search-prefers-male-names","occurred_on":"2016-09-06","title":"LinkedIn Search Prefers Male Names","description":"An investigation by The Seattle Times in 2016 found a gender bias in LinkedIn's search engine.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["US"],"deployers":["LinkedIn"],"developers":["LinkedIn"],"harmed":["Women and girls","Women"],"report_count":9,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/linkedin-search-prefers-male-names"},{"incident_id":49,"slug":"ai-beauty-judge-did-not-like-dark-skin","occurred_on":"2016-09-05","title":"AI Beauty Judge Did Not Like Dark Skin","description":"In 2016, after artificial inntelligence software Beauty.AI judged an international beauty contest and declared a majority of winners to be white, researchers found that Beauty.AI was racially biased in determining beauty.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["arts, entertainment and recreation"],"countries":[],"deployers":["Youth Laboratories"],"developers":["Youth Laboratories"],"harmed":["People With Dark Skin"],"report_count":9,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/ai-beauty-judge-did-not-like-dark-skin"},{"incident_id":37,"slug":"amazons-experimental-hiring-tool-allegedly-displayed-gender-bias","occurred_on":"2016-08-10","title":"Amazon's Experimental Hiring Tool Allegedly Displayed Gender Bias in Candidate Rankings","description":"Between 2014 and 2017, Amazon reportedly developed an AI-powered recruiting tool to score job applicants, trained on a decade of resumes purportedly drawn largely from men. Media reports say the system learned to favor male candidates, penalizing terms like \"women's\" and graduates from certain all-women's colleges. Efforts to remove these biases reportedly did not guarantee fairness, and the project was ultimately abandoned. Amazon reportedly states recruiters never solely relied on the tool.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm issue","sectors":["administrative and support service activities"],"countries":["IE"],"deployers":["Amazon"],"developers":["Amazon"],"harmed":["Women applying to Amazon","Women and girls","Women","Amazon applicants"],"report_count":34,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in Ireland.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/amazons-experimental-hiring-tool-allegedly-displayed-gender-bias"},{"incident_id":12,"slug":"common-biases-of-vector-embeddings","occurred_on":"2016-07-21","title":"Common Biases of Vector Embeddings","description":"Researchers from Boston University and Microsoft Research, New England demonstrated gender bias in the most common techniques used to embed words for natural language processing (NLP).","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["professional, scientific and technical activities"],"countries":["US"],"deployers":["Microsoft Research","Boston University"],"developers":["Microsoft Research","Google","Boston University"],"harmed":["Women and girls","Women","Minority groups"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/common-biases-of-vector-embeddings"},{"incident_id":11,"slug":"northpointe-risk-models","occurred_on":"2016-05-23","title":"Northpointe Risk Models","description":"An algorithm developed by Northpointe and used in the penal system is two times more likely to incorrectly label a black person as a high-risk re-offender and is two times more likely to incorrectly label a white person as low-risk for reoffense according to a ProPublica review.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm event","sectors":["law enforcement","public administration"],"countries":["US"],"deployers":["Northpointe"],"developers":["Northpointe"],"harmed":["Accused people"],"report_count":15,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/northpointe-risk-models"},{"incident_id":40,"slug":"compas-algorithm-reportedly-performs-poorly-in-crime-recidivism","occurred_on":"2016-05-23","title":"COMPAS Algorithm Reportedly Performs Poorly in Crime Recidivism Prediction","description":"Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), a recidivism risk-assessment algorithmic tool used in the judicial system to assess likelihood of defendants' recidivism, is found to be less accurate than random untrained human evaluators.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"AI tangible harm event","sectors":["law enforcement","public administration"],"countries":["US"],"deployers":["Equivant"],"developers":["Equivant"],"harmed":["Accused people"],"report_count":21,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/compas-algorithm-reportedly-performs-poorly-in-crime-recidivism"},{"incident_id":235,"slug":"chinese-insurer-ping-an-employed-facial-recognition-to-determine","occurred_on":"2016-04-15","title":"Chinese Insurer Ping An Employed Facial Recognition to Determine Customers’ Untrustworthiness, Which Critics Alleged to Likely Make Errors and Discriminate","description":"Customers’ untrustworthiness and unprofitability were reportedly determined by Ping An, a large insurance company in China, via facial-recognition measurements of micro-expressions and body-mass indices (BMI), which critics argue was likely to make mistakes, discriminate against certain ethnic groups, and undermine its own industry.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Ping An"],"developers":["Ping An"],"harmed":["Ping An customers","Chinese minority groups"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/chinese-insurer-ping-an-employed-facial-recognition-to-determine"},{"incident_id":332,"slug":"google-images-showed-racially-biased-results-for-professional-hairstyles","occurred_on":"2016-04-05","title":"Google Images Showed Racially Biased Results for 'Professional' Hairstyles","description":"Google Images reportedly showed disparate search results along racial lines, featuring almost exclusively white women for \"professional hairstyles\" and Black women for \"unprofessional hairstyles\" prompts.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Google"],"developers":["Google"],"harmed":["Women of color","Women and girls","Women","Google users","Google Images users","Black women","Black people"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/google-images-showed-racially-biased-results-for-professional-hairstyles"},{"incident_id":53,"slug":"biased-google-image-results","occurred_on":"2016-03-31","title":"Biased Google Image Results","description":"On June 6, 2016, Google image searches of \"three black teenagers\" resulted in mostly mugshot images whereas Google image searchers of \"three white teenagers\" consisted of mostly stock images, suggesting a racial bias in Google's algorithm.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":[],"deployers":["Google"],"developers":["Google"],"harmed":["Minority Groups"],"report_count":18,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/biased-google-image-results"},{"incident_id":73,"slug":"is-pokemon-go-racist-how-the-app-may-be-redlining-communities-of-color","occurred_on":"2016-03-01","title":"Is Pokémon Go racist? How the app may be redlining communities of color","description":"Through a crowdsourcing social media campaign in 2016, several journalists and researchers demonstrated that augmented reality locations in the popular smartphone game Pokemon Go were more likely to be in white neighborhoods.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication","arts, entertainment and recreation"],"countries":["US"],"deployers":["Niantic Labs"],"developers":["Niantic Labs"],"harmed":["Non White Neighborhoods, Communities Of Color"],"report_count":8,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/is-pokemon-go-racist-how-the-app-may-be-redlining-communities-of-color"},{"incident_id":296,"slug":"twitter-recommender-system-amplified-right-leaning-tweets","occurred_on":"2016-02-10","title":"Twitter Recommender System Amplified Right-Leaning Tweets","description":"Twitter’s “Home” timeline algorithm was revealed by its internal researchers to have amplified tweets and news of rightwing politicians and organizations more than leftwing ones in six out of seven studied countries.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Twitter"],"developers":["Twitter"],"harmed":["Twitter Left Leaning Politicians, Twitter Left Leaning News Organizations, Twitter Left Leaning Users, X (Twitter) Users"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/twitter-recommender-system-amplified-right-leaning-tweets"},{"incident_id":407,"slug":"ubers-surge-pricing-reportedly-offered-disproportionate-service-quality","occurred_on":"2016-02-03","title":"Uber's Surge Pricing Reportedly Offered Disproportionate Service Quality along Racial Lines","description":"Uber's surge-pricing algorithm which adjusts prices to influence car availability inadvertently caused better service offering such as shorter wait times for majority white neighborhoods.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Uber"],"developers":["Uber"],"harmed":["Poor Neighborhoods, Neighborhoods Of Color"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/ubers-surge-pricing-reportedly-offered-disproportionate-service-quality"},{"incident_id":54,"slug":"predictive-policing-biases-of-predpol","occurred_on":"2015-11-18","title":"Predictive Policing Biases of PredPol","description":"Predictive policing algorithms meant to aid law enforcement by predicting future crime show signs of biased output.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["law enforcement"],"countries":["US"],"deployers":["PredPol","Oakland Police Department"],"developers":["PredPol"],"harmed":["Oakland Residents"],"report_count":17,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/predictive-policing-biases-of-predpol"},{"incident_id":16,"slug":"images-of-black-people-labeled-as-gorillas","occurred_on":"2015-06-03","title":"Images of Black People Labeled as Gorillas","description":"Google Photos image processing software mistakenly labelled a black couple as \"gorillas.\"","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["US"],"deployers":["Google"],"developers":["Google"],"harmed":["Black People"],"report_count":23,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/images-of-black-people-labeled-as-gorillas"},{"incident_id":18,"slug":"gender-biases-of-google-image-search","occurred_on":"2015-04-04","title":"Gender Biases of Google Image Search","description":"Google Image returns results that under-represent women in leadership roles, notably with the first photo of a female \"CEO\" being a Barbie doll after 11 rows of male CEOs.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["US"],"deployers":["Google"],"developers":["Google"],"harmed":["Women and girls","Women"],"report_count":10,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/gender-biases-of-google-image-search"},{"incident_id":206,"slug":"tinders-personalized-pricing-algorithm-found-to-offer-higher-prices","occurred_on":"2015-03-01","title":"Tinder's Personalized Pricing Algorithm Found to Offer Higher Prices for Older Users","description":"Tinder’s personalized pricing was found by Consumers International to consider age as a major determinant of pricing, and could be considered a direct discrimination based on age, according to anti-discrimination law experts.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Tinder"],"developers":["Tinder"],"harmed":["Tinder Users Over 30 Years Old"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/tinders-personalized-pricing-algorithm-found-to-offer-higher-prices"},{"incident_id":335,"slug":"uk-visa-streamline-algorithm-allegedly-discriminated-based","occurred_on":"2015-03-01","title":"UK Visa Streamline Algorithm Allegedly Discriminated Based on Nationality","description":"UK Home Office's algorithm to assess visa application risks explicitly considered nationality, allegedly caused candidates to face more scrutiny and discrimination.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Uk Visas And Immigration"],"developers":["Uk Visas And Immigration, Uk Home Office"],"harmed":["Uk Visa Applicants From Some Countries"],"report_count":8,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/uk-visa-streamline-algorithm-allegedly-discriminated-based"},{"incident_id":336,"slug":"uk-home-offices-sham-marriage-detection-algorithm-reportedly-flagged","occurred_on":"2015-03-01","title":"UK Home Office's Sham Marriage Detection Algorithm Reportedly Flagged Certain Nationalities Disproportionately","description":"UK Home Office's opaque algorithm to detect sham marriages flagged some nationalities for investigation more than others, raising fears surrounding discrimination based on nationality and age.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Uk Home Office"],"developers":["Uk Home Office"],"harmed":["Uk Immigrant Newlyweds"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/uk-home-offices-sham-marriage-detection-algorithm-reportedly-flagged"},{"incident_id":260,"slug":"us-dhss-opaque-vetting-software-allegedly-relied-on-poor-quality-data","occurred_on":"2014-08-26","title":"US DHS’s Opaque Vetting Software Allegedly Relied on Poor-Quality Data and Discriminated against Immigrants","description":"US Citizenship and Immigration Services (USCIS)’s ATLAS software used in vetting immigration requests was condemned by advocacy groups as a threat to naturalized citizens for its secretive algorithmic decision-making, reliance on poor quality data and unknown sources, and alleged discrimination of immigrants using biometric and sensitive information.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Us Department Of Homeland Security, Us Citizenship And Immigration Services"],"developers":["Us Citizenship And Immigration Services"],"harmed":["Us Naturalized Citizens, Us Immigrants, Us Citizenship Applicants, Us Immigration Applicants"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/us-dhss-opaque-vetting-software-allegedly-relied-on-poor-quality-data"},{"incident_id":527,"slug":"tech-companies-reportedly-influenced-gig-workers-behaviors-using","occurred_on":"2014-05-08","title":"Tech Companies Reportedly Influenced Gig Workers' Behaviors Using Algorithms to Vary Pay for Same Amount of Work","description":"Amazon and Uber were alleged in a multiyear ethnographic study using algorithmic systems based on gig workers' data to vary pay, such as by offering them lower wages for the same amount of work.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Uber","Amazon"],"developers":["Uber","Amazon"],"harmed":["Uber drivers","gig workers","Amazon delivery workers"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/tech-companies-reportedly-influenced-gig-workers-behaviors-using"},{"incident_id":280,"slug":"coffee-meets-bagels-algorithm-reported-by-users-disproportionately","occurred_on":"2013-07-30","title":"Coffee Meets Bagel’s Algorithm Reported by Users Disproportionately Showing Them Matches of Their Own Ethnicities Despite Selecting “No Preference”","description":"Users selecting “no preference” were shown by Coffee Meets Bagels’s matching algorithm more potential matches with the same ethnicity, which was acknowledged and justified by its founder as a means to maximize connection rate without sufficient user information.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Coffee Meets Bagel"],"developers":["Coffee Meets Bagel"],"harmed":["Coffee Meets Bagel Users Having No Ethnicity Preference, Coffee Meets Bagel Users"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/coffee-meets-bagels-algorithm-reported-by-users-disproportionately"},{"incident_id":19,"slug":"sexist-and-racist-google-adsense-advertisements","occurred_on":"2013-01-23","title":"Sexist and Racist Google Adsense Advertisements","description":"Advertisements chosen by Google Adsense are reported as producing sexist and racist results.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["US"],"deployers":["Google"],"developers":["Google"],"harmed":["Women and girls","Women","Minority groups"],"report_count":27,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/sexist-and-racist-google-adsense-advertisements"},{"incident_id":135,"slug":"ut-austins-grade-algorithm-reportedly-reduced-review-of-lower-scored","occurred_on":"2012-12-01","title":"UT Austin's GRADE Algorithm Reportedly Reduced Review of Lower-Scored PhD Applicants Amid Bias Concerns","description":"From the 2013 through 2019 admissions cycles, UT Austin's Department of Computer Science used GRADE, a statistical machine-learning system trained on past admissions decisions, to score and organize PhD applications. Critics said the system could reproduce historical admissions inequities and reduce attention to lower-scored applicants, while UT Austin said human reviewers still evaluated each file and later discontinued the tool.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["education"],"countries":["US"],"deployers":["University of Texas at Austin's Department of Computer Science","Risto Miikkulainen","Austin Waters"],"developers":["University of Texas at Austin researchers"],"harmed":["University students","University of Texas at Austin PhD applicants of marginalized groups","University applicants","Students","PhD applicants from underrepresented groups","PhD applicants","Educational communities","Computer science PhD applicants"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/ut-austins-grade-algorithm-reportedly-reduced-review-of-lower-scored"},{"incident_id":433,"slug":"chicago-polices-strategic-subject-list-reportedly-biased-along-racial","occurred_on":"2012-08-01","title":"Chicago Police's Strategic Subject List Reportedly Biased Along Racial Lines","description":"The Chicago Police Department reportedly used person-based predictive policing models, including the Strategic Subject List and Crime and Victimization Risk Model, to assign violence-risk scores to hundreds of thousands of residents. Official and media reports alleged that the program gave police a mathematically fragile basis for treating heavily policed residents as future victims or suspects, with Black Chicago residents especially exposed to the consequences of that scoring.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Chicago Police Department"],"developers":["Chicago Police Department"],"harmed":["Minority groups","General public of Chicago","General public","Economically vulnerable communities","communities of color","Black people","Black Chicago residents"],"report_count":9,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/chicago-polices-strategic-subject-list-reportedly-biased-along-racial"},{"incident_id":257,"slug":"police-reportedly-deployed-shotspotter-sensors-disproportionately","occurred_on":"2012-05-04","title":"Police Reportedly Deployed ShotSpotter Sensors Disproportionately in Neighborhoods of Color","description":"Police departments disproportionately placed ShotSpotter sensors in black and brown neighborhoods, which is denounced by communities for allegedly creating dangerous situations, such as one involving in Adam Toledo's death.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Kansas City Police Department, Cleveland Division Of Police, Chicago Police Department, Atlanta Police Department"],"developers":["Shotspotter"],"harmed":["Neighborhoods Of Color, Brown Communities, Black Communities, Adam Toledo"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/police-reportedly-deployed-shotspotter-sensors-disproportionately"},{"incident_id":75,"slug":"google-instants-allegedly-anti-semitic-results-lead-to-lawsuit-in-france","occurred_on":"2012-01-05","title":"Google Instant's Allegedly 'Anti-Semitic' Results Lead To Lawsuit In France","description":"The organizations SOS Racisme, Union of Jewish Students of France, Movement Against Racism and for Friendship Among Peoples are suing Google due to its autocomplete software suggesting \"jewish\" when the names of certain public figures were searched on the platform.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["information and communication"],"countries":["FR"],"deployers":["Google"],"developers":["Google"],"harmed":["Jewish People, Jewish Public Figures"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in France.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/google-instants-allegedly-anti-semitic-results-lead-to-lawsuit-in-france"},{"incident_id":99,"slug":"major-universities-are-reportedly-using-race-as-a-high-impact-predictor","occurred_on":"2012-01-01","title":"Major Universities Are Reportedly Using Race as a 'High Impact Predictor' of Student Success","description":"Several major universities are reportedly using a tool that uses race as one factor to predict student success.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Intentional","timing":"Post-deployment","harm_level":"none","sectors":["administrative and support service activities","education"],"countries":["US"],"deployers":["University of Wisconsin–Milwaukee","University of Massachusetts Amherst","University of Houston","Texas A&M University","more than 500 colleges","Georgia State University"],"developers":["EAB"],"harmed":["University students","Students","Latinx college students","indigenous students","Educational communities","Black college students"],"report_count":1,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/major-universities-are-reportedly-using-race-as-a-high-impact-predictor"},{"incident_id":461,"slug":"irs-audited-black-taxpayers-more-frequently-reportedly-due-to-algorithm","occurred_on":"2008-07-18","title":"IRS Audited Black Taxpayers More Frequently Reportedly Due to Algorithm","description":"The IRS was auditing Black taxpayers more frequently than other groups allegedly due to the design of their algorithms, focusing on easier-to-conduct audits which inadvertently correlated with the group's pattern of tax filing errors.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["public administration","financial and insurance activities"],"countries":["US"],"deployers":["Internal Revenue Service"],"developers":["Internal Revenue Service"],"harmed":["Taxpayers in the United States","Taxpayers","Black taxpayers","Black people"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/irs-audited-black-taxpayers-more-frequently-reportedly-due-to-algorithm"},{"incident_id":15,"slug":"amazon-censors-gay-books","occurred_on":"2008-05-23","title":"Amazon Censors Gay Books","description":"Amazon's book store \"cataloging error\" led to books containing gay and lesbian themes to lose their sales ranking, therefore losing visibility on the sales platform.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"Human","intent":"Other","timing":"Post-deployment","harm_level":"none","sectors":["wholesale and retail trade"],"countries":[],"deployers":["Amazon"],"developers":["Amazon"],"harmed":["Amazon Customers"],"report_count":24,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/amazon-censors-gay-books"},{"incident_id":469,"slug":"automated-adult-content-detection-tools-showed-bias-against-women-bodies","occurred_on":"2006-02-25","title":"Automated Adult Content Detection Tools Showed Bias against Women Bodies","description":"Automated content moderation tools to detect sexual explicitness or \"raciness\" reportedly exhibited bias against women bodies, resulting in suppression of reach despite not breaking platform policies.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Meta, Linkedin, Instagram, Facebook"],"developers":["Microsoft, Google, Amazon"],"harmed":["Linkedin Users, Instagram Users, Facebook Users"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/automated-adult-content-detection-tools-showed-bias-against-women-bodies"},{"incident_id":274,"slug":"virginia-courts-algorithmic-recidivism-risk-assessment-failed-to-lower","occurred_on":"2003-07-01","title":"Virginia Courts’ Algorithmic Recidivism Risk Assessment Failed to Lower Incarceration Rates","description":"Virginia courts’ use of algorithmic predictions of future offending risks were found by researchers failing to reduce incarceration rates, showed racial and age disparities in risk scores and its application, and neither exacerbated or ameliorated historical racial differences in sentencing.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":null,"sectors":[],"countries":[],"deployers":["Virginia courts"],"developers":["Virginia Department of Criminal Justice Services"],"harmed":["Virginia young offenders","Virginia convicted felons","Virginia Black offenders"],"report_count":2,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.","related_policy_slugs":["us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai","us-new-york-city-local-law-144-automated-employment-decision-tools","eu-ai-act"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/virginia-courts-algorithmic-recidivism-risk-assessment-failed-to-lower"},{"incident_id":79,"slug":"kidney-testing-method-allegedly-underestimated-risk-of-black-patients","occurred_on":"1999-03-16","title":"Kidney Testing Method Allegedly Underestimated Risk of Black Patients","description":"Decades-long use of the estimated glomerular filtration rate (eGFR) method to test kidney function which considers race has been criticized by physicians and medical students for its racist history and inaccuracy against Black patients.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["human health and social work activities"],"countries":["US"],"deployers":["Chronic Kidney Disease Epidemiology Collaboration"],"developers":["Chronic Kidney Disease Epidemiology Collaboration"],"harmed":["Black Patients, African American Patients"],"report_count":3,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.","related_policy_slugs":["us-colorado-ai-act","us-new-york-city-local-law-144-automated-employment-decision-tools","us-illinois-artificial-intelligence-video-interview-act","us-illinois-hb-3773-ai-in-employment","us-nist-ai-rmf"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/kidney-testing-method-allegedly-underestimated-risk-of-black-patients"},{"incident_id":43,"slug":"st-georges-hospital-medical-school-allegedly-automated-admissions","occurred_on":"1998-03-05","title":"St George's Hospital Medical School allegedly Automated Admissions with Discriminatory Results","description":"From 1982 to 1986, St George's Hospital Medical School used a program to automate a portion of their admissions process that allegedly discriminated against women and members of ethnic minorities.","mit_domain":"Discrimination and Toxicity","mit_subdomain":"Unfair discrimination and misrepresentation","entity":"AI","intent":"Unintentional","timing":"Post-deployment","harm_level":"none","sectors":["education","human health and social work activities"],"countries":["GB"],"deployers":["St George's Hospital Medical School"],"developers":["Geoffrey Franglen"],"harmed":["Women and girls","Women","Minority groups"],"report_count":4,"harm_domain":"discrimination-toxicity","policy_angle":"Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United Kingdom.","related_policy_slugs":["uk-ai-regulation-white-paper","uk-ico-ai-data-protection-guidance","us-colorado-ai-act","india-dpdp-act","italy-law-132-2025-ai"],"sensitivity":"standard","url":"https://aipolicytracker.org/ai-risk/incidents/st-georges-hospital-medical-school-allegedly-automated-admissions"}]}