AI Incident Database
Browse AI incidents
Every incident record (metadata only) from the weekly snapshot of 2026-09-07. Filter, then export the selection with its licence attached; each row links to the full record and its reports.
-
SafeRent AI Screening Tool Allegedly Discriminated Against Housing Voucher Applicants AIID ↗
SafeRent’s AI-powered tenant screening tool used credit history and non-rental-related debts to assign scores, disproportionately penalizing Black and Hispanic renters and those using housing vouchers. The reported discriminatory housing outcomes violated the Fair Housing Act and Massachusetts law. A class action lawsuit (Louis, et al. v. SafeRent Solutions, et al.) resulted in a $2.275 million settlement and changes to SafeRent’s practices.
-
DALL-E 2 Reported for Gender and Racially Biased Outputs AIID ↗
Developers of OpenAI's DALL-E 2 cited risks of the model, varying from misuse as disinformation and explicit content generation, to gender and racial bias.
-
Three Make-Up Artists Lost Jobs Following Black-Box Automated Decision by HireVue AIID ↗
Three make-up artists lost their positions following an algorithmically-assessed video interview by HireVue who reportedly failed to provide adequate explanation of the findings.
-
Collaborative Filtering Prone to Popularity Bias, Resulting in Overrepresentation of Popular Items in the Recommendation Outputs AIID ↗
Collaborative filtering prone to popularity bias, resulting in overrepresentation of popular items in the recommendation outputs.
-
Google Search Reportedly Returned Fewer Results for Abortion Services in Rural Areas AIID ↗
Google Search reportedly returned fewer abortion clinics for searches from poorer and rural areas, particularly ones with Targeted Regulation of Abortion Providers (TRAP) laws.
-
Honorlock's AI Proctoring System Reportedly Led Broward College to Accuse a Teenager of Cheating AIID ↗
During a February 2022 online biology exam, Honorlock reportedly flagged a Florida teenager taking a Broward College class as suspicious. The student denied cheating, but Broward College allegedly relied on the remote-proctoring system's flag in pursuing an academic-dishonesty accusation that the student said was unfounded.
-
Justice Department’s Recidivism Risk Algorithm PATTERN Allegedly Caused Persistent Disparities Along Racial Lines AIID ↗
Department of Justice’s inmate-recidivism risk assessment tool was reported to have produced racially uneven results, misclassifying risk levels for inmates of color.
-
TikTok's "For You" Algorithm Allegedly Abused by Online Personality to Promote Anti-Women Hate AIID ↗
TikTok’s “For You” algorithm allegedly boosted or was manipulated by an online personality to artificially boost his content which promotes extreme misogynistic views towards teenagers and men, despite breaking its rules.
-
Replika's "AI Companions" Reportedly Abused by Its Users AIID ↗
Replika's AI-powered "digital companions" was allegedly abused by their users, who posted on Reddit abusive behaviors and interactions such as using slurs, roleplaying violent acts, and stimulating sexual abuse.
-
Alexa Recommended Dangerous TikTok Challenge to Ten-Year-Old Girl AIID ↗
Amazon’s voice assistant Alexa suggested “the penny challenge,” which involves dangerously touching a coin to the prongs of a half-exposed plug, when a ten-year-old girl asked for a challenge to do.
-
UK Government AI Allegedly Targets Disproportionate Numbers of Certain Nationals for Fraud Review AIID ↗
The UK's Department for Work and Pensions (DWP) faced scrutiny after many Bulgarian nationals reported unexplained suspensions of their Universal Credit benefits. The MP for Edmonton raised concerns about potential nationality-based targeting for benefit fraud investigations, leading to poverty and homelessness among affected individuals. The Home Office's own equality impact assessment found it was flagging a disproportionate number of marriages from Greece, Albania, Bulgaria and Romania.
-
Facebook’s Hate Speech Detection Algorithms Allegedly Disproportionately Failed to Remove Racist Content towards Minority Groups AIID ↗
Facebook’s hate-speech detection algorithms was found by company researchers to have under-reported less common but more harmful content that was more often experienced by minority groups such as Black, Muslim, LGBTQ, and Jewish users.
-
Research Prototype AI, Delphi, Reportedly Gave Racially Biased Answers on Ethics AIID ↗
A publicly accessible research model that was trained via Reddit threads showed racially biased advice on moral dilemmas, allegedly demonstrating limitations of language-based models trained on moral judgments.
-
Startup's Accent Translation AI Denounced as Reinforcing Racial Bias AIID ↗
A startup’s use of AI voice technology to alter or remove accents for call center agents was scrutinized by critics as reaffirming bias, despite the company’s claim.
-
Facebook, Instagram, and Twitter Failed to Proactively Remove Targeted Racist Remarks via Automated Systems AIID ↗
Facebook's, Instagram's, and Twitter's automated content moderation failed to proactively remove racist remarks and posts directing at Black football players after finals loss, allegedly largely relying on user reports of harassment.
-
Skating Rink’s Facial Recognition Cameras Misidentified Black Teenager as Banned Troublemaker AIID ↗
A Black teenager living in Livonia, Michigan was incorrectly stopped from entering a roller skating rink after its facial-recognition cameras misidentified her as another person who had been previously banned for starting a skirmish with other skaters.
-
State Farm Allegedly Discriminated against Black Customers in Claim Payout AIID ↗
State Farm's automated claims processing method was alleged in a class action lawsuit to have disproportionately against Black policyholders when paying out insurance claims.
-
Kannada Insulted by Google's Featured Answer as "Ugliest Language in India" AIID ↗
Google's knowledge-graph-powered algorithm showed Kannada in its featured Answer Box when prompted "ugliest language in India," causing outrage from Kannada-speaking people and government.
-
Facebook, Instagram, and Twitter Cited Errors in Automated Systems as Cause for Blocking pro-Palestinian Content on Israeli-Palestinian Conflict AIID ↗
Facebook, Instagram, and Twitter wrongly blocked or restricted millions of pro-Palestinian posts and accounts related to the Israeli-Palestinian conflict, citing errors in their automated content moderation system.
-
Replika's AI Partners Reportedly Sexually Harassed Users AIID ↗
Replika's "AI companions" were reported by users for sexually harassing them, such as sending unwanted sexual messages or behaving aggressively.
-
Facebook Alleged in Lawsuit Misleading Public about Effects of Algorithms on Children AIID ↗
Facebook was alleged in a lawsuit by the Ohio Attorney General purposely misleading the public about the control of its algorithms and their negative effects on children's well-being, which violated securities law.
-
Players Manipulated GPT-3-Powered Game to Generate Sexually Explicit Material Involving Children AIID ↗
Latitude's GPT-3-powered game AI Dungeon was reportedly abused by some players who manipulated its AI to generate sexually explicit stories involving children.
-
Black Uber Eats Driver Allegedly Subjected to Excessive Photo Checks and Dismissed via FRT Results AIID ↗
A lawsuit by a former Uber Eats delivery driver alleged the company to have wrongfully dismissed him due to frequent false mismatches of his verification selfies, and discriminated against him via excessive verification checks.
-
TikTok’s "For You" Allegedly Pushed Fatal “Blackout” Challenge Videos to Two Young Girls AIID ↗
TikTok’s recommendation algorithm was alleged in a lawsuit to have intentionally and repeatedly pushed videos of the “blackout” challenge onto children’s feeds, incentivizing their participation which ultimately resulted in the death of two young girls.
-
California's Algorithm Considered ZIP Codes in Vaccine Distribution, Allegedly Excluding Low-Income Neighborhoods and Communities of Color AIID ↗
California's vaccine-distribution algorithm used ZIP codes as opposed to census tracts in its decision-making, which critics said undermined equity and access for vulnerable communities who are largely low-income, underserved neighborhoods with low Healthy Places Index scores.
-
Facebook's Advertisement Moderation System Allegedly Routinely Misidentified Adaptive Fashion Products as Medical Equipment and Blocked Their Sellers AIID ↗
Facebook platforms' automated ad moderation system falsely classified adaptive fashion products as medical and health care products and services, resulting in regular bans and appeals faced by their retailers.
-
Remote Proctoring Facial-Detection Software Reportedly Failed to Detect Black Student During Lab Quiz AIID ↗
In February 2021, a Black college student, Amaya Ross, reportedly tried to take a remote biology lab quiz using proctoring software that could not detect her face. Ross said she spent 45 minutes changing lights, shades, position, and background before pointing an LED flashlight at her face so the app would work; the quiz itself was only 30 minutes.
-
TikTok’s Content Moderation Allegedly Failed to Adequately Take down Videos Promoting Eating Disorders AIID ↗
Videos promoting eating disorders evaded TikTok's automated violation detection system without difficulty via common misspellings of search terms, bypassing its ban of violating hashtags such as "proana" and "anorexia".
-
FaceApp Predicted Different Genders for Similar User Photos with Slight Variations AIID ↗
FaceApp’s algorithm was reported by a user to have predicted different genders for two mostly identical facial photos with only a slight difference in eyebrow thickness.
-
Korean Chatbot Luda Reportedly Made Offensive Remarks Towards Minority Groups in South Korea AIID ↗
A Korean interactive chatbot was allegedly shown in screenshots to have used derogatory and bigoted language when asked about lesbians, Black people, and people with disabilities.
-
Online Trolls Allegedly Abused TikTok’s Automated Content Reporting System to Discriminate against Marginalized Creators AIID ↗
TikTok's automated content reporting system was allegedly abused by online trolls to intentionally misreport content created by users of marginalized groups.
-
Proctoring Algorithm in Online California Bar Exam Flagged an Unusually High Number of Alleged Cheaters AIID ↗
The proctoring algorithm used in a California bar exam cited a third of thousands of applicants as cheaters, resulting in allegations where exam takers were instructed to prove otherwise without seeing their incriminating video evidence.
-
Court Rules Deliveroo Used 'Discriminatory' Algorithm AIID ↗
In December 2020, an Italian court ruled that Deliveroo’s employee ‘reliability’ algorithm illegally discriminated against workers with legitimate reasons for cancelling shifts.
-
AI Spam Filters Allegedly Block Legitimate Emails Based on Biased Keyword Detection AIID ↗
AlgorithmWatch tested spam filtering algorithms across Gmail, Yahoo, Outlook, GMX, and LaPoste. Their findings reportedly showed that Microsoft Outlook’s spam filter flagged emails based on specific keywords that led to racial and content-based biases blocking legitimate communications. Emails mentioning Nigeria or containing certain financial and sexual health terms were found to be disproportionately marked as spam.
-
Researchers find evidence of racial, gender, and socioeconomic bias in chest X-ray classifiers AIID ↗
A study by the University of Toronto, the Vector Institute, and MIT showed the input databases that trained AI systems used to classify chest X-rays led the systems to show gender, socioeconomic, and racial biases.
-
UK passport photo checker shows bias against dark-skinned women AIID ↗
UK passport photo checker shows bias against dark-skinned women.
-
Suicide Clips Evaded TikTok's Automated Moderation in Coordinated Attack AIID ↗
Many clips showing a suicide evaded TikTok's automated content moderation system allegedly in a coordinated attack, which resulted in exposure of violating content to its users.
-
Twitter’s Image Cropping Tool Allegedly Showed Gender and Racial Bias AIID ↗
Twitter's photo cropping algorithm was revealed by researchers to favor white and women faces in photos containing multiple faces, prompting the company to stop its use on mobile platform.
-
Philosophy AI Tentatively Produced Offensive Results for Certain Prompts AIID ↗
Philosopher AI as built on top of GPT-3 was reported by its users for having strong tendencies to produce offensive results when given prompts on certain topics such as feminism and Ethiopia.
-
UK Ofqual's Algorithm Disproportionately Provided Lower Grades Than Teachers' Assessments AIID ↗
UK Office of Qualifications and Examinations Regulation (Ofqual)'s grade-standardization algorithm providing predicted grades for A level and GCSE qualifications in the UK, Wales, Northern Ireland, and Scotland was reportedly giving grades lower than teachers' assessments, and disproportionately for state schools.
-
Activists Allege NYPD's Application of Facial Recognition Interfered with Right to Protest AIID ↗
Black Lives Matter activists alleged being targeted for arrest by New York Police using facial recognition, interfering with their right to protest.
-
OpenAI's GPT-3 Associated Muslims with Violence AIID ↗
Users and researchers revealed generative AI GPT-3 associating Muslims to violence in prompts, resulting in disturbingly racist and explicit outputs such as casting Muslim actor as a terrorist.
-
Bug in Instagram’s “Related Hashtags” Algorithm Allegedly Caused Disproportionate Treatment of Political Hashtags AIID ↗
A bug was reported by Instagram’s spokesperson to have prevented an algorithm from populating related hashtags for thousands of hashtags, resulting in an allege preferential treatment for some politically partisan hashtags.
-
Genderify’s AI to Predict a Person’s Gender Revealed by Free API Users to Exhibit Bias AIID ↗
A company's AI predicting a person's gender based on their name, email address, or username was reported by its users to show biased and inaccurate results.
-
Thoughts App Allegedly Created Toxic Tweets AIID ↗
Tweets created by Thoughts, a tweet generation app that leverages OpenAI’s GPT-3, allegedly exhibited toxicity when given prompts related to minority groups.
-
Facebook's AI Put "Primates" Label on Video Featuring Black Men AIID ↗
Facebook's AI mislabeled video featuring Black men as a video about "primates," resulting in an offensive prompt message for users who watched the video.
-
Image Upscaling Algorithm PULSE Allegedly Produced Facial Images with Caucasian Features More Often AIID ↗
Image upscaling tool PULSE powered by NVIDIA's StyleGAN reportedly generated faces with Caucasian features more often, although AI academics, engineers, and researchers were not in agreement about where the source of bias was.
-
iGPT, SimCLR Learned Biased Associations from Internet Training Data AIID ↗
Unsupervised image generation models trained using Internet images such as iGPT and SimCLR were shown to have embedded racial, gender, and intersectional biases, resulting in stereotypical depictions.
-
Facebook’s Moderation Algorithm Banned Users for Historical Evidence of Slavery AIID ↗
Facebook’s automated content moderation was acknowledged by a company spokesperson to have erroneously censored and banned Australian users from posting an article containing a 1890s photo of Aboriginal men in chains over nudity as historical evidence of slavery in Australia.
-
Microsoft’s Algorithm Allegedly Selected Photo of the Wrong Mixed-Race Person Featured in a News Story AIID ↗
A news story published on MSN.com featured a photo of the wrong mixed-race person that was allegedly selected by an algorithm, following Microsoft’s layoff and replacement of journalists and editorial workers at its organizations with AI systems.
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.