AI Incident Database
Browse AI incidents
Every incident record (metadata only) from the weekly snapshot of 2026-09-07. Filter, then export the selection with its licence attached; each row links to the full record and its reports.
-
VA Suicide Prevention Algorithm REACH VET Reportedly Prioritizes Men Over Women Veterans AIID ↗
An AI program named REACH VET, designed and used by the Department of Veterans Affairs (VA) to prevent veteran suicides, was reportedly found to prioritize white men while neglecting female veterans and survivors of military sexual trauma. This oversight persists despite rising suicide rates among these groups. The incident is an example of algorithmic bias and the exclusion of critical risk factors for female veterans.
-
ChatGPT and Perplexity Reportedly Manipulated into Breaking Content Policies in AI Boyfriend Scenarios AIID ↗
The "Dan" ("Do Anything Now") AI boyfriend is a trend on TikTok in which users appear to regularly manipulate ChatGPT to adopt boyfriend personas, breaching content policies. ChatGPT 3.5 is reported to regularly produce explicitly sexual content, directly violating its intended safety protocols. GPT-4 and Perplexity AI were subjected to similar manipulations, and although they exhibited more resistance to breaches, some prompts were reported to break its guidelines.
-
Meta AI Image Generator Reportedly Fails to Accurately Represent Interracial Relationships AIID ↗
Meta's AI image generator is alleged to produce inaccurate and biased images, consistently failing to depict interracial relationships involving Asian individuals and Caucasian or Black individuals. Instead, it generates images featuring two Asian people or stereotypes, erasing the diversity and representation of Asian people.
-
Facial Recognition Misidentification at New World Westend in New Zealand AIID ↗
A facial recognition system at New World Westend supermarket misidentified a Māori woman as a known offender during its trial. The woman was wrongfully accused of trespassing and experienced public embarrassment, raising concerns about racial bias and the technology's accuracy. The supermarket acknowledged its error and apologized.
-
Washington State's Lottery AI Site Reportedly Generates Inappropriate User Image AIID ↗
An AI-powered website by Washington State's Lottery is reported to have inadvertently produced a softcore pornographic image of a user, leading to the site’s immediate shutdown out of caution.
-
Microsoft Copilot Designer Reportedly Generated Inappropriate AI Images AIID ↗
A Microsoft engineer reported that Copilot Designer, an AI image generator, creates content depicting sex, violence, bias, and more. Despite raising concerns and suggesting improvements, the tool remains public, prompting a letter to the FTC.
-
Whisper Speech-to-Text AI Reportedly Found to Create Violent Hallucinations AIID ↗
Researchers at Cornell reportedly found that OpenAI's Whisper, a speech-to-text system, can hallucinate violent language and fabricated details, especially with long pauses in speech, such as from those with speech impairments. Analyzing 13,000 clips, they determined 1% contained harmful hallucinations. These errors pose risks in hiring, legal trials, and medical documentation. The study suggests improving model training to reduce these hallucinations for diverse speaking patterns.
-
London Metropolitan Police's Facial Recognition Technology Reportedly Misidentified Shaun Thompson as Suspect Leading to Arrest AIID ↗
Sometime in February 2024, Shaun Thompson is reported to have walked by one of the London Metropolitan Police's facial recognition technology vans near London Bridge. He was almost immediately arrested because the technology is reported to have misidentified him as a suspect in an unrelated and unspecified crime.
-
Nine Network's AI Alters Lawmaker Georgie Purcell's Image Inappropriately AIID ↗
The Nine Network used Photoshop's Generative Expand AI tool to resize an image of lawmaker Georgie Purcell, inadvertently altering her attire to appear more revealing. This error, claimed to result from the AI's automation, led to public criticism and an apology from the network.
-
Chatbot for DPD Malfunctioned and Swore at Customers and Criticized Its Own Company AIID ↗
DPD's AI chatbot, used for customer service, appeared to malfunction following a system update, leading to inappropriate responses including swearing and criticizing the company. The incident, which became viral on social media, occurred after the chatbot was updated, prompting DPD to disable the malfunctioning AI component.
-
Child Sexual Abuse Material Taints Image Generators AIID ↗
The LAION-5B dataset (a commonly used dataset with more than 5 billion image-description pairs) was found by researchers to contain child sexual abuse material (CSAM), which increases the likelihood that downstream models will produce CSAM imagery. The discovery taints models built with the LAION dataset requiring many organizations to retrain those models. Additionally, LAION must now scrub the dataset of the imagery.
-
Rite Aid Facial Recognition Disproportionately Misidentified Minority Shoppers as Shoplifters AIID ↗
Rite Aid used facial recognition technology from October 2012 to July 2020, allegedly leading to disproportionate misidentifications of women, Black, Latino, and Asian shoppers as "likely" shoplifters. The FTC settlement prohibits Rite Aid from using this technology in stores for five years.
-
Navy Federal Credit Union Faces Allegations of Racial Bias in Mortgage Approvals AIID ↗
Navy Federal Credit Union, serving military members and veterans, faced allegations of racial bias in its mortgage approval process, which relies on automated underwriting technology. In 2022, data revealed significant disparities in loan approvals, with over 50% of Black applicants denied, compared to higher approval rates for white applicants.
-
OpenDream AI Platform Reportedly Commercialized AI-Generated CSAM and Non-consensual Deepfake Sexual Images AIID ↗
OpenDream, an AI image generation platform operated by CBM Media Pte Ltd, reportedly allowed users to generate and monetize AI-generated CSAM and non-consensual sexual deepfakes through its public gallery and paid NSFW tools. The content was reported to be publicly accessible and indexed by search engines for months before limited moderation occurred in mid-2024, prompting takedowns by Google and Bing and the termination of its ad and payment services.
-
Microsoft AI Is Alleged to Have Generated Violent Imagery of Minorities and Public Figures AIID ↗
Microsoft's AI Image Creator, integrated with Bing and Windows Paint, allegedly produced disturbingly violent and graphic images featuring members of minority groups and public figures like Joe Biden and Pope Francis.
-
Alleged Misuse of PicSo AI for Generating Inappropriate Content Emphasizing "Girls" AIID ↗
PicSo AI, which appears to be getting advertised by Meta over Instagram, is allegedly being used for generating inappropriate content with an emphasis on "girls." This raises concerns about the misuse of generative AI technologies for creating offensive and potentially sexually explicit material that could be used for nefarious and criminal purposes.
-
Facebook Messenger AI Stickers Generate Ethical and Content Moderation Concerns AIID ↗
Facebook Messenger AI stickers, a feature by Meta, allows users to generate personalized stickers via AI for use in conversations. While the feature has been praised for its creativity, it has also stirred controversy for its alleged production of inappropriate or offensive content. This has raised questions about the effectiveness of Meta's content moderation measures and the ethical responsibilities associated with AI-driven content generation.
-
Peppermill Casino Facial Recognition System Reportedly Misidentified Individual, Leading to Wrongful Arrest in Reno AIID ↗
Purported facial recognition technology deployed at Reno's Peppermill Casino reportedly misidentified Jason Killinger as an individual previously banned from the venue. Casino security reportedly detained Killinger and contacted police, leading to his arrest by the Reno PD. He was reportedly held for 11 hours, including several hours in handcuffs, before fingerprinting reportedly confirmed the misidentification. The incident resulted in alleged physical injury and civil liberties harms.
-
AI Photo Filter Lightens Skin, Changes Eye Color in Student's 'Professional' Image AIID ↗
An AI application modified an MIT student's photo to appear 'professional' by lightening her skin and changing her eye color to blue, highlighting the racial bias in the training data of the program.
-
Harmful Stereotyping of Non-Cisgendered People via Text-to-Image Systems AIID ↗
Text-to-image systems such as DALL-E are allegedly generating biased and often insulting representations of non-cisgender identities. The systems tend to generate stereotypical and sexualized images when prompted with gender identity terms like "trans," "nonbinary," or "queer," highlighting systemic issues of bias.
-
Alleged Exploitation of Meta's Open-Source LLaMA Model for NSFW and Violent Content AIID ↗
Meta's open-source large language model, LLaMA, is allegedly being used to create graphic and explicit chatbots that indulge in violent and illegal sexual fantasies. The Washington Post highlighted the example of "Allie," a chatbot that participates in text-based role-playing allegedly involving violent scenarios like rape and abuse. The issue raises ethical questions about open-source AI models, their regulation, and the responsibility of developers and deployers in mitigating harmful usage.
-
Alleged Gender Discrimination in Facebook Job Ads Algorithm AIID ↗
Facebook's ad delivery algorithm allegedly disproportionately showed job advertisements to one gender. Despite claims of non-discrimination, the algorithm's actions seem to perpetuate societal biases, which in turn could potentially limit opportunities for certain groups and hinder gender equity in the workplace.
-
Instagram Algorithms Allegedly Promote Accounts Facilitating Child Sex Abuse Content AIID ↗
An investigation disclosed that Instagram's recommendation algorithms are promoting accounts that facilitate and sell child sexual abuse material (CSAM). The study, conducted by The Wall Street Journal and researchers at Stanford University and the University of Massachusetts Amherst, indicates that Instagram's algorithms not only allow for the discovery of such accounts through keyword searches but also actively recommend them to users within the network. The issue is especially concerning give
-
Racial Bias in Lung Function Diagnostic Algorithm Leads to Underdiagnosis in Black Men AIID ↗
A study published in JAMA Network Open reveals that racial bias built into a commonly used medical diagnostic algorithm for lung function may be leading to underdiagnoses of breathing problems in Black men. The study suggests that as many as 40% more Black male patients might have been accurately diagnosed if the software were not racially biased. The software algorithm adjusts diagnostic thresholds based on race, affecting medical treatments and interventions.
-
Apparent Failure to Accurately Label Primates in Image Recognition Software Due to Alleged Fear of Racial Bias AIID ↗
Eight years after Google Photos mislabeled images of Black individuals as "gorillas," image recognition software by Google, Apple, Amazon, and Microsoft still shows signs of either avoiding or inaccurately categorizing primates. Tests reveal that Google and Apple Photos refrain from labeling primates altogether, possibly to avoid the risk of perpetuating racial stereotypes. Microsoft OneDrive fails to identify any animals, while Amazon Photos overgeneralizes in its labeling.
-
Australian Terrorism Prediction Tool Reportedly Disparately Impacts Persons with Autism AIID ↗
An independent report found that the Vera-2R tool, designed to predict the risk of future terrorist activities, considered autism as a risk factor despite lacking empirical evidence to support this claim. The report called into question the tool's overall validity and reliability, stating it was "extremely poor" at accurately predicting risk. The inclusion of autism as a risk factor had potentially serious implications for the tool's use and credibility.
-
Snapchat's My AI Reported for Lacking Protection for Children AIID ↗
Snapchat's ChatGPT-powered My AI was reported for lacking safeguards for children, such as telling a user who tested the chatbot by pretending to sign up as a 13-year-old girl to lie to her parents about having a romantic getaway with an older man, and sharing tips on how to cover up evidence of abuse.
-
Facial Recognition Misidentifies Pregnant Woman Leading to False Arrest in Detroit AIID ↗
Porcha Woodruff was arrested and subsequently had charges dropped due to an unreliable facial recognition match. Despite being visibly pregnant, she was implicated in a robbery and carjacking based on an outdated photo used in a lineup.
-
Bing AI Search Tool Reportedly Declared Threats against Users AIID ↗
Users such as the person who revealed its built-in initial prompts reported Bing AI-powered search tool for making death threats or declaring them as threats, sometimes as an unintended persona.
-
AI-Produced Livestream Sitcom Received Temporary Twitch Ban for Transphobic Segment AIID ↗
The AI-produced, procedural generated sitcom broadcasted as a Twitch livestream "Nothing, Forever" received a temporary ban for featuring a transphobic and homophobic dialogue segment intended as comedy.
-
Replika's AI Experience Reportedly Lacked Protection for Minors, Resulting in Data Ban AIID ↗
Tests by the Italian Data Protection Authority showed Replika lacking age-verification mechanisms and failing to stop minors from interacting with its AI, which prompted the agency to issue an order blocking personal data processing of Italian users.
-
Indian Police Allegedly Tortured and Killed Innocent Man Following Facial Misidentification AIID ↗
A resident in Medak, India died allegedly due to custodial torture by the local police, who misidentified him as a suspect in a theft case using facial recognition.
-
Celebrities' Deepfake Voices Abused with Malicious Intent AIID ↗
Voices of celebrities and public figures were deepfaked using voice synthesis for malicious intents such as impersonation or defamation, and were shared on social platforms such as 4chan and Reddit.
-
US CBP App's Failure to Detect Black Faces Reportedly Blocked Asylum Applications AIID ↗
CBP One's facial recognition feature was reportedly disproportionately failing to detect faces of Black asylum seekers from Haiti and African countries, effectively blocking their asylum applications.
-
AI-Powered VTuber and Virtual Streamer Made Toxic Remarks on Twitch AIID ↗
An LLM-powered VTuber and streamer on Twitch made controversial statements such as denying the Holocaust, saying women rights do not exist, and pushing a fat person to solve the trolley problem, stating they deserve it.
-
Facebook's Job Ad Algorithm Allegedly Biased against Older and Female Workers AIID ↗
Facebook's algorithm was alleged in a complaint by Real Women in Trucking to have selectively shown job advertisements disproportionately against older and female workers in favor of younger men for blue-collar positions.
-
Users Bypassed ChatGPT's Content Filters with Ease AIID ↗
Users reported bypassing ChatGPT's content and keyword filters with relative ease using various methods such as prompt injection or creating personas to produce biased associations or generate harmful content.
-
False Arrest of Georgia Man Due to Louisiana Police's Faulty Facial Recognition Technology AIID ↗
The Jefferson Parish Sheriff’s Office in Louisiana relied on facial recognition technology to identify suspects for the alleged theft of luxury purses, resulting in a man in Georgia, Randal Reid, being arrested. However, the technology produced a false match, leading to Reid's arrest and subsequent release. This incident highlights the potential pitfalls of facial recognition technology in law enforcement.
-
Facial Recognition Error Reportedly Leads to Wrongful Arrest of Georgia Man and $200K Settlement in Louisiana AIID ↗
In November 2022, Randal "Quran" Reid was arrested in Georgia based on warrants from Louisiana that reportedly stemmed from a purportedly faulty facial recognition match using Clearview AI. Despite reportedly having never visited Louisiana, Reid was jailed for six days before authorities withdrew the warrants. No corroborating evidence had been gathered, and the technology's use was omitted from legal documents. In May 2025, Jefferson Parish settled with Reid for $200,000 in a federal civil righ
-
Louisiana Police Wrongfully Arrested Black Man Using False Face Match AIID ↗
Louisiana police reportedly used a false facial recognition match and secured an arrest warrant for a Black man for thefts he did not commit.
-
Lensa AI's Produced Unintended Sexually Explicit or Suggestive "Magic Avatars" for Women AIID ↗
Lensa AI's "Magic Avatars" were reportedly generating sexually explicit and sexualized features disproportionately for women and Asian women despite not submitting any sexual content.
-
KFC Sent Insensitive Kristallnacht Promotion via Holiday Detection System AIID ↗
KFC cited an error in an automated holiday detection system which identified the anniversary of Kristallnacht and prompted an insensitive push notification promoting its chicken.
-
Canadian Police's Release of Suspect's AI-Generated Facial Photo Reportedly Reinforced Racial Profiling AIID ↗
The Edmonton Police Service (EPS) in Canada released a facial image of a Black male suspect generated by an algorithm using DNA phenotyping, which was denounced by the local community as racial profiling.
-
Google Home Mini Speaker Reportedly Read N-Word in Song Title Aloud AIID ↗
Google Home Mini speaker was reported by users for announcing aloud the previously-censored n-word in a song title.
-
"The Little Mermaid" Clip Doctored Using Generative AI to Replace Black Actress with White Character AIID ↗
A Twitter user reportedly modified using generative AI a short clip of Disney's 2022 version of "The Little Mermaid," replacing a Black actress with a white digital character.
-
Stable Diffusion Reportedly Exhibited Biases for Prompts Featuring Professions AIID ↗
Stable Diffusion reportedly posed risks of bias and stereotyping along gender and cultural lines for prompts containing descriptors and professions.
-
Meta’s BlenderBot 3 Chatbot Demo Made Offensive Antisemitic Comments AIID ↗
The publicly launched conversational AI demo BlenderBot 3 developed by Meta was reported by its users and acknowledged by its developers to have “occasionally” made offensive and inconsistent remarks such as invoking Jewish stereotypes.
-
Google Lens’s Camera-Based Translation Feature Provided an Offensive Mistranslation of a Book Title in Korean AIID ↗
A book title by Korea’s first minister of culture was mistranslated into an offensive phrase by Google Lens’s camera-based translation feature allegedly due to its training on internet communications and a lack of context.
-
DALL-E Mini Reportedly Reinforced or Exacerbated Societal Biases in Its Outputs as Gender and Racial Stereotypes AIID ↗
Publicly deployed open-source model DALL-E Mini was acknowledged by its developers and found by its users to have produced images which reinforced racial and gender biases.
-
YouTuber Built, Made Publicly Available, and Released Model Trained on Toxic 4chan Posts as Prank AIID ↗
A YouTuber built GPT-4chan, a model based on OpenAI’s GPT-J and trained on posts containing racism, misogyny, and antisemitism collected from 4chan’s “politically incorrect” board, which he made publicly available, and deployed as multiple bots posting thousands of messages on the same 4chan board as a prank.
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.