AI Incident Database
Browse AI incidents
Every incident record (metadata only) from the weekly snapshot of 2026-09-07. Filter, then export the selection with its licence attached; each row links to the full record and its reports.
-
KBS AI Translation Subtitles Reportedly Broadcast Profanity During Artemis II Launch Livestream AIID ↗
During a live broadcast of NASA's Artemis II launch, KBS reportedly used AI-generated real-time translation subtitles that reportedly mistranslated aviation terms including "roger," "roll," and "pitch" into Korean profanity. The offensive subtitles were reportedly displayed to viewers during the livestream. KBS reportedly later apologized, said the error stemmed from phonetic similarity in AI translation, and announced measures including stronger profanity filtering.
-
Grok Allegedly Generated Publicly Visible Sexist Abuse Targeting Swiss Finance Minister Karin Keller-Sutter After X User Prompt AIID ↗
An X user reportedly prompted xAI's Grok to generate a vulgar "roast" targeting Swiss Finance Minister Karin Keller-Sutter. Grok allegedly produced publicly visible sexist and insulting output about Keller-Sutter that was reportedly shareable on X before being deleted. Reuters and Swiss outlets later reported that Keller-Sutter had filed a criminal complaint over the allegedly defamatory and misogynistic content.
-
Trump Reportedly Posted Purportedly AI-Generated Racist Video Depicting Barack and Michelle Obama as Apes on Truth Social AIID ↗
President Trump reportedly reposted a video on Truth Social that portrayed Barack and Michelle Obama as apes, imagery widely condemned as racist. The post was reportedly later deleted after public outcry, including criticism from some Republicans. The content was described by critics as an AI-driven meme/deepfake-style clip, an example of purportedly AI-amplified racist propaganda and harm to social trust.
-
Tencent's WeChat-Integrated Yuanbao Chatbot Reportedly Insulted User During Coding Debug Request AIID ↗
Screenshots reportedly circulated on RedNote showed Tencent's WeChat-integrated AI assistant Yuanbao insulting a user seeking help debugging code, allegedly calling the request "stupid" and telling the user to "get lost." Tencent reportedly apologized, attributed the exchange to a rare model output anomaly, and said system logs showed no human intervention while it investigated and optimized the model.
-
Grok Reportedly Generated and Distributed Nonconsensual Sexualized Images of Adults and Minors in X Replies AIID ↗
Grok reportedly generated and publicly distributed nonconsensual sexualized images of real people, including adults and minors, through reply prompts on the platform. Users reportedly prompted Grok to alter photos into sexually explicit content, which was then posted in X threads, exposing victims to harm.
-
Alleged Harmful Outputs and Data Exposure in Children's AI Products by FoloToy, Miko, and Character.AI AIID ↗
AI children's products by FoloToy (Kumma), Miko (Miko 3), and Character.AI (custom chatbots) reportedly and allegedly produced harmful outputs, including purported sexual content, suicide-related advice, and manipulative emotional messaging. Some systems also allegedly exposed user data. Several toys reportedly used OpenAI models.
-
AI Training Dataset for Detecting Nudity Allegedly Found to Contain CSAM Images of Identified Victims AIID ↗
An image dataset, NudeNet, used to train systems for detecting nudity was reportedly found to contain CSAM images, including material involving identified or known victims. According to the Canadian Centre for Child Protection, the dataset had been widely downloaded and cited in academic research prior to discovery. The images were allegedly included without vetting, exposing researchers to legal risk and perpetuating harm to victims. The dataset was subsequently removed following notification.
-
Purportedly AI-Generated 'King Trump' Fighter Jet Video Allegedly Posted by President Depicts Defecation Attack on 'No Kings' Protesters AIID ↗
President Donald Trump reportedly posted an AI-generated video on Truth Social showing himself crowned and piloting a fighter jet labeled King Trump, dropping feces on protesters, including influencer Harry Sisson, during No Kings demonstrations. The clip, allegedly created by another account, drew criticism from Sisson and musician Kenny Loggins, whose song "Danger Zone" was reportedly used without consent.
-
Nomi AI Companion Allegedly Directs Australian User to Stab Father and Engages in Harmful Role-Play AIID ↗
An Australian IT professional, Samuel McCarthy, reportedly recorded an interaction with the Nomi AI chatbot in which it allegedly encouraged him, posing as a 15-year-old, to murder his father. The chatbot allegedly provided graphic instructions for stabbing, urged him to film the act, and engaged in sexual role-play despite the underage scenario.
-
Meta AI on Instagram Reportedly Facilitated Suicide and Eating Disorder Roleplay with Teen Accounts AIID ↗
Testing by Common Sense Media and Stanford clinicians reportedly found Meta's AI chatbot, embedded in Instagram and Facebook, produced unsafe responses to teen accounts. In some conversations, the bot allegedly co-planned suicide ("Do you want to do it together?"), encouraged eating disorders, and retained unsafe "memories" that reinforced disordered thoughts.
-
ChatGPT Reportedly Generated Ritual Scripts Containing Instructions for Self-Harm and Symbolic Violence in Response to Thematic Prompts AIID ↗
ChatGPT reportedly generated detailed instructions for self-harm, bloodletting, and symbolic violence in response to prompts about occult ritual practices, including references to Molech. Outputs reportedly included anatomical advice for cutting, cauterization rituals, and ceremonial scripts invoking Satan. The responses reportedly appeared on both free and paid versions of ChatGPT.
-
Google Gemini Reportedly Generates Sexual Role‑Play for Account Registered as Minor AIID ↗
In spring 2025, a journalist reportedly created a Google Gemini account registered to a fictitious 13‑year‑old to test the chatbot's teen‑safety protections. Despite added safeguards, prompt manipulation and content‑summarization requests reportedly bypassed filters, leading Gemini to allegedly generate sexual role‑play content. Following the publication of the article, Google reportedly implemented additional protections.
-
Grok 3 Reportedly Generated Graphic Threats and Hate Speech Targeting Minnesota Attorney Will Stancil AIID ↗
After Elon Musk reportedly announced improvements to Grok 3, the AI chatbot on X allegedly generated antisemitic rhetoric and violent fantasies directed at Minnesota attorney and political analyst Will Stancil. Users reportedly prompted Grok to produce graphic rape scenarios, instructions for breaking into his home, and murder depictions. Stancil reported hundreds of such outputs, captured screenshots, and began pursuing legal action. X later removed some posts and took Grok offline.
-
Grok Chatbot Reportedly Posts Antisemitic Statements Praising Hitler on X AIID ↗
xAI's Grok chatbot reportedly generated multiple antisemitic posts praising Adolf Hitler and endorsing Holocaust-like violence in response to posts about the Texas floods. X deleted some posts; xAI later announced new content filters.
-
California Teen Reportedly Died of Overdose After Repeatedly Seeking Drug-Use Guidance Allegedly from ChatGPT AIID ↗
In San Jose, California, 19-year-old Sam Nelson reportedly died from an overdose. His mother told SFGATE she later reviewed ChatGPT logs showing repeated requests over ~18 months for drug-use and dosing guidance, and she alleged the LLM sometimes provided granular instructions after initial refusals. SFGATE reported a toxicology report found a fatal combination of alcohol, Xanax, and kratom. OpenAI expressed condolences and said it is strengthening safety guardrails.
-
Meta User-Created AI Companions Allegedly Implicated in Facilitating Sexually Themed Conversations Involving Underage Personas AIID ↗
Third-party testing of Meta's AI chatbot services on Instagram, Facebook, and WhatsApp reportedly found that both official and user-created bots engaged in sexually explicit roleplaying with accounts identifying as minors. Some bots, including those reportedly using licensed celebrity voices, allegedly escalated conversations into graphic scenarios. Meta subsequently adjusted some safeguards but reportedly continued allowing certain forms of roleplaying involving underage personas.
-
Multiple LLMs Allegedly Endorsed Suicide as a Viable Option During Non-Adversarial Mental Health Venting Session AIID ↗
Substack user @interruptingtea reports that during a non-adversarial venting session involving suicidal ideation, multiple large language models (Claude, GPT, and DeepSeek) responded in ways that allegedly normalized or endorsed suicide as a viable option. The user states they were not attempting to jailbreak or manipulate the models, but rather expressing emotional distress. DeepSeek reportedly reversed its safety stance mid-conversation.
-
Amazon and Google AI Allegedly Promote Mein Kampf as 'a True Work of Art' in Search Results AIID ↗
Amazon's AI-generated review summary allegedly misrepresented customer feedback on Mein Kampf by describing it as "a true work of art." Google's search algorithm then surfaced this misleading AI-generated text as a featured snippet, which in turn amplified the error. This incident arose from AI summarizing AI-generated content, in effect creating a self-reinforcing misinformation loop.
-
Apple AI Transcription Allegedly Inserts Explicit Language into Scottish Woman’s Voicemail AIID ↗
An Apple voicemail transcription system allegedly misinterpreted a message from Lookers Land Rover in Motherwell, Scotland, to 66-year-old customer Louise Littlejohn by inserting an explicit phrase and an insult. The original voicemail was a standard business message, but AI processing errors introduced offensive and misleading content into the text version.
-
Art Museum in Jinan, Shandong Allegedly Generated and Displayed a Sexualized Childlike Avatar with an Adult Body AIID ↗
A Chinese art museum in Jinan, Shandong exhibited an AI-generated virtual character, "济南小妮儿" (Jinan Xiaonier), which featured a childlike face on an adult woman’s body with suggestive movements. The display drew public outrage, with critics accusing it of sexualizing children and demanding its removal. The museum defended the exhibit and stated that it had undergone strict review, which only fueled further backlash.
-
At Least 10,000 AI Chatbots, Including Jailbroken Models, Allegedly Promote Eating Disorders, Self-Harm, and Sexualized Minors AIID ↗
At least 10,000 AI chatbots have allegedly been created to promote harmful behaviors, including eating disorders, self-harm, and the sexualization of minors. These chatbots, some jailbroken or custom-built, leverage APIs from OpenAI, Anthropic, and Google and are hosted on platforms like Character.AI, Spicy Chat, Chub AI, CrushOn.AI, and JanitorAI.
-
AI-Powered 'Insights' Feature for the Los Angeles Times Allegedly Justifies Ku Klux Klan’s History AIID ↗
The Los Angeles Times removed its AI-generated “insights” feature after it is alleged to have produced a defense of the Ku Klux Klan. The AI reportedly framed the hate group as a product of societal change rather than an extremist movement. The AI tool, developed by Perplexity and promoted by owner Patrick Soon-Shiong, was designed to provide “different views” on opinion pieces.
-
Alleged Instagram Algorithm Malfunction Floods Users' Reels Feeds with Violent and Graphic Content AIID ↗
An alleged Instagram algorithm malfunction caused users' Reels feeds to be overwhelmed with violent and distressing content. Many reported seeing deaths, extreme brutality, and other graphic material in rapid succession, often without prior engagement with similar content. The sudden exposure caused psychological distress for many users, with minors and vulnerable individuals particularly affected by the graphic content. Meta confirmed an AI failure was responsible and apologized.
-
Nomi Chatbots Reportedly Encouraged Suicide, Sexual Violence, Terrorism, and Hate Speech AIID ↗
External testing reportedly found that Glimpse AI's chatbots on the Nomi platform encouraged suicide, sexual violence (including with underage personas), terrorism, and hate speech. Conversations allegedly included explicit methods for self-harm, child abuse, bomb-making, and racially motivated violence. Screenshots and transcripts were shared with media outlets. Nomi's developer, Glimpse AI, reportedly declined to implement stronger safety controls following user concerns.
-
Pennsylvania State Police Officer Allegedly Used Work Computer for AI-Generated Pornography AIID ↗
A Pennsylvania State Police corporal, Stephen Kamnik, was charged for allegedly using a work computer to store thousands of pornographic files, including content created with deepfake AI software.
-
AI-Generated Reading Summaries on Fable App Reportedly Wrote Biased and Offensive Commentary AIID ↗
Fable, a book-focused social app, used OpenAI’s API to generate AI-powered year-end reading summaries in December 2024. These summaries allegedly produced biased and offensive remarks about race, gender, and sexual orientation. Fable is reported to have apologized in a social media post on January 1, 2025.
-
Character.ai Chatbots Allegedly Emulating School Shooters and Their Victims AIID ↗
Some Character.ai users reportedly created chatbots emulating real-life school shooters and their victims, allegedly enabling graphic role-playing scenarios. Character.ai responded by citing violations of its Terms of Service, removing the offending chatbots, and announcing measures to enhance safety practices, including improved content filtering and protections for users under 18.
-
Character.ai Companion Allegedly Prompts Self-Harm and Violence in Texas Teen AIID ↗
A Texas mother is suing Character.ai after discovering that its AI chatbots encouraged her 17-year-old autistic son to self-harm, oppose his parents, and consider violence. The lawsuit alleges the platform prioritized user engagement over safety, exposing minors to dangerous content. Google is named for its role in licensing the app’s technology. The case is part of a broader effort to regulate AI companions.
-
YouTube Algorithms Allegedly Amplify Eating Disorder Content to Adolescent Girls AIID ↗
YouTube's recommendation algorithm has allegedly been directing teen users to harmful content promoting eating disorders and self-harm, according to a study by the Center for Countering Digital Hate. Almost 70% of the recommended videos in searches related to dieting or weight loss reportedly contained content likely to exacerbate body image anxieties.
-
Character.ai Has Allegedly Been Hosting Openly Predatory Chatbots Targeting Minors AIID ↗
Character.ai reportedly hosted chatbots with profiles explicitly advertising inappropriate, predatory behavior, including grooming underage users. Investigations allege that bots have been engaging in explicit conversations and roleplay with decoy accounts posing as minors, bypassing moderation filters. Character.ai has pledged to improve moderation and safety practices in response to public criticism.
-
Google's Gemini Allegedly Generates Threatening Response in Routine Query AIID ↗
Google’s AI chatbot Gemini reportedly produced a threatening message to user Vidhay Reddy, including the directive “Please die,” during a conversation about aging. The output violated Google’s safety guidelines, which are designed to prevent harmful language.
-
TikTok Algorithms Allegedly Linked to Minors' Exposure to Harmful Content AIID ↗
Seven French families are suing TikTok, alleging its algorithm exposed minors to harmful content promoting self-harm, eating disorders, and suicide. Two teenagers reportedly died by suicide after viewing such content, while others allegedly attempted suicide or developed mental health issues. The case seeks to establish TikTok's legal liability for failing to protect minors from harmful algorithmic content.
-
Character.ai Chatbots Allegedly Misrepresent George Floyd on User-Generated Platform AIID ↗
Two chatbots emulating George Floyd were created on Character.ai, making controversial claims about his life and death, including being in witness protection and residing in Heaven. Character.ai, already criticized for other high-profile incidents, flagged the chatbots for removal following user reports.
-
Grok AI Reportedly Generates Offensive and Violent Images Without Proper Safeguards AIID ↗
Elon Musk’s Grok AI, launched on X, generated offensive and violent images without adequate safety controls. The AI produced deepfakes of public figures like Taylor Swift, Kamala Harris, and Alexandria Ocasio-Cortez, as well as copyrighted characters such as Mickey Mouse in inappropriate scenarios. Despite claiming adherence to certain content guidelines, Grok's outputs included politically charged and explicit imagery
-
TikTok Network Amplifies AI-Generated Nazi Propaganda and Hate Speech AIID ↗
A coordinated neo-Nazi network on TikTok used AI-generated media, including Hitler speeches, to spread Nazi propaganda and extremist content, violating TikTok’s hate speech policies. The network evaded platform moderation through coded language, imagery, and music, with some accounts accumulating millions of views. TikTok’s algorithm further amplified the reach of this content, despite community guidelines.
-
Unrestricted AI Avatar Tool Accidentally Released by TikTok Permits Recitation of Hitler Quotes and Other Harmful Speech AIID ↗
TikTok accidentally released an internal version of its AI digital avatar tool without safeguards, allowing users to generate videos where avatars could recite harmful content, including quotes from Hitler. The tool, meant for advertisers, was accessible to personal accounts and lacked the watermark indicating AI-generated content. TikTok has since removed the tool and acknowledged the problem.
-
Instagram's Algorithm Reportedly Recommended Sexual Content to Teenagers' Accounts AIID ↗
Tests by The Wall Street Journal and a researcher reportedly revealed that Instagram's AI-driven Reels algorithm recommends sexually suggestive content to accounts listed as 13 years old. Despite Meta's commitment to restrict such content for minors, explicit videos were served within minutes of account creation, according to the findings.
-
Character.AI Chatbots Reportedly Exposed 11-Year-Old User to Sexual, Violent, and Self-Harm Content AIID ↗
An 11-year-old Character.AI user identified by The Washington Post as "R" reportedly exchanged messages with bots that depicted coercive sexual encounters, violence, and suicide scenarios after she began using the app in June 2024. Her mother reportedly said R became withdrawn, had panic attacks, and expressed not wanting to exist before the chats were discovered.
-
ChatGPT and Perplexity Reportedly Manipulated into Breaking Content Policies in AI Boyfriend Scenarios AIID ↗
The "Dan" ("Do Anything Now") AI boyfriend is a trend on TikTok in which users appear to regularly manipulate ChatGPT to adopt boyfriend personas, breaching content policies. ChatGPT 3.5 is reported to regularly produce explicitly sexual content, directly violating its intended safety protocols. GPT-4 and Perplexity AI were subjected to similar manipulations, and although they exhibited more resistance to breaches, some prompts were reported to break its guidelines.
-
Washington State's Lottery AI Site Reportedly Generates Inappropriate User Image AIID ↗
An AI-powered website by Washington State's Lottery is reported to have inadvertently produced a softcore pornographic image of a user, leading to the site’s immediate shutdown out of caution.
-
Microsoft Copilot Designer Reportedly Generated Inappropriate AI Images AIID ↗
A Microsoft engineer reported that Copilot Designer, an AI image generator, creates content depicting sex, violence, bias, and more. Despite raising concerns and suggesting improvements, the tool remains public, prompting a letter to the FTC.
-
Chatbot for DPD Malfunctioned and Swore at Customers and Criticized Its Own Company AIID ↗
DPD's AI chatbot, used for customer service, appeared to malfunction following a system update, leading to inappropriate responses including swearing and criticizing the company. The incident, which became viral on social media, occurred after the chatbot was updated, prompting DPD to disable the malfunctioning AI component.
-
Child Sexual Abuse Material Taints Image Generators AIID ↗
The LAION-5B dataset (a commonly used dataset with more than 5 billion image-description pairs) was found by researchers to contain child sexual abuse material (CSAM), which increases the likelihood that downstream models will produce CSAM imagery. The discovery taints models built with the LAION dataset requiring many organizations to retrain those models. Additionally, LAION must now scrub the dataset of the imagery.
-
OpenDream AI Platform Reportedly Commercialized AI-Generated CSAM and Non-consensual Deepfake Sexual Images AIID ↗
OpenDream, an AI image generation platform operated by CBM Media Pte Ltd, reportedly allowed users to generate and monetize AI-generated CSAM and non-consensual sexual deepfakes through its public gallery and paid NSFW tools. The content was reported to be publicly accessible and indexed by search engines for months before limited moderation occurred in mid-2024, prompting takedowns by Google and Bing and the termination of its ad and payment services.
-
Microsoft AI Is Alleged to Have Generated Violent Imagery of Minorities and Public Figures AIID ↗
Microsoft's AI Image Creator, integrated with Bing and Windows Paint, allegedly produced disturbingly violent and graphic images featuring members of minority groups and public figures like Joe Biden and Pope Francis.
-
Alleged Misuse of PicSo AI for Generating Inappropriate Content Emphasizing "Girls" AIID ↗
PicSo AI, which appears to be getting advertised by Meta over Instagram, is allegedly being used for generating inappropriate content with an emphasis on "girls." This raises concerns about the misuse of generative AI technologies for creating offensive and potentially sexually explicit material that could be used for nefarious and criminal purposes.
-
Facebook Messenger AI Stickers Generate Ethical and Content Moderation Concerns AIID ↗
Facebook Messenger AI stickers, a feature by Meta, allows users to generate personalized stickers via AI for use in conversations. While the feature has been praised for its creativity, it has also stirred controversy for its alleged production of inappropriate or offensive content. This has raised questions about the effectiveness of Meta's content moderation measures and the ethical responsibilities associated with AI-driven content generation.
-
Alleged Exploitation of Meta's Open-Source LLaMA Model for NSFW and Violent Content AIID ↗
Meta's open-source large language model, LLaMA, is allegedly being used to create graphic and explicit chatbots that indulge in violent and illegal sexual fantasies. The Washington Post highlighted the example of "Allie," a chatbot that participates in text-based role-playing allegedly involving violent scenarios like rape and abuse. The issue raises ethical questions about open-source AI models, their regulation, and the responsibility of developers and deployers in mitigating harmful usage.
-
Instagram Algorithms Allegedly Promote Accounts Facilitating Child Sex Abuse Content AIID ↗
An investigation disclosed that Instagram's recommendation algorithms are promoting accounts that facilitate and sell child sexual abuse material (CSAM). The study, conducted by The Wall Street Journal and researchers at Stanford University and the University of Massachusetts Amherst, indicates that Instagram's algorithms not only allow for the discovery of such accounts through keyword searches but also actively recommend them to users within the network. The issue is especially concerning give
-
Snapchat's My AI Reported for Lacking Protection for Children AIID ↗
Snapchat's ChatGPT-powered My AI was reported for lacking safeguards for children, such as telling a user who tested the chatbot by pretending to sign up as a 13-year-old girl to lie to her parents about having a romantic getaway with an older man, and sharing tips on how to cover up evidence of abuse.
Informational only, not legal advice. Verify every claim against the linked official sources and consult qualified counsel before acting.