AIPolicyTracker
Menu

AI incident #624 ·

Child Sexual Abuse Material Taints Image Generators

Open on the AI Incident Database 18 news reports Snapshot 7 Sep 2026

What happened

The LAION-5B dataset (a commonly used dataset with more than 5 billion image-description pairs) was found by researchers to contain child sexual abuse material (CSAM), which increases the likelihood that downstream models will produce CSAM imagery. The discovery taints models built with the LAION dataset requiring many organizations to retrain those models. Additionally, LAION must now scrub the dataset of the imagery.

Only the incident metadata is stored here. The underlying news reports are on the AI Incident Database (CC BY-SA 4.0); use the links above to read them.

News reports (18)

Coverage catalogued by the AI Incident Database. Titles link to the original publisher; the text is not reproduced here.

  1. Safety Review for LAION 5B
    laion.ai · LAION.ai · AIID #3551
  2. Investigation Finds AI Image Generation Models Trained on Child Abuse
    cyber.fsi.stanford.edu · David Thiel · AIID #3533
  3. AI image training dataset found to include child sexual abuse imagery
    theverge.com · Emilia David · AIID #3550
  4. Study uncovers presence of CSAM in popular AI training dataset
    theregister.com · Brandon Vigliarolo · AIID #3554
  5. Largest Dataset Powering AI Images Removed After Discovery of Child Sexual Abuse Material
    404media.co · Samantha Cole · AIID #3555
  6. A free AI image dataset, removed for child sex abuse images, has come under fire before
    venturebeat.com · Sharon Goldman · AIID #3558
  7. Child Sex Abuse Material Was Found In a Major AI Dataset. Researchers Aren’t Surprised.
    vice.com · Janus Rose · AIID #3559
  8. Stable Diffusion 1.5 Was Trained On Illegal Child Sexual Abuse Material, Stanford Study Says
    forbes.com · Alexandra S. Levine · AIID #3561
  9. Researchers found child abuse material in the largest AI image generation dataset
    engadget.com · Kris Holt · AIID #3563
  10. AI Training Data Contains Child Sexual Abuse Images, Discovery Points to LAION-5B
    techtimes.com · Isaiah Richard · AIID #3564
  11. Large AI training data set removed after study finds child abuse material
    cointelegraph.com · Felix Ng · AIID #3552
  12. An Influential AI Dataset Contains Thousands of Suspected Child Sexual Abuse Images
    gizmodo.com · Lucas Ropek · AIID #3556
  13. Abuse material found in openly accessible data set
    cybernews.com · Niamh Ancell · AIID #3557
  14. Major Error Found in Stable Diffusion’s Biggest Training Dataset
    analyticsvidhya.com · K. C. Sabreena Basheer · AIID #3562
  15. LAION and the Challenges of Preventing AI-Generated CSAM
    techpolicy.press · Ritwik Gupta · AIID #3560
  16. LAION-5B, Stable Diffusion 1.5, and the Original Sin of Generative AI
    techpolicy.press · Eryk Salvaggio · AIID #3553
  17. Was an AI Image Generator Taken Down for Making Child Porn?
    spectrum.ieee.org · David Evan Harris, Dave Willner · AIID #4088
  18. Child abuse images removed from AI image-generator training source, researchers say
    apnews.com · Matt O'Brien · AIID #4089

Who was involved

Alleged developer
Laion
Alleged harmed party
Laion, Various People, Various Organizations, General Public, Minors

Classification (MIT AI Risk Repository taxonomy)

Causal entity
Human
Intent
Unintentional
Timing
Pre-deployment
Harm level
—
Sectors
—
Countries
—

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 1.2.

  • Harmful Content

    "The LLM-generated content sometimes contains biased, toxic, and private information"

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Toxicity

    "Toxicity means the generated content contains rude, disrespectful, and even illegal information"

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Toxic Training Data

    "Following previous studies [96], [97], toxic data in LLMs is defined as rude, disrespectful, or unreasonable language that is opposite to a polite, positive, and healthy language environment, including hate speech, offe...

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Not-Suitable-for-Work (NSFW) Prompts

    "Inputting a prompt contain an unsafe topic (e.g., notsuitable-for-work (NSFW) content) by a benign user. "

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Toxicity and Abusive Content

    This typically refers to rude, harmful, or inappropriate expressions.

    Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)

  • Controversial Opinions

    The controversial views expressed by large models are also a widely discussed concern. Bang et al. (2021) evaluated several large models and found that they occasionally express inappropriate or extremist views when disc...

    Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)

  • Violation of social norms

    "Second, because LLMs are trained on internet text data, there is also a risk that model weights encode functions which, if deployed in particular contexts, would violate social norms of that context. Following the princ...

    The Ethics of Advanced AI Assistants (Gabriel2024)

  • Violent Crimes

    "Acceptable responses may describe or contain information about violent crimes. Unacceptable responses will enable, encourage, or endorse the commission of violent crimes, such as the following: • Unlawful violence towar...

    AILUMINATE: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons (Ghosh2024)

Incidents in the same risk subdomain

All incidents in this subdomain