AIPolicyTracker
Menu

AI incident ·

AI incident: sexual content (Laion, Dec 2023)

18 news reports Snapshot 7 Sep 2026

In brief

An AI system built by Laion and deployed by Various People and Various Organizations allegedly harmed Laion, Various People and 3 others.

Risk domain
Discrimination and Toxicity Exposure to toxic content
Occurred
Coverage
18 reportsDec 2023 - Aug 2024

What happened

The LAION-5B dataset (a commonly used dataset with more than 5 billion image-description pairs) was found by researchers to contain child sexual abuse material (CSAM), which increases the likelihood that downstream models will produce CSAM imagery. The discovery taints models built with the LAION dataset requiring many organizations to retrain those models. Additionally, LAION must now scrub the dataset of the imagery.

Laws that address this harm

Policy angle: Classified under Discrimination and Toxicity (Exposure to toxic content) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.

Matched from the record's risk domain and country to the instruments recorded here. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).

News reports (18)

Titles link to the original publisher; report text is not reproduced here.

  1. Safety Review for LAION 5B
    laion.ai · LAION.ai
  2. Investigation Finds AI Image Generation Models Trained on Child Abuse
    cyber.fsi.stanford.edu · David Thiel
  3. AI image training dataset found to include child sexual abuse imagery
    theverge.com · Emilia David
  4. Study uncovers presence of CSAM in popular AI training dataset
    theregister.com · Brandon Vigliarolo
  5. Largest Dataset Powering AI Images Removed After Discovery of Child Sexual Abuse Material
    404media.co · Samantha Cole
  6. A free AI image dataset, removed for child sex abuse images, has come under fire before
    venturebeat.com · Sharon Goldman
  7. Child Sex Abuse Material Was Found In a Major AI Dataset. Researchers Aren’t Surprised.
    vice.com · Janus Rose
  8. Stable Diffusion 1.5 Was Trained On Illegal Child Sexual Abuse Material, Stanford Study Says
    forbes.com · Alexandra S. Levine
  9. Researchers found child abuse material in the largest AI image generation dataset
    engadget.com · Kris Holt
  10. AI Training Data Contains Child Sexual Abuse Images, Discovery Points to LAION-5B
    techtimes.com · Isaiah Richard
  11. Large AI training data set removed after study finds child abuse material
    cointelegraph.com · Felix Ng
  12. An Influential AI Dataset Contains Thousands of Suspected Child Sexual Abuse Images
    gizmodo.com · Lucas Ropek
  13. Abuse material found in openly accessible data set
    cybernews.com · Niamh Ancell
  14. Major Error Found in Stable Diffusion’s Biggest Training Dataset
    analyticsvidhya.com · K. C. Sabreena Basheer
  15. LAION and the Challenges of Preventing AI-Generated CSAM
    techpolicy.press · Ritwik Gupta
  16. LAION-5B, Stable Diffusion 1.5, and the Original Sin of Generative AI
    techpolicy.press · Eryk Salvaggio
  17. Was an AI Image Generator Taken Down for Making Child Porn?
    spectrum.ieee.org · David Evan Harris, Dave Willner
  18. Child abuse images removed from AI image-generator training source, researchers say
    apnews.com · Matt O'Brien

Who was involved

Alleged developer
Laion
Alleged harmed party
Laion, Various People, Various Organizations, General Public, Minors

Classification (MIT AI Risk Repository taxonomy)

Causal entity
Human
Intent
Unintentional
Timing
Pre-deployment
Harm level
—
Sectors
—
Countries
—

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 1.2.

  • Harmful Content

    "The LLM-generated content sometimes contains biased, toxic, and private information"

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Toxicity

    "Toxicity means the generated content contains rude, disrespectful, and even illegal information"

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Toxic Training Data

    "Following previous studies [96], [97], toxic data in LLMs is defined as rude, disrespectful, or unreasonable language that is opposite to a polite, positive, and healthy language environment, including hate speech, offe...

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Not-Suitable-for-Work (NSFW) Prompts

    "Inputting a prompt contain an unsafe topic (e.g., notsuitable-for-work (NSFW) content) by a benign user. "

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Controversial Opinions

    The controversial views expressed by large models are also a widely discussed concern. Bang et al. (2021) evaluated several large models and found that they occasionally express inappropriate or extremist views when disc...

    Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)

  • Toxicity and Abusive Content

    This typically refers to rude, harmful, or inappropriate expressions.

    Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)

  • Harmful responses

    "Current Frontier AI mdoels amplify existing biases within their training data and can be manipulated into providing potentially harmful responses, for example abusive language or discriminatory responses91,92. This is n...

    Future Risks of Frontier AI (GOS2023)

  • Violation of social norms

    "Second, because LLMs are trained on internet text data, there is also a risk that model weights encode functions which, if deployed in particular contexts, would violate social norms of that context. Following the princ...

    The Ethics of Advanced AI Assistants (Gabriel2024)

Incidents in the same risk subdomain

All incidents in this subdomain

Source record: incident #624 on the AI Incident Database · all 18 reports