Artificial Intelligence Policy Tracker
Search
Menu

AI incident ·

Meta AI's Scientific Paper Generator Reportedly Produced Inaccurate and Harmful Content

3 news reports Snapshot 7 Sep 2026
On this page 9 sections

In brief

An AI system built and deployed by Meta and Facebook allegedly harmed Minority Groups, Meta and 1 other.

Risk domain
Misinformation False or misleading information
Occurred
Coverage
3 reportsNov 2022

What happened

Meta AI trained and hosted a scientific paper generator that sometimes produced bad science and prohibited queries on topics and groups that are likely to produce offensive or harmful content.

Laws on record for this risk area

Policy angle: Classified under Misinformation (False or misleading information) in the MIT AI Risk Repository taxonomy; 5 instruments on record cover this risk area.

Matched from the record's risk category and country to the instruments in effect recorded here: a law listed covers the area, which is not a finding that it applies to this incident. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).

News reports (3)

Titles link to the original publisher; report text is not reproduced here.

  1. New Meta AI demo writes racist and inaccurate scientific literature, gets pulled
    arstechnica.com · Benj Edwards
  2. Why Meta’s latest large language model survived only three days online
    technologyreview.com · Will Douglas Heaven
  3. Thread by @osanseviero on Thread Reader App
    threadreaderapp.com · Omar Sanseviero

Who was involved

Alleged deployer
Meta, Facebook
Alleged developer
Meta, Facebook
Alleged harmed party
Minority Groups, Meta, Facebook, Minority Groups

Classification (MIT AI Risk Repository taxonomy)

Risk domain
Misinformation
Causal entity
AI
Intent
Unintentional
Timing
Post-deployment
Harm level
—
Sectors
—
Countries
—

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 3.1.

  • Pursuing Consistent Context

    "LLMs have been demonstrated to pursue consistent context [129]–[132], which may lead to erroneous generation when the prefixes contain false information. Typical examples include sycophancy [129], [130], false demonstra...

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Defective Decoding Process

    In general, LLMs employ the Transformer architecture [32] and generate content in an autoregressive manner, where the prediction of the next token is conditioned on the previously generated token sequence. Such a scheme...

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Noisy Training Data

    "Another important source of hallucinations is the noise in training data, which introduces errors in the knowledge stored in model parameters [111]–[113]. Generally, the training data inherently harbors misinformation....

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Untruthful Content

    "The LLM-generated content could contain inaccurate information"

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Hallucinations

    "LLMs generate nonsensical, untruthful, and factual incorrect content"

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Faithfulness Errors

    "The LLM-generated content could contain inaccurate information" which is is not true to the source material or input used

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Factuality Errors

    "The LLM-generated content could contain inaccurate information" which is factually incorrect

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Knowledge Gaps

    "Since the training corpora of LLMs can not contain all possible world knowledge [114]–[119], and it is challenging for LLMs to grasp the long-tail knowledge within their training data [120], [121], LLMs inherently posse...

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

Incidents in the same risk subdomain

All incidents in this subdomain

Other incidents involving Meta, Facebook

Source record: incident #399 on the AI Incident Database · all 3 reports