AIPolicyTracker
Menu

AI incident ·

AI incident: sexual content (Oct 2024)

1 news report Synced from source · record last edited 5 Sep 2026

In brief

An AI system built by Synthetic media generation technology developers, Synthetic image generation technology developers and 1 other and deployed by Unnamed 18-year-old Manuel Belgrano male student, University students and 5 others allegedly harmed Women and girls, Women and 6 others.

Risk domain
Malicious Actors & Misuse Fraud, scams, and targeted manipulation
Occurred
Coverage
1 reportOct 2024

What happened

An 18-year-old Argentine student at the pre-university institute of Manuel Belgrano of Córdoba allegedly used AI tools to generate explicit fake images of at least 22 female classmates by combining their faces with other bodies. These images, posted on pornography websites, included the victims' names, leading to harassment and significant psychological harm. Legal authorities charged the student with serious injuries aggravated by gender violence.

Editor's notes

Reconstructing the timeline of events: (1) Mid-2024: Two female students at the Manuel Belgrano Institute began receiving suspicious Instagram messages from men aged 40–50. (2) July 2024: Victims discovered explicit AI-generated images of themselves on pornographic websites, featuring their names and altered appearances. (3) August 2024: Families of victims reported the student to authorities, who launched an investigation and conducted a search of the accused's home. (4) October 18, 2024: Charges were filed against the accused for gender-based violence and psychological harm; legal experts advocated for updated laws addressing AI misuse. This fourth date is being marked as the incident date, even though the initial events are reported to have begun sometime in the middle of 2024.

Laws that address this harm

Policy angle: Classified under Malicious Actors & Misuse (Fraud, scams, and targeted manipulation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.

Matched from the record's risk domain and country to the instruments recorded here. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).

News reports (1)

Titles link to the original publisher; report text is not reproduced here.

  1. Inédita resolución contra un chico de 18 que con IA creó fotos porno falsas de más de 20 alumnas
    clarin.com · Emilia Vexler

Who was involved

Alleged harmed party
Women and girls Women Unnamed Manuel Belgrano female students University students Students Privacy Epistemic integrity Educational communities

AI systems implicated

Synthetic media generation technologySynthetic image generation technologyDeepfake technology

Classification (MIT AI Risk Repository taxonomy)

Causal entity
Human
Intent
Intentional
Timing
Post-deployment
Harm level
—
Sectors
—
Countries
—

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 4.3.

  • Cheating/plagiarism

    "Cheating/plagiarism - Use of another person’s or group’s words or ideas without consent and/or acknowledgement."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • IP/copyright loss

    "IP/copyright loss - Misuse or abuse of an individual or organisation’s intellectual property, including copyright, trademarks, and patents."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Financial and business

    "Financial and Business - Use or misuse of a technology system in a manner that damages the financial interests of an individual or group, or which causes strategic, operational, legal or financial harm to a business or...

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Impersonation/identity theft

    "Impersonation/identity theft - Theft of an individual, group or organisation’s identity by a third-party in order to defraud, mock or otherwise harm them."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Dehumanisation/objectification

    "Dehumanisation/objectification - Use or misuse of a technology system to depict and/or treat people as not human, less than human, or as objects."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Defamation/libel/slander

    "Defamation/libel/slander - Use of a technology system to create, facilitate or amplify false perception(s) about an individual, group, or organisation."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Misinformation and Manipulation

    "Recent studies have demonstrated that LLMs can be exploited to craft deceptive narratives with levels of persuasiveness similar to human-generated content (Pan et al., 2023b; Spitale et al., 2023), to fabri- cate fake n...

    Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

  • Cybersecurity

    "LLMs may exacerbate cybersecurity risks in various ways (Newman, 2024). Firstly, LLMs may significantly amplify the effectiveness of deceptive operations aimed at tricking people into disclosing sensitive information or...

    Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

Linked by editors or by text similarity in the source dataset.

Incidents in the same risk subdomain

All incidents in this subdomain

Source record: incident #848 on the AI Incident Database · all 1 report