AIPolicyTracker
Menu

AI incident ·

AI incident: sexual content (Aug 2024)

7 news reports Synced from source · record last edited 5 Sep 2026

In brief

An AI system built by Synthetic media generation technology developers and Deepfake technology developers and deployed by Synthetic media creators and Deepfake creators allegedly harmed Women and girls, Women and 6 others.

Risk domain
Malicious Actors & Misuse Fraud, scams, and targeted manipulation
Occurred
Coverage
7 reportsAug 2024 - Sep 2024

What happened

At the end of August 2024, South Korean authorities reportedly began investigating a significant surge in the creation and dissemination, often via Telegram, of explicit deepfake pornography created without consent from the stolen social media content of female classmates, teachers, and neighbors.

Editor's notes

In one report, seven suspects were arrested, six of whom were teenagers. Another report mentions a graduate of Seoul National University in his 40s. One of the Telegram channels dedicated to the deepfakes was reported to have 220,000 members. One report indicates that between January 1 and August 25, 781 deepfake victims sought assistance from the state agency handling digital sex crimes, with 288 of those victims, or approximately 37%, being minors. This incident ID is for cataloguing the reporting on this pronounced flurry of deepfake pornography incidents in South Korea at the end of summer 2024.

Laws that address this harm

Policy angle: Classified under Malicious Actors & Misuse (Fraud, scams, and targeted manipulation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.

Matched from the record's risk domain and country to the instruments recorded here. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).

News reports (7)

Titles link to the original publisher; report text is not reproduced here.

  1. South Korea faces deepfake porn 'emergency'
    bbc.com · Jean Mackenzie, Nick Marsh
  2. (Yonhap Feature) S. Korean women scurrying to take down selfies amid deepfake porn crimes
    en.yna.co.kr · Kim Hyun-soo
  3. Inside the deepfake porn crisis engulfing Korean schools
    bbc.com · Jean Mackenzie, Leehyun Choi
  4. South Korean Teenagers Detained Over Deepfake Sexual Images
    nytimes.com · Choe Sang-Hun
  5. South Korea Is Facing Deepfake Porn Crisis
    bloomberg.com. · Catherine Thorbecke
  6. In S​outh Korea, Misogyny Has a New Weapon: Deepfake Sex Videos
    nytimes.com · Choe Sang-Hun
  7. 他们给AI投币1美元，百万女性被拖入地狱｜深氪lite
    baijiahao.baidu.com · Deng Yongyi, Zhou Xinyu, Qiu Xiaofen

Who was involved

Alleged harmed party
Women and girls Women Victims of non-consensual deepfakes Victims of deepfake abuse Privacy General public of South Korea General public Epistemic integrity

AI systems implicated

TelegramSynthetic media generation technologyDeepfake technology

Classification (MIT AI Risk Repository taxonomy)

Causal entity
AI
Intent
Intentional
Timing
Post-deployment
Harm level
—
Sectors
—
Countries
—

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 4.3.

  • Cheating/plagiarism

    "Cheating/plagiarism - Use of another person’s or group’s words or ideas without consent and/or acknowledgement."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • IP/copyright loss

    "IP/copyright loss - Misuse or abuse of an individual or organisation’s intellectual property, including copyright, trademarks, and patents."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Financial and business

    "Financial and Business - Use or misuse of a technology system in a manner that damages the financial interests of an individual or group, or which causes strategic, operational, legal or financial harm to a business or...

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Impersonation/identity theft

    "Impersonation/identity theft - Theft of an individual, group or organisation’s identity by a third-party in order to defraud, mock or otherwise harm them."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Dehumanisation/objectification

    "Dehumanisation/objectification - Use or misuse of a technology system to depict and/or treat people as not human, less than human, or as objects."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Defamation/libel/slander

    "Defamation/libel/slander - Use of a technology system to create, facilitate or amplify false perception(s) about an individual, group, or organisation."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Misinformation and Manipulation

    "Recent studies have demonstrated that LLMs can be exploited to craft deceptive narratives with levels of persuasiveness similar to human-generated content (Pan et al., 2023b; Spitale et al., 2023), to fabri- cate fake n...

    Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

  • Cybersecurity

    "LLMs may exacerbate cybersecurity risks in various ways (Newman, 2024). Firstly, LLMs may significantly amplify the effectiveness of deceptive operations aimed at tricking people into disclosing sensitive information or...

    Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

Incidents in the same risk subdomain

All incidents in this subdomain

Other incidents involving Synthetic media creators

Source record: incident #777 on the AI Incident Database · all 7 reports