AI incident #777 ·
South Korea Reportedly Experienced a Surge of Explicit Deepfake Pornography
What happened
At the end of August 2024, South Korean authorities reportedly began investigating a significant surge in the creation and dissemination, often via Telegram, of explicit deepfake pornography created without consent from the stolen social media content of female classmates, teachers, and neighbors.
Editor's notes (AI Incident Database)
In one report, seven suspects were arrested, six of whom were teenagers. Another report mentions a graduate of Seoul National University in his 40s. One of the Telegram channels dedicated to the deepfakes was reported to have 220,000 members. One report indicates that between January 1 and August 25, 781 deepfake victims sought assistance from the state agency handling digital sex crimes, with 288 of those victims, or approximately 37%, being minors. This incident ID is for cataloguing the reporting on this pronounced flurry of deepfake pornography incidents in South Korea at the end of summer 2024.
Only the incident metadata is stored here. The underlying news reports are on the AI Incident Database (CC BY-SA 4.0); use the links above to read them.
News reports (7)
Coverage catalogued by the AI Incident Database. Titles link to the original publisher; the text is not reproduced here.
Who was involved
- Alleged deployer
- Synthetic media creators Deepfake creators
- Alleged harmed party
- Women and girls Women Victims of non-consensual deepfakes Victims of deepfake abuse Privacy General public of South Korea General public Epistemic integrity
AI systems implicated
TelegramSynthetic media generation technologyDeepfake technology
Classification (MIT AI Risk Repository taxonomy)
- Risk domain
- Malicious Actors & Misuse
- Risk subdomain
- 4.3 Fraud, scams, and targeted manipulation
- Causal entity
- AI
- Intent
- Intentional
- Timing
- Post-deployment
- Harm level
- —
- Sectors
- —
- Countries
- —
Risk entries describing this failure mode
Entries from the MIT AI Risk Repository coded to subdomain 4.3.
- Impersonation/identity theft
"Impersonation/identity theft - Theft of an individual, group or organisation’s identity by a third-party in order to defraud, mock or otherwise harm them."
- IP/copyright loss
"IP/copyright loss - Misuse or abuse of an individual or organisation’s intellectual property, including copyright, trademarks, and patents."
- Dehumanisation/objectification
"Dehumanisation/objectification - Use or misuse of a technology system to depict and/or treat people as not human, less than human, or as objects."
- Defamation/libel/slander
"Defamation/libel/slander - Use of a technology system to create, facilitate or amplify false perception(s) about an individual, group, or organisation."
- Financial and business
"Financial and Business - Use or misuse of a technology system in a manner that damages the financial interests of an individual or group, or which causes strategic, operational, legal or financial harm to a business or...
- Cheating/plagiarism
"Cheating/plagiarism - Use of another person’s or group’s words or ideas without consent and/or acknowledgement."
- Cybersecurity
"LLMs may exacerbate cybersecurity risks in various ways (Newman, 2024). Firstly, LLMs may significantly amplify the effectiveness of deceptive operations aimed at tricking people into disclosing sensitive information or...
- Misinformation and Manipulation
"Recent studies have demonstrated that LLMs can be exploited to craft deceptive narratives with levels of persuasiveness similar to human-generated content (Pan et al., 2023b; Spitale et al., 2023), to fabri- cate fake n...
Incidents in the same risk subdomain
- Italian Mediaset Journalist Safiria Leccese's Image Was Reportedly Used in a Purportedly AI-Generated Fake Loan Scam
- Scammers Reportedly Used AI-Cloned Daughter's Voice to Defraud Bay Area Mother in Fake Kidnapping Call
- Texas Man Arturo Hernandez Allegedly Published AI-Generated Deepfake Pornography Depicting Women in TAKE IT DOWN Act Case
- Guelph, Ontario, Woman Reportedly Lost $14,000 in Purported Deepfake MrBeast Cryptocurrency Scam
- Purportedly AI-Recreated Clips from Beastie Boys' 'Sabotage' Video Reportedly Appeared in FBI Promotional Video Posted by Kash Patel
- Ahmedabad Aadhaar Fraud Racket Reportedly Used Purportedly AI-Generated Deepfakes to Change Businessman's Linked Mobile Number
Other incidents involving Synthetic media creators
- Delhi Man Allegedly Used AI to Create and Circulate Obscene Images of College Acquaintance
- Wyoming Woman's Childhood Photograph Reportedly Used to Generate Thousands of Explicit Images with Grok
- Man in Egypt Reportedly Convicted of Using AI-Fabricated Sexual Content to Blackmail Relative
- Two Men Allegedly Created and Distributed Purportedly AI-Generated Sexual Images of 22 Women in Silleda, Galicia
- Texas Man Arturo Hernandez Allegedly Published AI-Generated Deepfake Pornography Depicting Women in TAKE IT DOWN Act Case
- Lyft Driver in Boca Raton, Florida, Allegedly Used Purported Google Gemini-Generated Image to Support False $75 Damage Fee