AIPolicyTracker
Menu

AI incident #1004 ·

San Francisco City Attorney Sues Operators of AI Deepfake Pornography Websites for Violations of State and Federal Law

Open on the AI Incident Database 2 news reports Synced from the AIID API · record last edited 5 Sep 2026

What happened

In August 2024, the San Francisco City Attorney’s Office filed a first-of-its-kind lawsuit against 16 websites accused of using generative AI to create and distribute nonconsensual pornographic deepfakes, including images of minors. The sites, which had over 200 million visits in six months, invited users to upload real images to generate explicit fakes. By March 2025, 11 of the targeted websites had shut down following legal pressure.

Editor's notes (AI Incident Database)

Timeline notes: The San Francisco City Attorney's office announced its actions against the 16 operators of these sites on August 15, 2024 (the date for this incident ID). By March 2025, reporting on these actions stated that 11 of the sites had been taken down.

Only the incident metadata is stored here. The underlying news reports are on the AI Incident Database (CC BY-SA 4.0); use the links above to read them.

News reports (2)

Coverage catalogued by the AI Incident Database. Titles link to the original publisher; the text is not reproduced here.

  1. City Attorney sues most-visited websites that create nonconsensual deepfake pornography
    sfcityattorney.org · Office of the City Attorney of San Francisco · AIID #5017
  2. SF City Attorney's Office takes down websites spreading deepfake nudes
    nbcbayarea.com · Ginger Conejero Saab · AIID #5016

Who was involved

Alleged harmed party
Women depicted in deepfake pornography Women and girls Women Victims of non-consensual deepfakes Victims of deepfake child abuse Victims of deepfake abuse Privacy Minors depicted in deepfake pornography Minors Girls
On AIID: Women depicted in deepfake pornography, Women and girls, Women, Victims of non-consensual deepfakes, Victims of deepfake child abuse, Victims of deepfake abuse, Privacy, Minors depicted in deepfake pornography, Minors, Girls

AI systems implicated

Synthetic media generation technologyDeepfake technology

Classification (MIT AI Risk Repository taxonomy)

Causal entity
AI
Intent
Intentional
Timing
Post-deployment
Harm level
—
Sectors
—
Countries
—

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 4.3.

  • Impersonation/identity theft

    "Impersonation/identity theft - Theft of an individual, group or organisation’s identity by a third-party in order to defraud, mock or otherwise harm them."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • IP/copyright loss

    "IP/copyright loss - Misuse or abuse of an individual or organisation’s intellectual property, including copyright, trademarks, and patents."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Dehumanisation/objectification

    "Dehumanisation/objectification - Use or misuse of a technology system to depict and/or treat people as not human, less than human, or as objects."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Defamation/libel/slander

    "Defamation/libel/slander - Use of a technology system to create, facilitate or amplify false perception(s) about an individual, group, or organisation."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Financial and business

    "Financial and Business - Use or misuse of a technology system in a manner that damages the financial interests of an individual or group, or which causes strategic, operational, legal or financial harm to a business or...

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Cheating/plagiarism

    "Cheating/plagiarism - Use of another person’s or group’s words or ideas without consent and/or acknowledgement."

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Cybersecurity

    "LLMs may exacerbate cybersecurity risks in various ways (Newman, 2024). Firstly, LLMs may significantly amplify the effectiveness of deceptive operations aimed at tricking people into disclosing sensitive information or...

    Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

  • Misinformation and Manipulation

    "Recent studies have demonstrated that LLMs can be exploited to craft deceptive narratives with levels of persuasiveness similar to human-generated content (Pan et al., 2023b; Spitale et al., 2023), to fabri- cate fake n...

    Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

Linked by AIID editors or by its text-similarity model.

Incidents in the same risk subdomain

All incidents in this subdomain

Other incidents involving Synthetic media creators