MIT AI Risk Repository · Risk Sub-Category · 31.03.02

Generative AI User Data

Category: Opaque Data Collection

Open the MIT Risk Navigator All entries from EPIC2023

Description

Many generative AI tools require users to log in for access, and many retain user information, including contact information, IP address, and all the inputs and outputs or “conversations” the users are having within the app. These practices implicate a consent issue because generative AI tools use this data to further train the models, making their “free” product come at a cost of user data to train the tools. This dovetails with security, as mentioned in the next section, but best practices would include not requiring users to sign in to use the tool and not retaining or using the user-genera

From Generating Harms - Generative AI's impact and paths forwards (EPIC2023), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 2. Privacy & Security
Subdomain: 2.1 Compromise of privacy by obtaining, leaking or correctly inferring sensitive information
Causal entity: Human
Intent: Unintentional
Timing: Post-deployment

Subdomain definition: AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or loss of confidential intellectual property.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Risks to privacy
International Scientific Report on the Safety of Advanced AI (Bengio2024)
Risks to privacy
International AI Safety Report 2025 (Bengio2025)
Privacy Leakage
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Private Training Data
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Memorization in LLMs
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Association in LLMs
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Privacy Leakage
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Privacy and regulation violations
Navigating the Landscape of AI Ethics and Responsibility (Cunha2023)

Other entries from EPIC2023

Information Manipulation Risk Category · 4.1
Scams Risk Sub-Category · 4.3
Disinformation Risk Sub-Category · 4.1
Misinformation Risk Sub-Category · 3.1
Security Risk Sub-Category · 4.2
Clickbait and feeding the surveillance advertising ecosystem Risk Sub-Category · 3.2
Harassment, Impersonation, and Extortion Risk Category · 4.3
Malicious intent Risk Sub-Category · 4.3
Privacy and consent Risk Sub-Category · 4.3
Believability Risk Sub-Category · 4.3
Opaque Data Collection Risk Category · 2.1
Scraping to train data Risk Sub-Category · 2.1