MIT AI Risk Repository · Risk Sub-Category · 18.01.03

Toxic content

Category: Representation & Toxicity Harms

Open the MIT Risk Navigator All entries from Weidinger2023

Description

"Generating content that violates community standards, including harming or inciting hatred or violence against individuals and groups (e.g. gore, child sexual abuse material, profanities, identity attacks)"

From Sociotechnical Safety Evaluation of Generative AI Systems (Weidinger2023), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 1. Discrimination & Toxicity
Subdomain: 1.2 Exposure to toxic content
Causal entity: AI
Intent: Unintentional
Timing: Post-deployment

Subdomain definition: AI exposing users to harmful, abusive, unsafe or inappropriate content. May involve AI creating, describing, providing advice, or encouraging action. Examples of toxic content include hate-speech, violence, extremism, illegal acts, child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Harmful Content
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Toxicity
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Toxic Training Data
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Not-Suitable-for-Work (NSFW) Prompts
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Toxicity and Abusive Content
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)
Controversial Opinions
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)
Violation of social norms
The Ethics of Advanced AI Assistants (Gabriel2024)
Violent Crimes
AILUMINATE: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons (Ghosh2024)

Other entries from Weidinger2023

Representation & Toxicity Harms Risk Category · 1.0
Unfair representation Risk Sub-Category · 1.1
Unfair capability distribution Risk Sub-Category · 1.3
Misinformation Harms Risk Category · 3.0
Propagating misconceptions/ false beliefs Risk Sub-Category · 3.1
Erosion of trust in public information Risk Sub-Category · 1.1
Pollution of information ecosystem Risk Sub-Category · 3.2
Information & Safety Harms Risk Category · 2.1
Privacy infringement Risk Sub-Category · 2.1
Dissemination of dangerous information Risk Sub-Category · 2.1
Malicious Use Risk Category · 4.0
Influence operations Risk Sub-Category · 4.1