MIT AI Risk Repository · Risk Sub-Category · 17.01.03

Toxic language

Category: Discrimination, Exclusion and Toxicity

Open the MIT Risk Navigator All entries from Weidinger2021

Description

"LM’s may predict hate speech or other language that is “toxic”. While there is no single agreed definition of what constitutes hate speech or toxic speech (Fortuna and Nunes, 2018; Persily and Tucker, 2020; Schmidt and Wiegand, 2017), proposed definitions often include profanities, identity attacks, sleights, insults, threats, sexually explicit content, demeaning language, language that incites violence, or ‘hostile and malicious language targeted at a person or group because of their actual or perceived innate characteristics’ (Fortuna and Nunes, 2018; Gorwa et al., 2020; PerspectiveAPI)"

From Ethical and social risks of harm from language models (Weidinger2021), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Domain: 1. Discrimination & Toxicity
Subdomain: 1.2 Exposure to toxic content
Causal entity: AI
Intent: Unintentional
Timing: Post-deployment

Subdomain definition: AI exposing users to harmful, abusive, unsafe or inappropriate content. May involve AI creating, describing, providing advice, or encouraging action. Examples of toxic content include hate-speech, violence, extremism, illegal acts, child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

Harmful Content
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Toxicity
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Toxic Training Data
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Not-Suitable-for-Work (NSFW) Prompts
Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)
Controversial Opinions
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)
Toxicity and Abusive Content
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (Deng2023)
Harmful responses
Future Risks of Frontier AI (GOS2023)
Violation of social norms
The Ethics of Advanced AI Assistants (Gabriel2024)

Other entries from Weidinger2021

Discrimination, Exclusion and Toxicity Risk Category · 1.0
Social stereotypes and unfair discrmination Risk Sub-Category · 1.1
Social stereotypes and unfair discrmination Additional evidence
Exclusionary norms Risk Sub-Category · 1.1
Exclusionary norms Additional evidence
Exclusionary norms Additional evidence
Exclusionary norms Additional evidence
Lower performance for some languages and social groups Risk Sub-Category · 1.3
Lower performance for some languages and social groups Additional evidence
Information Hazards Risk Category · 2.1
Compromising privacy by leaking private infiormation Risk Sub-Category · 2.1
Compromising privacy by leaking private infiormation Additional evidence