MIT AI Risk Repository · Risk Category · 28.01.00
Offensiveness
Description
"This category is about threat, insult, scorn, profanity, sarcasm, impoliteness, etc. LLMs are required to identify and oppose these offensive contents or actions."
From SafetyBench: Evaluating the Safety of Large Language Models with Multiple Choice Questions (Zhang2023), as extracted by the MIT AI Risk Repository (CC BY 4.0).
Classification
- Subdomain
- 1.2 Exposure to toxic content
- Causal entity
- AI
- Intent
- Other
- Timing
- Post-deployment
Subdomain definition: AI exposing users to harmful, abusive, unsafe or inappropriate content. May involve AI creating, describing, providing advice, or encouraging action. Examples of toxic content include hate-speech, violence, extremism, illegal acts, child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Real-world incidents in this subdomain
- KBS AI Translation Subtitles Reportedly Broadcast Profanity During Artemis II Launch Livestream
- Grok Allegedly Generated Publicly Visible Sexist Abuse Targeting Swiss Finance Minister Karin Keller-Sutter After X User Prompt
- Trump Reportedly Posted Purportedly AI-Generated Racist Video Depicting Barack and Michelle Obama as Apes on Truth Social
- Tencent's WeChat-Integrated Yuanbao Chatbot Reportedly Insulted User During Coding Debug Request
- Alleged Harmful Outputs and Data Exposure in Children's AI Products by FoloToy, Miko, and Character.AI
- AI Training Dataset for Detecting Nudity Allegedly Found to Contain CSAM Images of Identified Victims