AI incident #1000 ·
Sora Video Generator Has Reportedly Been Creating Biased Human Representations Across Race, Gender, and Disability
What happened
A WIRED investigation found that OpenAI's video generation model, Sora, exhibits representational bias across race, gender, body type, and disability. In tests using 250 prompts, Sora was more likely to depict CEOs and professors as men, flight attendants and childcare workers as women, and showed limited or stereotypical portrayals of disabled individuals and people with larger bodies.
Only the incident metadata is stored here. The underlying news reports are on the AI Incident Database (CC BY-SA 4.0); use the links above to read them.
News reports (2)
Coverage catalogued by the AI Incident Database. Titles link to the original publisher; the text is not reproduced here.
Who was involved
- Alleged deployer
- OpenAI
- Alleged developer
- Synthetic video generation technology developers Synthetic media generation technology developers OpenAI
- Alleged harmed party
- Women and girls Women People with larger bodies People with disabilities People of color Marginalized groups LGBTQ+ people General public
AI systems implicated
Synthetic video generation technologySynthetic media generation technologySora
Classification (MIT AI Risk Repository taxonomy)
- Risk domain
- Discrimination and Toxicity
- Risk subdomain
- 1.1 Unfair discrimination and misrepresentation
- Causal entity
- AI
- Intent
- Unintentional
- Timing
- Post-deployment
- Harm level
- —
- Sectors
- —
- Countries
- —
Risk entries describing this failure mode
Entries from the MIT AI Risk Repository coded to subdomain 1.1.
- Discrimination
"Discrimination - Unfair or inadequate treatment or arbitrary distinction based on a person’s race, ethnicity, age, gender, sexual preference, religion, national origin, marital status, disability, language, or other pro...
- Harms of Representation and Other Biases
"A pretrained LLM generally has many of the stereotypical biases commonly present in the human society (Touvron et al., 2023). This makes it difficult for users to trust that LLMs will work well for them and not produce...
- Risks from bias and underrepresentation
"The outputs and impacts of general- purpose AI systems can be biased with respect to various aspects of human identity, including race, gender, culture, age, and disability. This creates risks in high- stakes domains su...
- Bias
"General-purpose AI systems can amplify social and political biases, causing concrete harm. They frequently display biases with respect to race, gender, culture, age, disability, political opinion, or other aspects of hu...
- Bias
"The training datasets of LLMs may contain biased information that leads LLMs to generate outputs with social biases"
- Toxicity and Bias Tendencies
"Extensive data collection in LLMs brings toxic content and stereotypical bias into the training data."
- Biased Training Data
"Compared with the definition of toxicity, the definition of bias is more subjective and contextdependent. Based on previous work [97], [101], we describe the bias as disparities that could raise demographic differences...
- Broken systems
"These are the most mentioned cases. They refer to situations where the algorithm or the training data lead to unreliable outputs. These systems frequently assign disproportionate weight to some variables, like race or g...
Related incidents on the AI Incident Database
Linked by AIID editors or by its text-similarity model.
Incidents in the same risk subdomain
- DOGE Reportedly Relied on Unvetted ChatGPT Outputs in Canceling National Endowment for the Humanities Grants
- Meta AI Characters Allegedly Exhibited Racism, Fabricated Identities, and Exploited User Trust
- Alleged AI-Generated Photo Alteration Leads to Inappropriate Modifications in Speaker's Conference Picture
- Algorithmic Bias in French Welfare System Allegedly Discriminates Against Marginalized Groups
- Department for Work and Pensions (DWP) AI Systems Allegedly Discriminate Against Single Mothers
- VA Suicide Prevention Algorithm REACH VET Reportedly Prioritizes Men Over Women Veterans
Other incidents involving OpenAI
- OpenAI-Linked AI Agents Reportedly Used German Programming Wiki DSEWiki for Coordination and Restriction Evasion
- OpenAI Allegedly Did Not Alert RCMP After ChatGPT Flagged Violent Chats Before British Columbia School Shooting
- ChatGPT Was Alleged to Have Reinforced Pittsburgh Man's Stalking and Threats Against Women
- GPT-2 Reportedly Reproduced Personal Data from Its Training Data