AIPolicyTracker
Menu

MIT AI Risk Repository · Risk Sub-Category · 16.05.01

Promoting harmful stereotypes by implying gender or ethnic identity

Category: Risk area 5: Human-Computer Interaction Harms

Open the MIT Risk Navigator All entries from Weidinger2022

Description

"CAs can perpetuate harmful stereotypes by using particular identity markers in language (e.g. referring to “self” as “female”), or by more general design features (e.g. by giving the product a gendered name such as Alexa). The risk of representational harm in these cases is that the role of “assistant” is presented as inherently linked to the female gender [19, 36]. Gender or ethnicity identity markers may be implied by CA vocabulary, knowledge or vernacular [124]; product description, e.g. in one case where users could choose as virtual assistant Jake - White, Darnell - Black, Antonio - Hisp

From Taxonomy of Risks posed by Language Models (Weidinger2022), as extracted by the MIT AI Risk Repository (CC BY 4.0).

Classification

Causal entity
AI

Subdomain definition: Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and representation of those groups.

Real-world incidents in this subdomain

Browse all incidents in this subdomain

How other frameworks describe this risk

  • Discrimination

    A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms (Abercrombie2024)

  • Harms of Representation and Other Biases

    Foundational Challenges in Assuring Alignment and Safety of Large Language Models (Anwar2024)

  • Risks from bias and underrepresentation

    International Scientific Report on the Safety of Advanced AI (Bengio2024)

  • Bias

    International AI Safety Report 2025 (Bengio2025)

  • Bias

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Toxicity and Bias Tendencies

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Biased Training Data

    Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems (Cui2024)

  • Broken systems

    Navigating the Landscape of AI Ethics and Responsibility (Cunha2023)

Other entries from Weidinger2022