UK passport photo checker shows bias against dark-skinned women

Laws that address this harm

Policy angle: Classified under Discrimination and Toxicity (Unequal performance across groups) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United Kingdom.

• UK AI regulation framework United Kingdom · Guidance · guidance
• ICO AI guidance United Kingdom · Guidance · guidance
• Colorado AI Act Colorado (United States) · Repealed · binding
• India DPDP Act India · Partially applicable · binding
• Law No. 132/2025 on artificial intelligence Italy · In force · binding

Matched from the record's risk domain and country to the instruments recorded here. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).

News reports (1)

Titles link to the original publisher; report text is not reproduced here.

1. 7 Oct 2020
   UK passport photo checker shows bias against dark-skinned women

   bbc.co.uk · Maryam Ahmed

Who was involved

Alleged deployer

UK Home Office Government of the United Kingdom

Alleged developer

UK Home Office Government of the United Kingdom

Alleged harmed party

Women and girls Women dark-skinned women dark-skinned people

AI systems implicated

UK passport photo checker

Classification (MIT AI Risk Repository taxonomy)

Risk domain

Discrimination and Toxicity

Risk subdomain

1.3 Unequal performance across groups

Causal entity

AI

Intent

Unintentional

Timing

Post-deployment

Harm level

none

Sectors

public administration

Countries

GB

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 1.3.

• Bias and discrimination (value embedding)

  "Generative AI models may also be subject to the “value embedding” phenomenon.361 “Value embedding” refers to the fact that developers of generative AI models strive to minimize biased outputs by retraining their models...

  Regulating under Uncertainty: Governance Options for Generative AI (G'sell2024)

• Impact on affected communities

  "It is important to include the perspectives or concerns of communities that are affected by model outcomes when designing and building models. Failing to include these perspectives makes it difficult to understand the r...

  AI Risk Atlas (IBM2025)

• Unfair capability distribution

  "Performing worse for some groups than others in a way that harms the worse-off group"

  A Closer Look at the Existing Risks of Generative AI: Mapping the Who, What, and How of Real-World Incidents (Li2025)

• Disparate Performance

  The LLM’s performances can differ significantly across different groups of users. For example, the question-answering capability showed significant performance differences across different racial and social status groups...

  Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models’ Alignment (Liu2024)

• Fairness

  Avoiding bias and ensuring no disparate performance

  Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models’ Alignment (Liu2024)

• Ideological Homogenization from Value Embedding

  "The increasing integration of general purpose AI models into every-day life raises concerns around their embedded normative values. The reach of a small number of AI models to a large number of people around the world c...

  Governing General Purpose AI: A Comprehensive Map of Unreliability, Misuse and Systemic Risks (Maham2023 )

• Fairness

  This challenge appears when the learning model leads to a decision that is biased to some sensitive attributes... data itself could be biased, which results in unfair decisions. Therefore, this problem should be solved o...

  A Survey of Artificial Intelligence Challenges: Analyzing the Definitions, Relationships, and Evolutions (Saghiri2022)

• Increased labor

  increased burden (e.g., time spent) or effort required by members of certain social groups to make systems or products work as well for them as others

  Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction (Shelby2023)

Related incidents

Linked by editors or by text similarity in the source dataset.

• 1 Jan 2014ETS Used Allegedly Flawed Voice Recognition Evidence to Accuse and Assess Scale of Cheating, Causing Thousands to be Deported from the UK
• 7 Dec 2016Passport checker Detects Asian man's Eyes as Closed
• 15 Oct 2019Opaque Fraud Detection Algorithm by the UK’s Department of Work and Pensions Allegedly Discriminated against People with Disabilities

Incidents in the same risk subdomain

• 27 Feb 2026Washington State DOL's AI Phone System Reportedly Failed to Provide Spanish-Language Service to Callers Requesting Spanish
• 5 Dec 2025UK Facial Recognition System Reportedly Exhibits Higher False Positive Rates for Black and Asian Subjects
• 11 Oct 2024Infinite Campus AI-Driven Student Risk Model Leads to Cuts in Support for Nevada's Low-Income Schools
• 6 Oct 2024Police Use of Facial Recognition Software Causes Wrongful Arrests Without Defendant Knowledge
• 23 Jun 2024Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud
• 25 May 2024Facewatch Reported to Have Wrongfully Flagged Home Bargains Customer as Shoplifter