AI incident #815 ·
Police Use of Facial Recognition Software Causes Wrongful Arrests Without Defendant Knowledge
What happened
Police departments across the U.S. have used facial recognition software to identify suspects in criminal investigations, leading to multiple false arrests and wrongful detentions. The software's unreliability, especially in identifying people of color, has resulted in misidentifications that were not disclosed to defendants. In some cases, individuals were unaware that facial recognition played a role in their arrest, violating their legal rights and leading to unjust detentions.
Editor's notes (AI Incident Database)
This collective incident ID, based on a Washington Post investigation, details many harm events, the overarching theme of which is widespread facial recognition technology assisting in arrests made by police departments across the United States combined with a lack of transparency about the technology's use in making the arrests. Some of the documented incidents in the Washington Post's investigation are as follows: (1) 2019: Facial recognition technology used to misidentify Francisco Arteaga in New Jersey, which led to his wrongful detention for four years (see Incident 816). (2) 2020-2024: Miami Police Department conducts 2,500 facial recognition searches, leading to at least 186 arrests and 50 convictions. Less than 7% of defendants were informed of the technology's use. (3) 2022: Quran Reid is wrongfully arrested in Louisiana due to a facial recognition match, despite never visiting the state (see Incident 515). (4) June 2023: New Jersey appeals court rules that a defendant has the right to information regarding the use of facial recognition technology in their case. (5) July 2023: Miami Police Department acknowledges that they may not have informed prosecutors about the use of facial recognition in many cases. (6) October 6, 2024: The Washington Post publishes its investigation on these incidents and practices.
Only the incident metadata is stored here. The underlying news reports are on the AI Incident Database (CC BY-SA 4.0); use the links above to read them.
News reports (2)
Coverage catalogued by the AI Incident Database. Titles link to the original publisher; the text is not reproduced here.
Who was involved
- Alleged deployer
- West New York PD Police departments Pflugerville PD NYPD Miami PD Law enforcement Jefferson Parish Sheriff’s Office Facial recognition system deployers Evansville PD Coral Springs PD Arvada PD
- Alleged developer
- Facial recognition system developers Clearview AI
- Alleged harmed party
- Quran Reid Privacy People misidentified by facial recognition systems General public of the United States General public Francisco Arteaga Biometric data subjects
AI systems implicated
Facial recognition systemsAI-enabled decision support systems
Classification (MIT AI Risk Repository taxonomy)
- Risk domain
- Discrimination and Toxicity
- Risk subdomain
- 1.3 Unequal performance across groups
- Causal entity
- Human
- Intent
- Intentional
- Timing
- Post-deployment
- Harm level
- —
- Sectors
- —
- Countries
- —
Risk entries describing this failure mode
Entries from the MIT AI Risk Repository coded to subdomain 1.3.
- Bias and discrimination (value embedding)
"Generative AI models may also be subject to the “value embedding” phenomenon.361 “Value embedding” refers to the fact that developers of generative AI models strive to minimize biased outputs by retraining their models...
- Impact on affected communities
"It is important to include the perspectives or concerns of communities that are affected by model outcomes when designing and building models. Failing to include these perspectives makes it difficult to understand the r...
- Unfair capability distribution
"Performing worse for some groups than others in a way that harms the worse-off group"
- Disparate Performance
The LLM’s performances can differ significantly across different groups of users. For example, the question-answering capability showed significant performance differences across different racial and social status groups...
- Fairness
Avoiding bias and ensuring no disparate performance
- Ideological Homogenization from Value Embedding
"The increasing integration of general purpose AI models into every-day life raises concerns around their embedded normative values. The reach of a small number of AI models to a large number of people around the world c...
- Fairness
This challenge appears when the learning model leads to a decision that is biased to some sensitive attributes... data itself could be biased, which results in unfair decisions. Therefore, this problem should be solved o...
- Quality-of-Service Harms
"These harms occur when algorithmic systems disproportionately underperform for certain groups of people along social categories of difference such as disability, ethnicity, gender identity, and race."
Incidents in the same risk subdomain
- Washington State DOL's AI Phone System Reportedly Failed to Provide Spanish-Language Service to Callers Requesting Spanish
- UK Facial Recognition System Reportedly Exhibits Higher False Positive Rates for Black and Asian Subjects
- Infinite Campus AI-Driven Student Risk Model Leads to Cuts in Support for Nevada's Low-Income Schools
- Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud
- Facewatch Reported to Have Wrongfully Flagged Home Bargains Customer as Shoplifter
- Whisper Speech-to-Text AI Reportedly Found to Create Violent Hallucinations