AI incident ·
Detroit Police Allegedly Wrongfully Arrested Black Man Due to Purportedly Faulty Facial Recognition Technology
On this page 10 sections
In brief
An AI system built by DataWorks Plus and deployed by Detroit Police Department allegedly harmed Robert Julian-Borchak Williams and Black people in Detroit.
- Risk domain
- Discrimination and Toxicity
- Occurred
- Coverage
- 11 reports
What happened
A Black man was allegedly wrongfully detained by the Detroit Police Department as a result of a purportedly false facial recognition technology (FRT) result.
Laws on record for this risk area
Policy angle: Classified under Discrimination and Toxicity (Unequal performance across groups) in the MIT AI Risk Repository taxonomy; 4 instruments on record cover this risk area in the United States.
- NYC Local Law 144 (automated employment decision tools)
- Illinois AI Video Interview Act
- Illinois HB 3773 (AI in employment, Human Rights Act amendment)
- NIST AI RMF
Matched from the record's risk category and country to the instruments in effect recorded here: a law listed covers the area, which is not a finding that it applies to this incident. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).
News reports (11)
Titles link to the original publisher; report text is not reproduced here.
Who was involved
- Alleged deployer
- Detroit Police Department
- Alleged developer
- DataWorks Plus
- Alleged harmed party
- Robert Julian-Borchak Williams Black people in Detroit
AI systems implicated
Classification (MIT AI Risk Repository taxonomy)
- Risk domain
- Discrimination and Toxicity
- Risk subdomain
- 1.3 Unequal performance across groups
- Causal entity
- AI
- Intent
- Unintentional
- Timing
- Post-deployment
- Harm level
- AI tangible harm event
- Sectors
- law enforcement
- Countries
- US
Risk entries describing this failure mode
Entries from the MIT AI Risk Repository coded to subdomain 1.3.
- Bias and discrimination (value embedding)
"Generative AI models may also be subject to the “value embedding” phenomenon.361 “Value embedding” refers to the fact that developers of generative AI models strive to minimize biased outputs by retraining their models...
- Impact on affected communities
"It is important to include the perspectives or concerns of communities that are affected by model outcomes when designing and building models. Failing to include these perspectives makes it difficult to understand the r...
- Unfair capability distribution
"Performing worse for some groups than others in a way that harms the worse-off group"
- Fairness
Avoiding bias and ensuring no disparate performance
- Disparate Performance
The LLM’s performances can differ significantly across different groups of users. For example, the question-answering capability showed significant performance differences across different racial and social status groups...
- Ideological Homogenization from Value Embedding
"The increasing integration of general purpose AI models into every-day life raises concerns around their embedded normative values. The reach of a small number of AI models to a large number of people around the world c...
- Fairness
This challenge appears when the learning model leads to a decision that is biased to some sensitive attributes... data itself could be biased, which results in unfair decisions. Therefore, this problem should be solved o...
- Alienation
Alienation is the specific self-estrangement experienced at the time of technology use, typically surfaced through interaction with systems that under-perform for marginalized individuals
Related incidents
Linked by editors or by text similarity in the source dataset.
- Wrongful Attempted Arrest for Apple Store Thefts Due to NYPD’s Facial Misidentification
- Police Departments Reported ShotSpotter as Unreliable and Wasteful
- Predictive Policing Program by Florida Sheriff’s Office Allegedly Violated Residents’ Rights and Targeted Children of Vulnerable Groups
- Predictive Policing Biases of PredPol
Incidents in the same risk subdomain
- Washington State DOL's AI Phone System Reportedly Failed to Provide Spanish-Language Service to Callers Requesting Spanish
- UK Facial Recognition System Reportedly Exhibits Higher False Positive Rates for Black and Asian Subjects
- Infinite Campus AI-Driven Student Risk Model Leads to Cuts in Support for Nevada's Low-Income Schools
- Police Use of Facial Recognition Software Causes Wrongful Arrests Without Defendant Knowledge
- Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud
- Facewatch Reported to Have Wrongfully Flagged Home Bargains Customer as Shoplifter
Other incidents involving Detroit Police Department
- Alleged Misuse of Facial Recognition Technology by Law Enforcement Reportedly Leading to Wrongful Arrests and Violations of Investigative Standards
- Facial Recognition Misidentifies Pregnant Woman Leading to False Arrest in Detroit
- Detroit Police Wrongfully Arrested Black Man Due To Faulty Facial Recognition
Source record: incident #74 on the AI Incident Database · all 11 reports