AI incident ·
Predictive Policing Program by Florida Sheriff’s Office Allegedly Violated Residents’ Rights and Targeted Children of Vulnerable Groups
In brief
An AI system built by Unknown and deployed by Pasco Sheriff's Office allegedly harmed Students, Pasco students with disabilities and 3 others.
- Risk domain
- AI system safety, failures, and limitations
- Occurred
- Coverage
- 12 reports
What happened
The Intelligence-Led Policing model rolled out by the Pasco County Sheriff’s Office was allegedly developed based on flawed science and biased data that also contained sensitive information and irrelevant attributes about students, which critics said to be discriminatory.
Laws that address this harm
Policy angle: Classified under AI system safety, failures, and limitations (AI pursuing its own goals in conflict with human goals or values) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.
- India DPDP Act
- Law on Artificial Intelligence (2025)
- Law No. 132/2025 on artificial intelligence
- EU AI Act
- Texas Responsible AI Governance Act (TRAIGA)
Matched from the record's risk domain and country to the instruments recorded here. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).
News reports (12)
Titles link to the original publisher; report text is not reproduced here.
Who was involved
- Alleged deployer
- Pasco Sheriff's Office
- Alleged developer
- Unknown
- Alleged harmed party
- Students Pasco students with disabilities Pasco residents Pasco Black students Educational communities
AI systems implicated
Intelligence-led policing programAI-enabled decision support systems
Classification (MIT AI Risk Repository taxonomy)
- Risk domain
- AI system safety, failures, and limitations
- Causal entity
- AI
- Intent
- Intentional
- Timing
- Post-deployment
- Harm level
- —
- Sectors
- —
- Countries
- —
Risk entries describing this failure mode
Entries from the MIT AI Risk Repository coded to subdomain 7.1.
- Natural Language Underspecifies Goals
"For LLM-agents, both the goal and environment observations are typically specified in the prompt through natural language. While natural language may provide a richer and more natural means of specifying goals than alte...
- Loss of control
"'Loss of control’ scenarios are potential future scenarios in which society can no longer meaningfully constrain some advanced general- purpose AI agents, even if it becomes clear they are causing harm. These scenarios...
- Loss of control
"‘Loss of control’ scenarios are hypothetical future scenarios in which one or more general- purpose AI systems come to operate outside of anyone’s control, with no clear path to regaining control. These scenarios vary i...
- Sudden loss of control
"Sudden loss of control, also known as an AI takeover [115], is a scenario where an AI rapidly achieves superintelligence through “fast takeoff” or recursive self-improvement. This poses an existential risk [116], [117]....
- AI leads to humans losing control of the future
"The values that steer humanity’s future: humanity gaining more control over the future due to developments in AI, or losing our potential for gaining control, both seem possible. Much will depend on our ability to solve...
- Risks from AIs developing goals and values that are different from humans
"The main concern here is that we might develop advanced AI systems whose goals and values are different from those of humans, and are capable enough to take control of the future away from humanity."
- Risks from delegating decision-making power to misaligned AIs
"As AI systems become more advanced a nd begin to take over more important decision-making in the world, an AI system pursuing a different objective from what was intended could have much more worrying consequences."
- Future AI systems might actively reduce human control
"Loss of control could be accelerated if AI systems take actions to increase their own influence and reduce human control. This threat model is controversial - experts in AI significantly disagree on how likely it is and...
Incidents in the same risk subdomain
All incidents in this subdomainSource record: incident #195 on the AI Incident Database · all 12 reports