SN Technologies Reportedly Lied to a New York State School District about Its Facial and Weapon Detection Systems’ Performance

Laws that address this harm

Policy angle: Classified under Discrimination and Toxicity (Unequal performance across groups) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case in the United States.

• Colorado AI Act Colorado (United States) · Repealed · binding
• NYC Local Law 144 (automated employment decision tools) New York (United States) · In force · binding
• Illinois AI Video Interview Act Illinois (United States) · In force · binding
• Illinois HB 3773 (AI in employment, Human Rights Act amendment) Illinois (United States) · In force · binding
• NIST AI RMF United States · Voluntary standard · guidance

Matched from the record's risk domain and country to the instruments recorded here. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).

News reports (1)

Titles link to the original publisher; report text is not reproduced here.

1. 1 Dec 2020
   Facial Recognition Company Lied to School District About its Racist Tech

   vice.com · Todd Feathers

Who was involved

Alleged deployer

Lockport City School District

Alleged developer

SN Technologies

Alleged harmed party

Students Epistemic integrity Educational communities Black students

AI systems implicated

Weapons detection systemFacial recognition systemsAI-enabled decision support systemsAEGIS

Classification (MIT AI Risk Repository taxonomy)

Risk domain

Discrimination and Toxicity

Risk subdomain

1.3 Unequal performance across groups

Causal entity

AI

Intent

Unintentional

Timing

Post-deployment

Harm level

AI tangible harm near-miss

Sectors

education, law enforcement, public administration

Countries

US

Risk entries describing this failure mode

Entries from the MIT AI Risk Repository coded to subdomain 1.3.

• Bias and discrimination (value embedding)

  "Generative AI models may also be subject to the “value embedding” phenomenon.361 “Value embedding” refers to the fact that developers of generative AI models strive to minimize biased outputs by retraining their models...

  Regulating under Uncertainty: Governance Options for Generative AI (G'sell2024)

• Impact on affected communities

  "It is important to include the perspectives or concerns of communities that are affected by model outcomes when designing and building models. Failing to include these perspectives makes it difficult to understand the r...

  AI Risk Atlas (IBM2025)

• Unfair capability distribution

  "Performing worse for some groups than others in a way that harms the worse-off group"

  A Closer Look at the Existing Risks of Generative AI: Mapping the Who, What, and How of Real-World Incidents (Li2025)

• Disparate Performance

  The LLM’s performances can differ significantly across different groups of users. For example, the question-answering capability showed significant performance differences across different racial and social status groups...

  Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models’ Alignment (Liu2024)

• Fairness

  Avoiding bias and ensuring no disparate performance

  Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models’ Alignment (Liu2024)

• Ideological Homogenization from Value Embedding

  "The increasing integration of general purpose AI models into every-day life raises concerns around their embedded normative values. The reach of a small number of AI models to a large number of people around the world c...

  Governing General Purpose AI: A Comprehensive Map of Unreliability, Misuse and Systemic Risks (Maham2023 )

• Fairness

  This challenge appears when the learning model leads to a decision that is biased to some sensitive attributes... data itself could be biased, which results in unfair decisions. Therefore, this problem should be solved o...

  A Survey of Artificial Intelligence Challenges: Analyzing the Definitions, Relationships, and Evolutions (Saghiri2022)

• Increased labor

  increased burden (e.g., time spent) or effort required by members of certain social groups to make systems or products work as well for them as others

  Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction (Shelby2023)

Incidents in the same risk subdomain

• 27 Feb 2026Washington State DOL's AI Phone System Reportedly Failed to Provide Spanish-Language Service to Callers Requesting Spanish
• 5 Dec 2025UK Facial Recognition System Reportedly Exhibits Higher False Positive Rates for Black and Asian Subjects
• 11 Oct 2024Infinite Campus AI-Driven Student Risk Model Leads to Cuts in Support for Nevada's Low-Income Schools
• 6 Oct 2024Police Use of Facial Recognition Software Causes Wrongful Arrests Without Defendant Knowledge
• 23 Jun 2024Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud
• 25 May 2024Facewatch Reported to Have Wrongfully Flagged Home Bargains Customer as Shoplifter