AI incident ·
Algorithmic Bias in French Welfare System Allegedly Discriminates Against Marginalized Groups
In brief
An AI system built by Government of France and deployed by Government of France and Caisse Nationale des Allocations Familiales (CNAF) allegedly harmed Women and girls, Women and 10 others.
- Risk domain
- Discrimination and Toxicity
- Occurred
- Coverage
- 2 reports
What happened
A coalition of 15 human rights groups has launched legal action against the French government alleging that an algorithm used to detect welfare fraud discriminates against single mothers and disabled people. The algorithm assigns risk scores based on personal data. The process allegedly subjects vulnerable recipients to invasive investigations, violates privacy and anti-discrimination laws, and disproportionately affects marginalized groups.
Editor's notes
Reconstructing the timeline of events: (1) Since the 2010s: The algorithm has been in use to detect errors and fraud in France’s welfare system. (2) 2014: One version of the algorithm scored single-parent families, particularly those recently divorced, and disabled individuals receiving the Allocation Adulte Handicapé (AAH) as higher risk. (3) 2020: A suspected update to the algorithm took place, though the CNAF has not publicly shared the source code of the current model. (4) October 15, 2024: A coalition of 15 human rights groups, including La Quadrature du Net and Amnesty International, filed a legal challenge in France’s top administrative court, arguing the algorithm discriminates against marginalized groups.
Laws that address this harm
Policy angle: Classified under Discrimination and Toxicity (Unfair discrimination and misrepresentation) in the MIT AI Risk Repository taxonomy; 5 recorded instruments address this use case.
- Colorado AI Act
- India DPDP Act
- Law No. 132/2025 on artificial intelligence
- NYC Local Law 144 (automated employment decision tools)
- EU AI Act
Matched from the record's risk domain and country to the instruments recorded here. A reviewer can correct the match in the repository (data/external/incident_overrides.yaml).
News reports (2)
Titles link to the original publisher; report text is not reproduced here.
Who was involved
- Alleged developer
- Government of France
- Alleged harmed party
- Women and girls Women Single parents Single mothers in France Single mothers Privacy People with disabilities in France People with disabilities Parents General public of France General public Allocation Adulte Handicapé recipients
AI systems implicated
Welfare fraud detection algorithmsPublic benefits risk-scoring systemsCNAF risk-scoring algorithmAI-enabled decision support systems
Classification (MIT AI Risk Repository taxonomy)
- Risk domain
- Discrimination and Toxicity
- Risk subdomain
- 1.1 Unfair discrimination and misrepresentation
- Causal entity
- AI
- Intent
- Intentional
- Timing
- Post-deployment
- Harm level
- —
- Sectors
- —
- Countries
- —
Risk entries describing this failure mode
Entries from the MIT AI Risk Repository coded to subdomain 1.1.
- Discrimination
"Discrimination - Unfair or inadequate treatment or arbitrary distinction based on a person’s race, ethnicity, age, gender, sexual preference, religion, national origin, marital status, disability, language, or other pro...
- Harms of Representation and Other Biases
"A pretrained LLM generally has many of the stereotypical biases commonly present in the human society (Touvron et al., 2023). This makes it difficult for users to trust that LLMs will work well for them and not produce...
- Risks from bias and underrepresentation
"The outputs and impacts of general- purpose AI systems can be biased with respect to various aspects of human identity, including race, gender, culture, age, and disability. This creates risks in high- stakes domains su...
- Bias
"General-purpose AI systems can amplify social and political biases, causing concrete harm. They frequently display biases with respect to race, gender, culture, age, disability, political opinion, or other aspects of hu...
- Biased Training Data
"Compared with the definition of toxicity, the definition of bias is more subjective and contextdependent. Based on previous work [97], [101], we describe the bias as disparities that could raise demographic differences...
- Toxicity and Bias Tendencies
"Extensive data collection in LLMs brings toxic content and stereotypical bias into the training data."
- Bias
"The training datasets of LLMs may contain biased information that leads LLMs to generate outputs with social biases"
- Broken systems
"These are the most mentioned cases. They refer to situations where the algorithm or the training data lead to unreliable outputs. These systems frequently assign disproportionate weight to some variables, like race or g...
Incidents in the same risk subdomain
- DOGE Reportedly Relied on Unvetted ChatGPT Outputs in Canceling National Endowment for the Humanities Grants
- Sora Video Generator Has Reportedly Been Creating Biased Human Representations Across Race, Gender, and Disability
- Meta AI Characters Allegedly Exhibited Racism, Fabricated Identities, and Exploited User Trust
- Alleged AI-Generated Photo Alteration Leads to Inappropriate Modifications in Speaker's Conference Picture
- Department for Work and Pensions (DWP) AI Systems Allegedly Discriminate Against Single Mothers
- VA Suicide Prevention Algorithm REACH VET Reportedly Prioritizes Men Over Women Veterans
Source record: incident #822 on the AI Incident Database · all 2 reports