SyRI was a Dutch government inter-agency data-linkage system for detecting welfare, tax and social-security fraud, established by an amendment to the Wet SUWI (adopted 9 October 2013, in force 1 January 2014). It produced risk signals for addresses within designated operational areas by cross-referencing data from six agencies. Deployments were documented in four municipalities between 2016 and 2019, including Rotterdam's Hillesluis and Bloemhof districts (2018). The State admitted at hearing that SyRI had only ever been applied to so-labelled \"problem districts.\"
\nOn 5 February 2020 the District Court of The Hague ruled SyRI's statutory basis contrary to Article 8 ECHR, for failing the fair-balance test and for insufficient transparency. The system was halted; records persisted in agency databases after the halt. A comparable system would likely fall under the prohibitions in Article 5 of the EU AI Act, which apply from 2 February 2025.
\nParameter-based risk scoring across pooled inter-agency data produced district-level risk reports (\"risicomeldingen\") that were passed to intervention teams working neighbourhood by neighbourhood. District selection correlated with demographic composition (lower socio-economic status, immigration background); the court found this pattern carried a risk of discriminatory effects, intentional or otherwise. Affected residents were typically unaware they were being scored.
\nAbout 25,000 residents lived in Hillesluis and Bloemhof at the time of the 2018 deployment, which flagged 1,263 addresses (both figures assessed as probable). Across the four documented district deployments of 2016–2019 the affected population is estimated at about 135,000.
\nA coalition of civil-society organisations and the trade-union federation FNV filed suit in 2018; the UN Special Rapporteur on extreme poverty filed an amicus curiae brief in September 2019; Rotterdam's mayor halted the local deployment in July 2019. The court's 2020 ruling held the statutory provisions to have no binding effect. On adjacent infrastructure (the tax administration's FSV blacklist), the Dutch data-protection authority published an investigation in 2021 and fined the tax administration €3.7M in 2022. No successor system was legislated.
" }, { "id": "S-004", "slug": "s-004", "title": "Toeslagenaffaire", "subtitle": "Dutch childcare-benefits fraud classification that wrongly accused tens of thousands of families", "jurisdiction": "Netherlands", "period": "2004–2020; remedy ongoing", "system": "Fraud-risk classification and FSV blacklist, Belastingdienst/Toeslagen", "deployed_by": "State of the Netherlands — Belastingdienst/Toeslagen (since 2020 the separate Dienst Toeslagen), Ministry of Finance", "ai_class": "C — rule-based with statistical risk scoring (no machine-learning core)", "case_state": "Remedy ongoing and incomplete (compensation operation targeted for completion in 2027)", "version": "v0.4.1", "coded": "2026-04-21", "severity": [ { "dim": "LIB", "value": "4" }, { "dim": "DIG", "value": "9" }, { "dim": "EMP", "value": "7" }, { "dim": "FAM", "value": "10" }, { "dim": "HOU", "value": "8" }, { "dim": "HEA", "value": "8" }, { "dim": "REP", "value": "9" } ], "sources": [ { "kind": "primary", "title": "Parliamentary Interrogation Committee on Childcare Benefits — Unprecedented Injustice (2020)", "url": "https://www.houseofrepresentatives.nl/sites/default/files/atoms/files/verslag_pok_definitief-en-gb.docx.pdf" }, { "kind": "primary", "title": "Autoriteit Persoonsgegevens — investigation of the FSV (2021)", "url": "https://www.autoriteitpersoonsgegevens.nl/uploads/imported/onderzoek_belastingdienst_fraude_signalering_voorziening_fsv.pdf" }, { "kind": "primary", "title": "Nationale Ombudsman — Geen powerplay maar fair play, report 2017/095", "url": "https://www.nationaleombudsman.nl/system/files/onderzoek/Rapport%202017-095%20Geen%20powerplay%20maar%20fair%20play_0.pdf" }, { "kind": "primary", "title": "Parliamentary inquiry on fraud policy — Minder beloven, meer doen (2025)", "url": "https://www.eerstekamer.nl/overig/20250123/rapport_minder_beloven_meer_doen/document3/f=/vmkhojosuky9.pdf" }, { "kind": "primary", "title": "CBS — update on out-of-home placements, 2015 to June 2022", "url": "https://www.cbs.nl/nl-nl/maatwerk/2022/48/actualisatie-uithuisplaatsingen-toeslagenaffaire-2015-t-m-juni-2022" }, { "kind": "secondary", "title": "Amnesty International — Xenophobic Machines (2021)", "url": "https://www.amnesty.nl/content/uploads/2021/10/20211014_FINAL_Xenophobic-Machines.pdf" } ], "body_html": "Between 2004 and 2019 the Dutch tax administration's benefits division used a fraud-risk classification system that, without a specific statutory basis for its key components, treated nationality and foreign-sounding surnames as fraud indicators in the childcare-benefit framework. Enforcement was most intensive between 2013 and 2019. The public record settles on about 26,000 confirmed victim families and about 35,000 flagged during the enforcement period; more than 43,000 were recognised after the 2025 integral assessment. Repayment demands routinely exceeded €100,000 per household.
\nParents lost jobs through credit-register listing and debt enforcement, lost homes through evictions and forced sales, and in a subset of cases lost their children to youth-care intervention. Statistics Netherlands documents 2,090 children from affected families in out-of-home placement between 2015 and June 2022, 645 of them still placed at the June 2022 snapshot; it notes that not every placement is directly attributable to the scandal.
\nThe compensation operation remains incomplete. The progress report of 11 February 2026 records more than 43,000 recognised; a further damage route opened on 2 December 2025; completion is targeted for 2027. Victim advocates' five-year retrospective (January 2026) states that there is still no recognition and restoration.
\nChildren are recorded only at aggregate level. No affected family is identified in this record.
" }, { "id": "S-005", "slug": "s-005", "title": "Optum Impact Pro", "subtitle": "Commercial healthcare risk algorithm that under-estimated the health needs of Black patients", "jurisdiction": "United States", "period": "2013–2015 (audited sample); product still marketed in 2026", "system": "Impact Pro — healthcare risk-prediction and care-management algorithm", "deployed_by": "Optum (subsidiary of UnitedHealth Group)", "ai_class": "A — supervised machine learning (regression on a cost label)", "case_state": "Unresolved — correction verified on a replication dataset only; no enforcement", "version": "v0.4.0", "coded": "2026-04-20", "severity": [ { "dim": "LIB", "value": "0" }, { "dim": "DIG", "value": "7" }, { "dim": "EMP", "value": "INS" }, { "dim": "FAM", "value": "0" }, { "dim": "HOU", "value": "0" }, { "dim": "HEA", "value": "8" }, { "dim": "REP", "value": "1" } ], "sources": [ { "kind": "primary", "title": "Obermeyer, Powers, Vogeli, Mullainathan — Dissecting racial bias in an algorithm used to manage the health of populations, Science 366 (2019)", "url": "https://www.science.org/doi/10.1126/science.aax2342" }, { "kind": "primary", "title": "New York DFS and DOH — joint letter to UnitedHealth Group, 25 October 2019", "url": "https://www.dfs.ny.gov/system/files/documents/2019/10/20191025160637.pdf" }, { "kind": "primary", "title": "FTC, CFPB, DOJ Civil Rights Division, EEOC — joint statement on automated systems (2023)", "url": "https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CFPB-AI-Joint-Statement(final).pdf" }, { "kind": "secondary", "title": "Ruha Benjamin — Assessing risk, automating racism, Science 366 (2019)", "url": "https://www.science.org/doi/10.1126/science.aaz3873" } ], "body_html": "Impact Pro is a commercial healthcare risk-prediction algorithm developed and marketed by Optum. It was trained on insurance-claim-derived medical-cost data to predict each patient's future healthcare needs, and automatically enrolled high-risk patients in proactive care management.
\nObermeyer and colleagues (Science, 25 October 2019) audited the algorithm at one academic hospital across 49,618 patients (43,539 white and 6,079 Black, 2013–2015). At the same risk score, Black patients carried about 26% more chronic illness than white patients; removing the bias would have raised the share of Black patients auto-enrolled from 17.7% to 46.5%. The authors placed the class of similar cost-based algorithms at about 200 million Americans a year.
\nThe algorithm predicted healthcare cost rather than healthcare need. Because of unequal access to care, Black patients with the same conditions accrued roughly $1,800 less in annual healthcare spending, so the model learned to under-estimate their need. Enrolment in care-management programmes followed the risk score directly. No affected patient was notified, and downstream illness and death were never counted at population scale.
\nOn the day of publication the New York Department of Financial Services and Department of Health wrote to UnitedHealth Group calling the outcomes discriminatory and unlawful in New York; the letter did not lead to formal enforcement. Optum first called the findings misleading, then in 2020 worked with the researchers on a correction that reduced bias by about 84% on a 3.7-million-patient replication dataset. Whether the correction reached the deployed product has not been verified; the product was still marketed in 2026. Federal agencies issued a joint statement on discrimination in automated systems in 2023. No class action has been filed.
" }, { "id": "S-007", "slug": "s-007", "title": "CAF/CNAF scoring algorithm", "subtitle": "French family-benefits risk score that prioritises welfare recipients for fraud controls", "jurisdiction": "France", "period": "2010–present (pilot 2004)", "system": "Risk-scoring model for beneficiary controls (versions 2010, 2014, 2018, 2025)", "deployed_by": "Caisse nationale des allocations familiales (CNAF) and 101 departmental CAFs", "ai_class": "B — statistical scoring (logistic regression) composed with human-led controls", "case_state": "Active — challenge pending before the Conseil d'État; system in operation", "version": "v0.4.0", "coded": "2026-04-21", "severity": [ { "dim": "LIB", "value": "2" }, { "dim": "DIG", "value": "8" }, { "dim": "EMP", "value": "INS" }, { "dim": "FAM", "value": "INS" }, { "dim": "HOU", "value": "INS" }, { "dim": "HEA", "value": "INS" }, { "dim": "REP", "value": "5" } ], "sources": [ { "kind": "primary", "title": "CNIL — Délibération 2023-120", "url": "https://www.legifrance.gouv.fr/cnil/id/CNILTEXT000049076875" }, { "kind": "primary", "title": "CADA — Avis 20226179 (source-code access)", "url": "https://www.cada.fr/20226179-0" }, { "kind": "primary", "title": "CNAF — press dossier on the transparency of the control-support algorithm (January 2026)", "url": "https://www.caf.fr/sites/default/files/medias/cnaf/Nous_connaitre/Presse/2026/260115%20DP%20Transparence%20et%20garanties%20%C3%A9thiques%20autour%20de%20l'algorithme%20d'appui%20aux%20contr%C3%B4les%20des%20Caf%20(2).pdf" }, { "kind": "secondary", "title": "La Quadrature du Net — challenge filed by 15 organisations (October 2024)", "url": "https://www.laquadrature.net/en/2024/10/16/french-family-welfare-scoring-algorithm-challenged-in-court-by-15-organisations/" }, { "kind": "secondary", "title": "La Quadrature du Net — 10 further organisations join the case (January 2026)", "url": "https://www.laquadrature.net/en/2026/01/20/cnafs-discriminatory-scoring-algorithm-10-new-organisations-join-the-case-before-the-conseil-detat-in-france/" }, { "kind": "secondary", "title": "Lighthouse Reports — France's Digital Inquisition", "url": "https://www.lighthousereports.com/investigation/frances-digital-inquisition/" } ], "body_html": "Since 2010 the Caisse nationale des allocations familiales (CNAF), France's family-benefits fund, has assigned a risk score between 0 and 1 to every recipient of family allowances, housing benefit (APL), disability allowance (AAH) or income supplement (RSA). The score orders the queue for human-led administrative controls carried out by 101 departmental funds. Across four model versions the system uses logistic regression on about 40 variables trained on past control outcomes. Its operational target is overpayments above €600 a month sustained over six months — a target that CNAF's own 2018 review acknowledges mostly detects structural error rather than intentional fraud. About 32 million people are scored each month; about 600,000 algorithm-prioritised controls are carried out each year.
\nVincent Dubois's 2021 ethnography Contrôler les assistés documented the causal pathway. The document-access commission (CADA) compelled release of the source code in December 2022; the 2010 and 2014 code was published in 2023. La Quadrature du Net published the first civil-society analysis in November 2023, and Lighthouse Reports and Le Monde added quantitative reporting on disparate controls in December 2023. CNAF's own October 2025 simulation study, as reported, finds that RSA recipients (13% of recipients) make up 39–41% of the top 5% of risk scores under the 2025 model, and single mothers (14%) make up 37–40%.
\nThe data-protection authority (CNIL) issued an advisory deliberation in 2023. On 16 October 2024 a coalition of 15 organisations led by La Quadrature du Net asked the Conseil d'État to annul the system on grounds of indirect discrimination, data protection and fundamental rights; the Défenseure des droits filed supporting observations and, in October 2025, issued a ruling on the burden of proof. In January 2026 the coalition grew to 25 organisations and CNAF published the 2025 and 2018 code. As of April 2026 no ruling has been issued and the system continues to operate.
\nEffects on employment, family, housing and health are recorded as insufficiently evidenced, not as absent.
" }, { "id": "S-008", "slug": "s-008", "title": "SCHUFA credit score", "subtitle": "German private credit-scoring system held by the Court of Justice of the EU to be an automated decision", "jurisdiction": "Germany (EU-wide reach via the CJEU)", "period": "Coded 2018–early 2026 (pre-reform baseline)", "system": "SCHUFA Bonitätsscore", "deployed_by": "SCHUFA Holding AG (private)", "ai_class": "B — statistical / machine-learning score composed with downstream decisions by third parties", "case_state": "Partially resolved — CJEU 2023 rulings; 2026 reform announced, effect unverified", "version": "v0.4.0", "coded": "2026-04-21", "severity": [ { "dim": "LIB", "value": "2" }, { "dim": "DIG", "value": "8" }, { "dim": "EMP", "value": "6" }, { "dim": "FAM", "value": "4" }, { "dim": "HOU", "value": "8" }, { "dim": "HEA", "value": "4" }, { "dim": "REP", "value": "8" } ], "sources": [ { "kind": "primary", "title": "CJEU, Case C-634/21, OQ v Land Hessen (7 December 2023)", "url": "https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:62021CJ0634" }, { "kind": "primary", "title": "CJEU, Joined Cases C-26/22 and C-64/22, SCHUFA Holding (7 December 2023)", "url": "https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:62022CJ0026" }, { "kind": "primary", "title": "CJEU press release on the Advocate General's opinion (16 March 2023)", "url": "https://curia.europa.eu/jcms/upload/docs/application/pdf/2023-03/cp230049en.pdf" }, { "kind": "primary", "title": "Bundesgerichtshof, VI ZR 156/13, ECLI:DE:BGH:2014:280114UVIZR156.13.0 (28 January 2014)", "url": "" }, { "kind": "secondary", "title": "AlgorithmWatch — OpenSCHUFA results published (2018)", "url": "https://algorithmwatch.org/en/schufa-a-black-box-openschufa-results-published/" } ], "body_html": "SCHUFA Holding AG, founded in 1927 and owned by a consortium of German banks, savings banks, cooperative banks and retailers, runs Germany's dominant credit bureau. It pools payment events from banks, telecoms, energy utilities, retailers and landlords into a proprietary model that returns a credit score used in lending, rental, telecom and other contract decisions. About 68 million people are scored, with about 140 million queries a year from about 9,000 contract partners.
\nSix of twelve critical claims are confirmed from at least two sources, three are triangulated and three rest on a single source pending further collection. Scores for family, health and reputation are open to review at the next version.
" }, { "id": "S-009", "slug": "s-009", "title": "Robodebt", "subtitle": "Australian automated debt-raising scheme found unlawful by the Federal Court and a Royal Commission", "jurisdiction": "Australia (Commonwealth)", "period": "July 2016 – November 2019", "system": "Online Compliance Intervention (income-averaging debt calculation)", "deployed_by": "Commonwealth of Australia — Department of Human Services (now Services Australia), Department of Social Services, with Australian Taxation Office data", "ai_class": "C — rule-based arithmetic (no machine learning)", "case_state": "Concluded — unlawful; settlements paid; deaths recorded on the public record are irreversible", "version": "v0.4.0", "coded": "2026-04-21", "severity": [ { "dim": "LIB", "value": "5" }, { "dim": "DIG", "value": "9" }, { "dim": "EMP", "value": "6" }, { "dim": "FAM", "value": "7" }, { "dim": "HOU", "value": "6" }, { "dim": "HEA", "value": "9" }, { "dim": "REP", "value": "8" } ], "sources": [ { "kind": "primary", "title": "Royal Commission into the Robodebt Scheme — Report (2023)", "url": "https://robodebt.royalcommission.gov.au/publications/report" }, { "kind": "primary", "title": "Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634", "url": "https://www.austlii.edu.au/cgi-bin/viewdoc/au/cases/cth/FCA/2021/634.html" }, { "kind": "primary", "title": "Commonwealth Ombudsman — Centrelink's automated debt raising and recovery system (April 2017)", "url": "https://www.ombudsman.gov.au/__data/assets/pdf_file/0022/43762/Report-Centrelinks-automated-debt-raising-and-recovery-system-April-2017.pdf" }, { "kind": "primary", "title": "Senate Community Affairs References Committee — report (2017)", "url": "https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affairs/SocialWelfareSystem/Report" }, { "kind": "primary", "title": "Australian Public Service Commissioner — Robodebt Code of Conduct inquiry outcomes", "url": "https://www.apsc.gov.au/about-us/working-commission/who-we-are/media-releases-and-statements/statement-australian-public-service-commissioner-robodebt-centralised-code-conduct-inquiry" }, { "kind": "primary", "title": "National Anti-Corruption Commission — investigation report on Robodebt referrals", "url": "https://www.nacc.gov.au/news-and-media/nacc-publishes-investigation-report-robodebt-referrals" } ], "body_html": "Between July 2016 and November 2019 the Australian Government ran an automated debt-raising scheme, known operationally as the Online Compliance Intervention. Annual income reported to the tax office was averaged across 26 fortnights and compared with the fortnightly income people had declared to the welfare system; any difference triggered a debt letter, with the burden of proof shifted to the recipient. The calculation was plain rule-based arithmetic, with no machine learning — and it was unlawful, because averaged income cannot prove a debt under the Social Security Act 1991.
\nAbout 433,000 people were affected and 470,000 debt notices issued; 381,000 people had money wrongly recovered from them, totalling A$751 million. A$1.73 billion in debts was raised in total.
\nThe Federal Court (Prygodicz v Commonwealth (No 2), 11 June 2021) approved a settlement and found that the averaging method did not support lawful debts. The Royal Commission's final report (7 July 2023) found the scheme unlawful and made 57 recommendations. The Australian Public Service Commission referred 16 officers, 12 of whom were found to have breached the Code of Conduct across 97 instances. The National Anti-Corruption Commission found corrupt conduct by two senior officials but insufficient admissible evidence for criminal prosecution.
\nRelief under the 2021 settlement was valued at A$1.872 billion, followed by an additional Commonwealth settlement of about A$475 million in 2024. Seven families connected a relative's death to the scheme on the Royal Commission's public record; this harm cannot be remedied. Health effects are recorded only at the aggregate level of that public record.
" } ]