Collection methodology

Status: working version, October 2026. Licence: CC-BY 4.0.

This document describes the automated pipeline that finds candidate registry records. The code is maintained privately; this description covers every decision it makes. Case-files (S-NNN) are not produced by this pipeline: they are researched and coded by hand (see Method).

Stages

collect → triage → score → select → extract → verify → publish

  1. Collect. Feeds and archives of civil-society organisations, investigative newsrooms, regulators and court databases are read on a schedule (among them AlgorithmWatch, Access Now, EFF, EDRi, Human Rights Watch, Big Brother Watch, Statewatch, AI Now, Bits of Freedom, Amnesty International, The Markup, Lighthouse Reports, CORRECTIV, CNIL, EDPB and CourtListener). The collector identifies itself honestly, respects robots.txt and limits itself to one request every three seconds per site. Near-duplicate articles are detected by text similarity and set aside.
  2. Triage. A language model (Claude Haiku) reads each article and decides whether it documents a deployed automated system that caused documented harm to real people. Most articles are rejected at this stage: of 4,934 articles screened by October 2026, 69 passed.
  3. Score. A rule-based verifiability score (0–100) counts signals such as links to court or regulator documents, case citations, publisher tier, and whether the system is independently documented. No language model is involved.
  4. Select. A person chooses which candidates go forward. A high score is a reason to look, not a decision.
  5. Extract. A language model (Claude Sonnet) checks relevance again, rates how concrete the account is (named actor, named system, identifiable people affected, documented outcome) and extracts structured fields with a confidence for each. Personal data that must not be stored is removed. Cited primary sources are collected from the article and from public references.
  6. Verify. A person reviews every field and every source against the documents. Only then can a record be marked verified. Duplicates of existing records are merged or deferred.
  7. Publish. Verified records are exported to the public data files. Records in review are published only as a count.

What the language models do and do not do

The models filter and structure. They do not decide what is published, do not assign severity, and do not write case-files. Their full prompts are published in Extraction prompts. In October 2026 one batch of records was structured with Claude (Anthropic) in an assisted review session instead of the automated extraction step; the same prompt and the same human check applied.

Known limitations