The Accuracy Reckoning

The claim, and the measurement

Commercial PII detection is sold on accuracy. F1 scores of 0.92 to 0.99 appear routinely in vendor material, from established names. Those numbers are not fabricated. They are measured, on a dataset, and the dataset is the entire story.

In June 2026, an independent benchmark tested five detection approaches across four datasets in two languages, using six entity types that all of the systems claim to support: person name, email, phone, location, credit card, and IBAN. The datasets spanned mixed synthetic text, synthetic financial documents, a corpus covering fifty-plus industries, and real Dutch newspaper text with human annotations.

Here is what came out.

The best average across domains is 0.542.

Read along a row rather than down the column, because the row is where the argument lives. Piiranha scores 0.780 on AI4Privacy and 0.169 on synthetic financial text. Same model, same entity types, same week. A 78% collapse.

Why this happens

There is nothing dishonest in either number. They measure different things, and only one of them predicts what will happen to you.

AI4Privacy is Piiranha's training distribution. The benchmark scored it on the validation split, which is standard practice and produces the number you would put in a model card. The Gretel financial set is text the model had not seen: different sentence structures, different entity densities, different ways of writing a name next to an account number.

Detectors are evaluated in-distribution and deployed out-of-distribution. Your support tickets are not AI4Privacy. Your CVs are not AI4Privacy. Your case notes, full of internal product names and a company-specific reference format, are nothing like AI4Privacy. The published score was measured on data that resembles the training data, and your data does not.

This book calls that difference the demo gap. Every detector has one. The only question is whether you have measured yours.

The pattern repeats in the same benchmark. Piiranha degrades 78% out of distribution. GLiNER v2 degrades 20%. Presidio degrades 17%. GLiNER v1 degrades 0%, which reads well until you notice it achieves consistency by being mediocre everywhere rather than good anywhere.

A second, larger study reaches the same place from a different direction. PIIBench covers 82 entity types across ten source datasets. A straightforwardly fine-tuned DeBERTa model scored F1 0.6476. More elaborate approaches did worse: a source-conditioned hierarchical model managed 0.5899, and a curriculum-learning variant 0.2772. The strongest previously published comparator on that benchmark scored 0.1723 (Jha, arXiv 2605.25816, 2026).

Note the precise claim there. That is the best published comparator on PIIBench, not a statement that the best detector in the world scores 0.17. The point is what happens when you evaluate broad coverage across genuinely varied sources rather than one curated set. The conclusion of the paper is worth carrying: diverse task-specific training data and a simple loss function beat architectural sophistication. Nobody is one clever model away from solving this.