If a model gets 0.54 and regex gets 0.17, the obvious move is to run both and take the union. Two independent methods, complementary strengths, better coverage.
The benchmark tested it. Adding regex detection to the model-based detectors produced:
Nothing. And it occasionally made things worse, because regex matches displaced correct model detections with false positives.
That result kills a reflex worth killing. The models have already internalised the easy structural patterns. Layering rules on top adds noise, not recall. Rules earn their place for a different reason, which Chapter 5 is about, and the reason is precision on checksummed identifiers rather than incremental F1.
Applying the chapter's own standard to itself.
Three of the four datasets are synthetic. Only CoNLL-2002 carries human annotations of real text, and it only covers two of the six shared entity types. Synthetic PII data has regularities that real data does not.
The sample is small. A paired t-test across the four datasets found no statistically significant difference between the top three models (p well above 0.10). Piiranha at 0.542 and Presidio at 0.481 are not reliably distinguishable on this evidence. Do not use this table to pick a winner by decimal places.
Span matching used a ±5 character tolerance, which is a judgement call that moves the numbers.
And these are six entity types. A production system usually wants twenty or more, including ones with no public benchmark at all, like your internal customer reference format.
None of that rescues the headline. The caveats affect the ranking. They do not touch the finding, which is that cross-domain PII detection lands somewhere around 0.5 rather than somewhere around 0.95, and that the gap between published and achieved is large enough to invalidate a design that assumes the published number.
F1 is a harmonic mean of precision and recall, which makes it a poor number for making decisions, because it hides the trade-off you actually care about.
Take a detector with recall of 0.85 on your traffic, which would be a good day. You process 10,000 prompts. Each contains an average of three personal data entities. That is 30,000 entities, and 4,500 of them go through untouched.
Not 4,500 catastrophes. Most will be a name in a context where the name was harmless. Some proportion will be the account number, the medical detail, the home address.
Now raise the threshold to catch them. Recall goes up, precision goes down, and you begin redacting things that were not personal data: the product name that looked like a surname, the order reference that looked like a national ID, the word "Paris" in a sentence about an office. Your summaries get worse. Users notice, and users route around controls that make their tools worse, which is where Chapter 1 came in.
There is no threshold that makes both problems go away. That is the nature of a probabilistic classifier, and pretending otherwise is how teams end up surprised.
The wrong conclusion is that detection is pointless. The benchmark's own author, having produced these numbers, still chose Presidio for production. Not on accuracy, on architecture: a complete framework, a replaceable NER backend, a working anonymisation and de-anonymisation pipeline, active maintenance, and 8 to 11 times the speed of the transformer detectors (15.1 ms versus 118 to 198 ms per text on CPU).
That is exactly the right reasoning, and this book follows it. You pick a detector for how well it fits an architecture you can improve later, not for a decimal place you cannot reproduce.
The right conclusion is that detection changes job. It is not the wall. It is the second net: the thing that catches what slipped past a design that was already trying not to send the data. A second net that catches half of what reaches it is genuinely valuable, as long as you sized the rest of the system knowing that is what it does.
Every defensive decision in Part IV follows from this chapter. Fail closed when the detector is unavailable, because you cannot assume it was catching much. Minimise at source, because the field that never entered the prompt cannot be missed. Keep the identifier out of band, because then a miss is a miss on a token rather than on a person. Filter the output as well as the input, because the model may surface what the detector let through.
Detection is a net, not a wall.
Knowing that, build the best net you reasonably can. Start with the one layer that genuinely is close to certain.
Everything quantitative here comes from two independent sources. Both are worth reading directly if you intend to act on the conclusion.
The worked example of 10,000 prompts and 4,500 missed entities is arithmetic on an assumed recall of 0.85, presented as illustration. "Demo gap" is this book's coinage for a phenomenon the sources describe but do not name.
Download the full PDF for free?
Free download — no account required