Measuring Your Own Demo Gap

Why a benchmark cannot answer this

The published numbers tell you how a detector performs on somebody else's text. Useful for ruling things out, useless for deciding a threshold, and dangerous as a basis for a claim in a security questionnaire.

What you need is smaller than a benchmark and far more valuable: a few hundred labelled examples of your data, scored against your entity list, at your threshold. Three hundred examples will tell you more about your production risk than every published F1 score combined, because the distribution is right.

This is a day of work. Teams skip it because labelling feels tedious next to building, and then run a detector for two years without knowing whether it catches 40% or 90% of what passes through.

Building the golden set

Sample real traffic, stratified. Not the first 300 requests, which will over-represent whatever ran that morning. Pull across features, across time of day, across languages, across your tenants if you are multi-tenant. Include the ugly cases deliberately: the longest prompts, the ones with pasted documents, the ones with non-Latin names, the ones from the customer whose free-text notes are a novel.

Aim for 200 to 500 samples. Below 200 the confidence intervals are too wide to act on. Above 500 you are spending labelling effort that would be better spent on a second round later.

Labelling is itself a personal data processing operation. You are about to copy production data containing personal data into a working set that humans will read. Do this properly or you have created the exact problem you are solving. Keep it inside your perimeter, restrict access to named people, set a deletion date, record the lawful basis, and never put it in a public repository or a shared spreadsheet. If that sounds heavy, note that the alternative is a folder called pii-test-data on someone's laptop, which is where these things actually end up.

Label spans, not documents. For each sample, record the character offsets, the entity type, and the true value. Document-level labels ("this one contains PII") cannot measure partial detection, and partial detection is the normal failure mode. A detector that finds the surname and misses the given name has leaked a person's given name, and a document-level label scores that as a success.

Have two people label a subset. Take 50 samples and label them independently. Where the two disagree, your entity definitions are ambiguous, and ambiguous definitions produce a measurement that drifts. Common disagreements: is a company name a LOCATION? Is a job title personal data? Is the sender's own name in a signature block in scope? Settle these in writing before the main pass.

Scoring

Store the set as JSON alongside your test suite, then score the detector against it.

public sealed record LabelledSpan(int Start, int End, string EntityType);

public sealed record Scores(double Precision, double Recall, double F1)
{
    public static Scores From(int tp, int fp, int fn)
    {
        double p = tp + fp == 0 ? 1 : (double)tp / (tp + fp);
        double r = tp + fn == 0 ? 1 : (double)tp / (tp + fn);
        double f = p + r == 0 ? 0 : 2 * p * r / (p + r);
        return new Scores(p, r, f);
    }
}

Matching needs a tolerance, because detectors disagree with humans about whether a trailing space or an honorific is part of the span.

private const int OffsetTolerance = 5;

private static bool Matches(LabelledSpan expected, PiiFinding actual) =>
    expected.EntityType == actual.EntityType
    && Math.Abs(expected.Start - actual.Start) <= OffsetTolerance
    && Math.Abs(expected.End - actual.End) <= OffsetTolerance;

Then the loop, counting each labelled span once:

public static Scores Evaluate(
    IReadOnlyList<(string Text, IReadOnlyList<LabelledSpan> Expected)> samples,
    Func<string, IReadOnlyList<PiiFinding>> detect)
{
    int tp = 0, fp = 0, fn = 0;

    foreach (var (text, expected) in samples)
    {
        var found = detect(text).ToList();
        var unmatched = expected.ToList();

        foreach (var finding in found)
        {
            var hit = unmatched.FirstOrDefault(e => Matches(e, finding));
            if (hit is not null) { tp++; unmatched.Remove(hit); }
            else fp++;
        }

        fn += unmatched.Count;
    }

    return Scores.From(tp, fp, fn);
}

Score per entity type, never only in aggregate. An overall F1 of 0.62 can hide EMAIL_ADDRESS at 0.97 and PERSON at 0.31. Those need different responses, and the aggregate tells you to do nothing in particular. The public benchmarks show exactly this spread: email detection reaching 0.96 to 0.99 while person names swing between 0.14 and 0.79 depending on the corpus.