Why this is structural

Why this is structural

Step back from the measurements, because the argument does not depend on any of them.

A filter is a classifier over an unbounded input space. The attacker has unlimited attempts, no rate limit worth the name, and can test offline against an approximation of your model. You must be right every time. They must be right once. That asymmetry is not a bug in current products and no amount of training data closes it.

There is a second problem underneath the first. Willison put it in five words: LLMs follow instructions in content. A language model has no mechanism for distinguishing an instruction from its operator from an instruction embedded in a document it was asked to read. Both arrive as tokens. Both are instruction-shaped. The distinction you want is about provenance, and provenance is metadata the model never sees.

You cannot ask a system to enforce a boundary it has no representation of.

Note what the best-performing detector in this chapter actually is. PromptArmor's method is to prompt a language model to spot injections. The strongest published defense against attacks on language models is another language model, and it is exactly the class of system the evasion research defeats. That is not a contradiction and PromptArmor's results stand. It does mean the ceiling on detection is set by the same property that created the vulnerability.

Four years after naming the problem, Willison's assessment is that "we still don't know how to 100% reliably prevent this from happening."

So use them, and do not rely on them

Detection is worth deploying. It raises the cost of casual attacks, it catches the copy-pasted payloads that make up most real traffic, and it produces signal for chapter 17. Run it.

Just do not let it be the thing standing between an injected model and your database. Put it in the same category as a spam filter: useful, constantly worked around, and never the reason you are comfortable.

The rest of this book assumes the filter failed and asks what happens next. Chapter 5 defines what a good answer to that question looks like.

Methodology note

The numbers above come from different evaluations and are not directly comparable. The 480-query table is a single controlled holdout with a 77/23 split of injection to benign traffic, which is far heavier in attacks than production. NotInject measures one specific failure, over-defense on trigger words, and does not name the models it tested, so nothing here attributes near-random benign accuracy to a named product. PromptArmor's sub-1% figures are on AgentDojo with three specific models. The evasion results are adaptive and white-box-informed, which is a different and harder threat model than any of the others.

They are presented together because they answer different questions. A fixed benchmark tells you how a defense handles known attacks. Only the adaptive work tells you how it handles an attacker who has read the manual, and that is the attacker you are building for.

Sources for this chapter

The asymmetry argument (unbounded input space, unlimited attempts, offline approximation) is the author's framing of why the measured results hold in general, not itself a research finding. The observation that false positives concentrate in content about the system is the author's, drawn from the trigger-word-bias mechanism rather than measured. The spam filter comparison is the author's, as is the reading of delimiters and instruction hierarchies as mitigations rather than boundaries. Figures reported elsewhere for PromptGuard and AgentWatcher were omitted because they could not be traced to a primary source before drafting.