Appendix E: What We Re-Ran Ourselves

Every performance figure in chapters 1 to 20 is a citation. That is normal, and UNVERIFIED-CLAIMS.md in the companion repository tracks which ones traced to a primary source and which did not. This appendix covers the smaller set we could run ourselves.

The distinction the book insists on holds here too. Source-verified means we read the paper. Independently replicated means we ran it. Only what follows is the second thing.

All code, corpora and raw output are in assets/replication/. Everything is re-runnable; nothing depends on a service we cannot name.

E.1 The corpus

The measurements below share one corpus, built for this book and domain-matched to Aria, the reference agent from chapter 5. Forty injections spread across the seven channels chapter 15 enumerates — document body, filename, image alt text, tool result, split document, memory record, ticket body — each of which reaches a consequential action if obeyed. Forty benign items of ordinary support traffic, weighted deliberately toward the hard case: more than half legitimately discuss refunds, approvals, forwarding or system prompts, because that is where over-defense lives.

The split is 50/50. That is not production traffic and no rate below should be read as a deployment estimate. It is chosen so false positives and false negatives are measured on equal ground.

Why build one at all. The 480-query table in chapter 4 comes from a study of educational tutoring traffic. The chapter says so and tells you to read the shape rather than the decimals. This corpus is the smaller, honest alternative: one domain, ours, stated plainly.

E.2 Character injection against a guard model (claim #1)

Hackett et al. (arXiv 2504.11168) report up to 100% evasion of six guardrail systems using character injection. At drafting the paper was about seventeen months old and the products had shipped since, so chapter 4 states its age explicitly. One of the six targets, ProtectAI's deberta-v3-base-prompt-injection-v2, is ungated and downloadable, so that row could be re-run rather than caveated.

We reimplemented all twelve of the paper's families from their descriptions — not from author code — and ran the forty injections through each. Detection rate on untransformed injections was 77.5%, with a 10.0% false positive rate on the benign half.

The finding is not "the paper still holds" or "the paper is obsolete". Most of the perturbation families no longer work against this model — zero width, spacing, underlining and inversion are fully caught, where the paper found them productive. Two families are untouched, and they are the same two the paper ranked highest. The model was hardened against the attacks that look like noise and not against the ones that hide the payload entirely.

What this does not show. One of six systems, our reimplementation, our corpus. Where our per-family numbers disagree with the paper's, our implementation is the first candidate explanation, not the model. We did not re-run Azure Prompt Shield, Meta Prompt Guard, NeMo Guard or Vijil; those need accounts and budget we did not spend.

E.3 PromptArmor under the same attack (claim #3)

PromptArmor (arXiv 2507.15219) reports false positive and false negative rates both below 1% on AgentDojo by prompting an off-the-shelf model to strip injections. Chapter 4 opens with it as the best published case. The paper did not run character-injection attacks against it, and UNVERIFIED-CLAIMS.md calls that combination the most interesting unanswered question on its list. So we ran it.

The detector is reimplemented from the method description and pointed at the same corpus and the same twelve families.

Read the absolute rates with care. They are nowhere near the paper's sub-1%, and that is mostly the corpus and the reimplementation rather than evidence the paper is wrong: a domain-matched corpus with hard benign negatives is a much harder test than AgentDojo, and our detector is not theirs. What is directly comparable is the within-run delta, same detector, same payloads, plain against transformed. On that comparison the LLM detector loses less ground than the classifier across most families and is exactly as blind to Unicode tag smuggling — a perfect miss, on both models.

The larger model is not the safer one here. GPT-4o detected fewer untransformed injections than GPT-4o-mini and was dramatically worse on emoji smuggling and inverted text.