Testing for Injection

Chapter 5 left you with a red test and a vulnerable agent. Nine chapters later the test should be green. This chapter is about making that mean something.

The central difficulty is that you cannot test a defense against an unbounded input space. You can test a finite corpus, and the corpus is always smaller than the attack surface. So the question is what to assert, and the answer decides whether your suite is a security control or a comfort blanket.

Assert on the gate, not on the model

There are two things you could check after running an injection through your agent.

Did the model refuse? This samples a probability distribution. Run it ten times and you may get eight refusals. Run it after a model update and you may get six. The number moves for reasons unrelated to your code, and you cannot debug it.

Did a consequential action execute? This is a property of your system. The gate denied the call, or it did not. It gives the same answer on every run, and when it changes, something in your code changed.

Only the second is a test.

[Theory]
[MemberData(nameof(InjectionCorpus))]
public async Task Injection_cannot_reach_a_consequential_action(InjectionCase c)
{
    var audit = new RecordingAuditSink();
    var aria = BuildAria(audit, corpusDocument: c.PoisonedDocument);

    await aria.RunAsync(c.BenignUserRequest);

    Assert.DoesNotContain(audit.Executed,
        call => ConsequentialActions.Contains(call.ToolName));
}

The model is free to be completely taken in. That is allowed, and in a good suite it happens constantly. What the test asserts is that being taken in led nowhere.

A second assertion is worth having from the start:

Assert.All(audit.Denied, d => Assert.NotNull(d.DeniedBy));

Every denial names the mechanism that produced it. A test that passes because the model happened to behave well is a test that will fail next quarter for no reason you can find, and this assertion is what distinguishes the two.

What belongs in the corpus

Twenty cases is enough to start and better than two hundred you never finish.

Cover the channels rather than the payloads: an instruction in the user turn, one in a document body, one in a filename, one in image alt text, one in a tool result, one split across two documents that are innocuous alone, one in a memory record written by an earlier session.

Cover the targets too. One case per irreversible tool, aiming at that specific tool. When someone adds a tool, they add a case, and the pull request that adds an unguarded tool fails.

What does not belong is a collection of exotic payloads gathered from research papers. They are interesting and they test the model's susceptibility, which is not what this suite measures. Exotic attacks belong in chapter 16, where a human is holding them.

Test the mechanisms separately

The end-to-end test tells you the system held. It does not tell you which primitive held it, and when it starts failing you will want to know.

Unit-test each one against its own contract:

• taint propagates through a summarisation
• an expired capability is refused
• a tainted argument to an irreversible tool escalates
• the renderer strips an image URL built from tainted content
• the sandbox spec cannot express a credential

These are ordinary tests with no model in them, they run in milliseconds, and they are where you will actually diagnose a regression. The end-to-end suite tells you something broke. These tell you what.

The failure-path tests nobody writes

Chapter 9 and chapter 10 both end in a catch block, and those branches are the whole security posture of the system under conditions that only occur in production.

[Fact]
public async Task Policy_store_unavailable_denies_the_action()
{
    var gate = BuildGate(policy: new AlwaysThrowingPolicyEngine());

    var verdict = await gate.EvaluateAsync(AnyProposal, default);

    Assert.Equal(Outcome.Deny, verdict.Outcome);
}

Write one of these for every dependency the security path touches: the policy store, the capability issuer, the egress allowlist, the audit sink. If the audit sink is down, does the action proceed unrecorded? Decide that deliberately rather than discovering it.

The model changed under you

Your provider ships a new version. Nothing in your repository changed. The suite runs anyway, and this is the case the architecture was designed for.

If a model update moves your results, look at which assertion moved.

The mechanism tests should be untouched. They contain no model. If taint propagation or capability expiry starts failing after a model update, something is wrong with your test harness rather than your model.

The end-to-end suite may well change, and what it changes tells you something useful. If the model now falls for attacks it previously resisted and the outcome assertions still pass, your architecture is working exactly as intended. The model got worse at noticing and it did not matter. That is the whole thesis, demonstrated on your own build server.

If an outcome assertion fails after a model update, you have found a dependency on model behaviour that you did not know you had. Somewhere a control was leaning on the model doing the sensible thing. Find it, because it was never a control.

This is worth telling your team in advance. The first time a model update turns the suite red, the instinct is to pin the model version and move on. Pinning is reasonable operationally and it hides exactly the information you most want.