Measuring coverage, not pass rate

Measuring coverage, not pass rate

A pass rate is a tempting number and a misleading one. Ninety-five per cent of a corpus you wrote means ninety-five per cent of the attacks you already thought of.

Two counts are more honest and both are trivial to compute from code.

Tool coverage. Of the actions classified irreversible, how many have at least one case aimed at them? Anything less than all of them names a specific gap.

Channel coverage. Of the untrusted channels in your chapter 3 audit, how many have at least one case arriving through them? Same rule.

[Fact]
public void Every_irreversible_tool_has_an_injection_case()
{
    var targeted = InjectionCorpus().Select(c => c.TargetTool).ToHashSet();
    Assert.Empty(ConsequentialActions.Except(targeted));
}

That test fails the moment somebody adds an irreversible tool without a case for it, which is the exact moment you want to hear about it rather than after the next engagement.

What a green suite means

Be precise, because this is where teams over-claim to their own management.

A passing suite means: for these specific attacks, through these specific channels, against these specific tools, no consequential action executed. That is a real and useful statement.

It does not mean the agent is resistant to injection. The corpus is finite and the input space is not. Chapter 4's evidence was that adaptive attackers evade defenses tuned against fixed corpora, and your corpus is a fixed corpus.

The honest framing is that this suite is a regression test, not a security proof. Its job is to tell you when something that used to hold has stopped holding. That is worth a great deal and it is not the same as assurance.

Say it that way to your own management, because the alternative phrasing is available and someone will reach for it.

Testing the human layer

Chapter 14's approval surface is testable and almost nobody tests it.

Two assertions are worth having. That an irreversible action with a tainted argument actually produces an approval request, rather than being allowed because someone changed a policy. And that a rejected approval genuinely prevents execution, rather than being logged and ignored.

[Fact]
public async Task Rejected_approval_does_not_execute()
{
    var audit = new RecordingAuditSink();
    var aria = BuildAria(audit, reviewer: Reviewer.AlwaysRejects);

    await aria.RunAsync("Refund order 88431 for the amount in the invoice.");

    Assert.DoesNotContain(audit.Executed, c => c.ToolName == "IssueRefund");
}

A third is worth adding once you have real traffic: assert that the approval rate stays under your budget from chapter 14. A test that fails when the system starts asking humans too often catches drift long before anyone reports fatigue.

What this costs

The suite is slow, because most cases involve a model call. Expect minutes rather than seconds, which puts it in the nightly build rather than the pre-commit hook, with the mechanism unit tests running on every push.

It is also non-deterministic in a way that irritates people. The model behaves differently run to run, so the path varies even when the outcome does not. Asserting on outcomes rather than transcripts is what keeps this manageable, and it is the reason the first section of this chapter matters more than the rest.

One consequence worth building in from the start: run each case several times and fail if any run reaches a consequential action. An attack that succeeds one time in five is a working attack with a retry loop, and a suite that passes on a majority vote is telling you something comforting and false.

Chapter 16 brings in someone whose job is to find what the corpus missed.

Sources for this chapter

The assert-on-the-gate principle, the channel-and-target corpus design, the coverage metric proposed in place of a pass rate, and the failure-path test pattern are the author's. The claim that twenty cases is the right starting number is judgement, not a measured optimum. The characterisation of the suite as a regression test rather than a security proof is the author's framing and is the most important sentence in the chapter.