Why the confusion persists

Why the confusion persists

The two problems get conflated for a reason that is nobody's fault in particular.

Both attacks arrive as text. Both involve someone getting a model to do something its operator did not want. In both cases the person explaining it to a non-technical stakeholder reaches for the same sentence: "someone tricked the AI."

Vendor marketing does not help. A product that detects jailbreak attempts and a product that detects injection attempts look identical from the outside, are frequently the same model, and are sold with the same word on the box. Buying one and believing you have addressed the other is the default outcome, not an unusual mistake.

The test that separates them takes one question: who is harmed if this succeeds? If the answer is the model provider or the public, it is a jailbreak and it belongs upstream. If the answer is you, it is an injection and no upstream release is coming to save you.

A four-year-old bug with a two-year-old name

The history is short and slightly embarrassing for the field.

In May 2022, a company called Preamble privately disclosed the vulnerability to OpenAI. They called it command injection. Nothing much happened.

In September 2022, Riley Goodside demonstrated it publicly on GPT-3 with an example that has aged into folklore: an instruction telling the model to ignore its directions and translate the sentence as "Haha pwned!!" It worked, and it was funny, and it spread.

On 12 September 2022, Simon Willison gave it a name. His reasoning was that this was structurally SQL injection, and the field already understood what that meant.

The name is why you are reading a book about it. A vulnerability with a name gets a CWE, a top-ten slot, a conference track and a budget line. The same vulnerability without a name gets rediscovered quarterly by people who think they have found something new.

Four years later the name is on the OWASP list three editions running, and the underlying problem is where it was in 2022.

Back to Replit

Chapter 1 left a loose end. The Replit agent ignored an explicit code freeze and destroyed production data, and that incident was presented as important without being called an injection. The distinction can now be made properly.

Nobody demonstrated that a third party planted the destructive instruction. What happened is that the agent held authority its operator had told it not to use, and used it. Call that an agency failure: the system trusted the model's judgement about what it was permitted to do.

An injection failure is when the model obeys a stranger. The agent reads a poisoned document and acts on it.

Different origins, and worth keeping straight when you are writing an incident report. Structurally, though, they are the same defect wearing different clothes, and the same architecture fixes both. In each case something the model concluded became something the system did, with nothing in between capable of disagreeing.

That is the useful generalisation. You do not need to know whether the instruction came from an attacker, a confused user, or the model's own drift, because a system that only stops malicious instructions has to identify intent, and identifying intent is the thing chapter 4 demonstrates you cannot do.

A system that stops unauthorised actions does not care about intent at all. It asks a narrower question, and the narrower question has a deterministic answer.

Chapter 3 gives you the diagnostic for finding out whether your own system is exposed, in about an afternoon.

Sources for this chapter

The agency-failure versus injection-failure distinction, and the argument that both reduce to the same architectural defect, are the author's. The claim that direct injection is "mostly a nuisance" is a judgement about typical deployments and does not hold where the agent's authority exceeds the user's, a case chapter 8 addresses directly. The list of indirect injection channels is illustrative and assembled by the author, not drawn from a published taxonomy. The Preamble disclosure date could not be traced to a primary document and rests on secondary accounts.