The reference architecture

The reference architecture

Seven components. Five trust boundaries. One direction of travel.

Four properties make this an architecture rather than a pile of controls.

The application cannot reach the provider. Not by policy, by network rule. This is the single most important line in the diagram, and it is the one most often missing. A gateway that developers should use is a convention. A gateway that is the only route with egress to api.provider.com is an architecture.

The sidecar has no egress at all. It receives text and returns spans. It does not need the internet, so it does not get the internet.

The vault is reachable only from the gateway, and it holds ciphertext whose key lives in a KMS the vault cannot read.

Filtering happens in both directions at the same chokepoint, which is what closes the output door.

What runs where

# Topology, expressed as network zones. The zones are the control.
zones:
  app:
    workloads: [web, api, workers]
    egress: [gateway]                    # and nothing else

  detection:
    workloads: [gateway, sidecar, vault]
    egress:
      gateway: [provider-allowlist, vault, sidecar, log-sink]
      sidecar: []                        # deliberately empty
      vault:   []                        # deliberately empty

  provider-allowlist:
    hosts: ["api.openai.com"]            # one entry, reviewed

If you take one thing from this chapter into your next design review, take the two empty lists. A component that cannot make outbound connections cannot exfiltrate, regardless of what a dependency does or what an injected instruction asks for. It is the cheapest hard boundary available to you.

The build order, and what each step buys

Do these in order. The ordering is deliberate: each step reduces the work the next one has to do.

1. Classify (a day). Walk every LLM call. For each field, decide its tier. Write the table down. You cannot design the architecture before you know what is in Tier 3, because Tier 3 sizes everything else.

2. Push everything possible into Tier 1 (days). Projection records, the IPromptSafe marker, out-of-band slots. Chapter 12. This is where the absolute guarantee comes from and it costs less than any other step.

3. Build the gateway as a network boundary (a week). Chapter 13. Policy, audit record, failure mode, both-direction filtering. Then change the egress rules so applications cannot reach the provider directly. Step 3 is not finished until that rule exists, because until then you have a library, not a wall.

4. Add Tier 2 validators in-process (days). Chapter 5 and Appendix A. They run at 0.1 ms, so they run on every payload, every log line, every tool-call argument. They also become your degraded mode when step 5 is unavailable.

5. Deploy the sidecar, hardened from the start (days). Chapter 14. TLS, no disk, no egress, non-root, pinned digest, logging off. Retrofitting those is a migration.

6. Wire the round trip and the vault (a week). Chapter 9. Stable tokens, scoped restoration, round-trip invariant test, fail closed.

7. Close the remaining doors (days). Log sink redaction and retention, provider contract terms, and if you have RAG, ingestion-time de-identification plus permission-filtered retrieval. Chapters 15 and 16.

8. Measure and prove (days). Golden set in CI, audit record on, task quality comparison. Chapter 17.

A competent team does this in about a month. Chapter 17 lays it out as four weeks.

What this architecture actually guarantees

Precision matters here, because this is the paragraph people will quote back at you in a security review.

It guarantees, absolutely:

• Fields excluded by projection never reach the provider from the paths that route through the gateway.
• The application cannot reach the provider by any other route, if the egress rule is in place.
• The sidecar and vault cannot initiate outbound connections.
• Real values for tokenised entities never leave your perimeter; only tokens do.
• Every model call has an audit record naming the policy version in force.

It guarantees, deterministically but within a bounded class:

• Well-formed checksummed identifiers are caught on every payload the gateway sees.

It does not guarantee, and cannot:

• That personal data in free text is detected. Around half of it will be, on the evidence in Chapter 4.
• That the model does not infer personal data that was never in the input.
• That a person with database access cannot read the same data directly. This architecture addresses accidental egress to a model, not insider access.
• That the provider honours its contract. You have a contractual control, not a mechanism.
• Anything at all about code paths that do not route through the gateway.

That last one is the failure mode in practice. The architecture covers what it covers.

The caveats, stated plainly

The gateway is bypassable until the network says otherwise. A developer with an API key and a HttpClient can call the provider from anywhere. Egress control is the only thing that converts intent into enforcement, and it is the step most often skipped because it needs the platform team.

Every new service is a hole until it is routed. Six months from now someone will build a feature, add the SDK, and ship. Your defence is that the API key is not available outside the detection zone, and that the egress rule denies by default. Make obtaining a provider credential a deliberate act.

The vault concentrates risk. You have taken a diffuse problem and put the mapping from token to person in one place. That is easier to defend and worse when it fails. Short TTLs and key separation are what make the trade worth it.

Tier 3 drift is invisible. Your detector's real performance changes when your traffic changes, and nothing alerts. The golden-set test in CI is the only thing that surfaces it.

Latency and availability are now coupled. Your LLM feature depends on a container. Fail closed and you have an availability dependency. Fail open and you have no control. Choose per route, write it in policy, and count the degradations.

None of this makes you compliant. It produces the evidence a compliance process needs. The process is still someone's job.

So what is the actual 100% solution?

You asked, so here it is without hedging.

The only configuration in which no third party receives your personal data is one where no third party is involved. Self-host the model. Run inference inside your perimeter, on hardware you control, with the same egress rule applied to the inference service that you applied to the sidecar. Then the provider row disappears from the diagram, and the whole class of "what does the vendor retain" questions disappears with it.

This is a real option, and for some organisations it is the right one. Open-weight models have closed much of the capability gap for the bounded tasks that most enterprise features actually perform: classification, extraction, summarisation, routing. If your workload is one of those and your data is regulated, self-hosting is worth pricing seriously rather than dismissing.

Three honest caveats, because this is not a free win.

You trade a data risk for an operating cost. GPUs, capacity planning, model updates, evaluation, an on-call rotation for an inference service. That is a standing engineering commitment, not a project.

The capability gap is real at the frontier. For hard reasoning, long context, and tool use, the hosted frontier models remain ahead. If your feature needs that, self-hosting costs you quality.

It closes exactly one door. Look back at Chapter 3. Self-hosting removes the provider from the prompt door. Ingestion, tool calls, logs and output are all still open, and they are all inside your perimeter. Your vector store still holds customer text. Your logs still capture prompts. Your agent can still be injected into calling a tool that emails someone. Self-hosting is not an alternative to this architecture. It is one substitution inside it, and every other component stays exactly where it is.

Which is the closing point, and the reason this chapter exists. There is no configuration that gets you to 100% on all five doors, because door three is a human typing prose and prose is unbounded. There is a configuration that gets you to 100% on the data you chose not to send, near-100% on the identifiers you can validate, and a measured, named, survivable number on everything else.

That is the wall. The next five chapters build it, one component at a time.

Sources for this chapter

This chapter is synthesis rather than new evidence. Every quantitative claim restates something sourced elsewhere in the book.

• Cross-domain detection landing around 0.5 F1, and the Tier 3 ceiling — Sikkema benchmark and PIIBench, both cited in full in Chapter 4: https://albertsikkema.com/python/security/privacy/2026/06/01/benchmarking-open-source-pii-detection.html · https://arxiv.org/abs/2605.25816
• Deterministic validator latency (0.1 ms) and the checksum algorithms behind Tier 2 — cited in Chapter 5 and Appendix A.
• Egress control, capability limitation and provenance-aware deployment as the mitigations that bound blast radius — named across the agent-security literature cited in Chapter 15: https://arxiv.org/pdf/2506.01055 · https://arxiv.org/pdf/2604.05432
• The chokepoint-with-versioned-policy pattern, and filtering both directions — practitioner consensus, sources listed in Chapter 13.
• Provider zero data retention and regional processing as contractual rather than mechanical controls — cited in Chapter 16.

Everything structural in this chapter is the author's argument: the three-tier model, the claim that only Tier 1 carries an absolute guarantee, the seven-component reference architecture, the build order, and the guarantee/non-guarantee lists. These are engineering positions, not findings, and they are offered so you can disagree with them specifically rather than generally.

The self-hosting section makes no quantitative claim about the capability gap between open-weight and frontier models, deliberately. Any number there would be stale within a quarter, and Chapter 4's argument about decontextualised benchmark figures applies with equal force to model leaderboards.