Home

/

Keep PII Out of Your LLM

/

What Counts as PII

What Counts as PII

Chapter 2
Part I
10
min read

Three definitions that do not agree

Ask a US security engineer, a healthcare compliance lead, and a European privacy officer what counts as personal data. You get three answers.

State law: "personal information." The CCPA, as amended by the CPRA, defines it as information that identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked with a particular consumer or household. That last word does more work than engineers expect: a household identifier is in scope even when no individual is named. California is not alone. Twenty states have a comprehensive consumer privacy law in effect, and four more were enacted during 2026 with future effective dates, taking the enacted total to twenty-four (IAPP tracker, 2026). Counts published elsewhere differ only because some report laws enacted and others laws in force. The definitions differ in the details and agree on the shape.

HIPAA's 18 identifiers. A literal checklist of what to remove from health data: names, all geographic subdivisions smaller than a state, all date elements except year, phone numbers, email addresses, Social Security numbers, medical record numbers, account numbers, biometric identifiers, full-face photographs, and the rest. Remove all eighteen and you have met the Safe Harbor standard. If you touch health data, this is your working entity list, and Appendix A uses it as a coverage checklist.

GDPR: "personal data." Any information relating to an identified or identifiable natural person. The broadest of the three by a distance, and it applies to you if you have customers in the EU regardless of where your company sits. It is not a list, it is a test, and the test is about identifiability rather than about which field the value sits in. An IP address can be personal data. So can a device identifier, a pseudonymous customer reference, or a free-text note that says "the tall guy in accounts who drives the red truck."

Build to an enumerated list alone and you will build something that passes your own tests and misses most of your obligation, because every one of these regimes reaches data your list does not name. The practical answer is to use the broadest available test to decide what matters and HIPAA's list to decide what to detect first, which is what the rest of this book does.

The distinction engineers get wrong

Direct identifiers point at one person on their own. A full name, an email address, a Social Security number, a passport number. These are the ones you think of, and they are the ones detectors are best at, because many of them have structure.

Quasi-identifiers do not point at anyone on their own and point at exactly one person in combination. ZIP code. Date of birth. Gender. Job title. Employer. The date of a hospital admission.

None of those is personal data in isolation, in any intuitive sense. Together they routinely identify a single individual.

There is a number attached to this, and it is worth handling carefully because it is one of the most misquoted figures in privacy. Latanya Sweeney, working from 1990 US Census data, found that 87% of the US population was likely unique on the combination of five-digit ZIP code, gender, and full date of birth (Sweeney, 2000). That figure is repeated constantly, usually without its date. When Philippe Golle repeated the analysis on 2000 Census data, the proportion came out at 63% (Golle, 2006).

Both numbers make the same point and the honest one to quote is the lower, more recent one. Roughly three Americans in five are uniquely identified by three attributes that no engineer would flag as personal data in a code review. That is the whole argument for taking quasi-identifiers seriously, and it does not need the inflated version.

It is also the reason HIPAA's Safe Harbor list, at 45 CFR §164.514(b)(2), is both useful and blunt. Safe Harbor removes geographic subdivisions smaller than a state and truncates all dates to the year specifically because ZIP code and date of birth are the classic re-identifying pair. And the standard's own guidance is candid that removing the eighteen does not eliminate re-identification risk from combinations of what remains. That is why the alternative route exists: Expert Determination, at §164.514(b)(1), where a qualified person applies statistical methods, certifies that the risk of re-identification is very small, and documents the analysis. Note what the standard does not do: HHS sets no numeric threshold for "very small". The expert defines it for the dataset and the release environment, and must retain the justification for HHS to inspect. If you were hoping for a number to engineer against, there isn't one. It retains more detail, such as month-level dates and sub-state geography, at the cost of needing a defensible analysis rather than a checklist.

For prompt data specifically, quasi-identifiers matter less than they do for datasets, because you are usually sending one record rather than a population. They matter enormously the moment you build a corpus, which is Chapter 15's problem.

Pseudonymised data is still personal data

This is the single most consequential sentence in the chapter, and it is worth stating without hedging.

Under GDPR, replacing a name with a token does not take the data outside the regulation. If the mapping from token back to person exists anywhere, under anyone's control, the data is pseudonymised, and pseudonymised data is personal data. Your obligations travel with it.

US state law reaches the same place by a different route. The CCPA carves out "deidentified" information, and then defines it strictly: you must take reasonable measures to prevent reidentification, publicly commit to keeping it deidentified, and contractually bind anyone you give it to. A token plus a lookup table you still hold is not that. It is pseudonymisation, and the data remains personal information.

Anonymisation is a much higher bar: the mapping must not exist, for anyone, irreversibly. If you built the round trip in Chapter 9, where a vault holds PERSON_1 → Sarah Whitfield so the answer can be restored before the user sees it, you have built pseudonymisation. That is a good and useful control. It reduces exposure at the model, it limits what a provider retains, it shrinks blast radius. It does not move you outside the regulation, and any vendor telling you it does is selling something.

Why this matters in practice: it changes what you promise. "We anonymise data before sending it to the model" is a claim you probably cannot support and should not put in a security questionnaire. "We pseudonymise personal data before it reaches the model provider, and the mapping never leaves our infrastructure" is accurate, defensible, and describes a stronger control than most of your competitors have.

Quiet data

Most teams can name the fields that hold personal data in their primary tables. Almost no team can name where it ends up.

Quiet data is personal data sitting somewhere nobody classified. It shows up in five places with grim reliability.

Free-text fields. Notes, descriptions, comments, the "anything else we should know?" box. Users put anything in there: medical conditions, other people's names, account numbers, the reason they are disputing a charge. A free-text field is an unschematised store of arbitrary personal data and it is usually the field most likely to be sent to a model, because free text is exactly what language models are for.

Exception messages and logs. An exception thrown while processing a record often carries the record. Serialised request bodies attached to error reports. A structured logger that helpfully logs the whole DTO at debug level in production because someone was debugging in March. Chapter 16 goes at this properly, because for many organisations the log store holds more personal data than the database does, with weaker access controls and longer retention.

Document and file metadata. A Word document carries its author. A photograph carries GPS coordinates and a device identifier. A PDF carries the name of whoever printed it. If you are extracting text from uploaded files and sending it onward, you are probably also sending the metadata, or discarding it without noticing that you had it.

Identifiers that look like plumbing. A customer reference, a session token, an internal user ID. These feel like infrastructure and they function as pseudonyms, which by the paragraph above makes them personal data when a mapping exists. It does exist. It is in your users table.

Derived and inferred values. A risk score, a churn probability, a segment label. These relate to an identified person and can be more sensitive than the inputs, because an inference about someone's health or finances is an inference regardless of how it was computed.

A working definition for the rest of this book

You need something you can implement against, so here it is.

Treat a value as personal data if it identifies a person on its own, if it identifies a person in combination with other values in the same payload, or if a mapping exists anywhere that ties it back to a person. Apply this to free text and file contents, not only to typed columns.

That definition is deliberately broader than what you will end up sending to a detector, and it should be. The definition sets the scope of your thinking. The detector's entity list is a subset you choose knowingly, with the gap documented, which is Chapter 7's job.

Two practical consequences follow immediately, and both are cheap.

Build an inventory before you build a control. For each LLM feature, write down what fields enter the prompt, what enters any corpus, and what enters logs. Most teams discover at least one field they did not know they were sending, and the discovery is usually free to fix.

Classify the sensitivity tiers you actually have. Contact details, financial identifiers, government identifiers, health data, and data about children are not the same risk and should not attract the same safeguard. A single global policy forces you to pick between over-redacting everything and under-protecting the serious categories.

With a definition in hand, the next question is where it physically escapes. There are five exits, most teams guard one of them, and the one they guard is not the widest.

Sources for this chapter

  • CCPA/CPRA definition of personal information including household; 24 states with comprehensive privacy laws as of August 2026; the CCPA "deidentified" standard — US state privacy law trackers: https://www.osano.com/us-data-privacy-laws · https://secureprivacy.ai/blog/us-state-privacy-law-tracker-2026 · Statutory text is the authority; these are current summaries. Verify the state count before quoting it, since four 2026 laws are enacted but not yet effective.
  • HIPAA 18 identifiers; Safe Harbor (45 CFR §164.514(b)(2)) vs Expert Determination (§164.514(b)(1)); no numeric standard for "very small" risk; residual combination risk — HHS, Guidance Regarding Methods for De-identification of Protected Health Information: https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html · Primary. Regulatory text at 45 CFR §164.514.
  • 87% of the US population unique on ZIP + gender + date of birth (1990 Census) — L. Sweeney, Simple Demographics Often Identify People Uniquely, Carnegie Mellon Data Privacy Working Paper 3, 2000: https://dataprivacylab.org/projects/identifiability/paper1.pdf
  • 63% on the same combination using 2000 Census data — P. Golle, Revisiting the Uniqueness of Simple Demographics in the US Population, ACM WPES 2006: https://crypto.stanford.edu/~pgolle/papers/census.pdf · The book quotes the lower, more recent figure in the prose and names both, because the 87% number is widely repeated without its 1990 date.
  • GDPR "personal data" (Art. 4); pseudonymised data remains personal data (Recital 26) — statutory.

Unsourced and offered as the author's argument: the "quiet data" taxonomy (free text, logs, file metadata, plumbing identifiers, derived values) and the working definition at the end of the chapter. They organise the problem; they are not findings.

the-leak-you-cant-see
blast-radius
what-counts-as-pii
the-five-doors
the-accuracy-reckoning
the-hybrid-that-does-not-work
deterministic-detection
npi-in-c
the-three-way-choice
calling-the-analyzer-from-c
measuring-your-own-demo-gap
choosing-the-operating-point
the-ladder-of-safeguards
pseudonymisation
the-round-trip
restoring-safely
when-masking-breaks-the-task
plausibility-hazard
the-architecture-that-holds
the-reference-architecture
dont-send-it-at-all
structure-beats-prose
the-gateway
failure-is-a-policy-decision
the-sidecar-you-can-trust
egress-deny-it-at-the-network
rag-and-agents
de-identify-before-you-embed
dual-model-separation
the-boring-controls
evidence-and-the-first-thirty-days
week-two-the-chokepoint-and-the-fast-layer
entity-catalogue-and-c-validators
mbi-positional-rules
tooling-at-a-glance
azure-ai-language-pii-in-detail
container-trust-checklist
sources
azure-ai-language
provider-retention

Download the full PDF for free?

Free download — no account required

Get the PDF
Get the PDF
Related Chapters
Free Download
Get the full PDF
All pages, including all code examples, diagrams, and the appendix reference card.
No spam. Unsubscribe at any time.
Your email won't be shared.
Oops! There's a problem with your request. We're working on fixing it. Please try again later.