The Transactional Outbox Pattern Doesn't Remove Dual Writes

Your service writes a row and publishes an event. Two systems, one operation, no shared transaction. The standard fix is the transactional outbox pattern, and the thing most write-ups get wrong is that it doesn't remove that dual write. It moves it somewhere you can survive.

The transactional outbox pattern is a design in which a service writes an outgoing message into a table in its own database, inside the same transaction that writes the business data, and a separate relay process later reads that table and publishes the message to the broker. ^[R19]

What it guarantees is atomicity between the business row and the message row: "When you save the business object and its events within the same database transaction, the system guarantees no data loss." ^[R26] What it does not guarantee is delivery. The relay still has to send the message and then record that it sent it, which is two durable operations again. ^[R52]

Key takeaways

• The transactional outbox moves the dual write out of your request handler; it does not remove it, because "relay delivery and checkpointing remain separate durable operations." ^{[R52] [F2]}
• Writing the message in the same transaction as the business data is an atomicity guarantee about your database, not a delivery guarantee. ^[R26]
• The pattern gives at-least-once delivery, never exactly-once; at-least-once plus a consumer that absorbs duplicates is what Microsoft calls effectively-once processing. ^{[F1] [R31]}
• Nobody has measured how often dual writes lose messages in production: the research literature and vendor documentation reviewed here publish no rate of message loss. ^[F4]
• Two-phase commit is a blocking protocol, not a broken one, and Helland carves out "a tight cluster which looks like one machine"; what forces the outbox in practice is that "the database and/or the message broker might not support 2PC." ^{[R48] [R6] [R20] [F5]}

The dual write problem: two writes that disagree

The failure mode is boring, which is why it survives code review. Your handler commits an order to Postgres, then calls the broker. Both lines look fine. Neither is wrong. The problem is the gap between them.

AWS Prescriptive Guidance defines it in one sentence: "A dual write operation occurs when an application writes to two different systems; for example, when a microservice needs to persist data in the database and send a message to notify other systems. A failure in one of these operations might result in inconsistent data." ^[R39] It breaks in two directions, and AWS names both. If the database update succeeds and the publish fails, "the downstream service will not be aware of the change, and the system can enter an inconsistent state." If the update fails but the notification goes out anyway, "data could get corrupted, which might affect the reliability of the system." ^[R40]

Microsoft's Azure Architecture Center is blunter: the naive approach "works until an error occurs between saving the order object and publishing the event." ^[R25] Pat Helland named the underlying reason in 2007: "the message delivery is not directly coupled to the update of the durable data other than through application action... The absence of this coupling leads to failure windows in which the message is delivered more than once." ^[R9] He walks the window step by step — message consumed but not acknowledged, database updated, message then acknowledged, and "In a failure, this is restarted and the message is processed again." ^[R8] Nothing has changed about that. The frameworks got better; the physics didn't.

Chris Richardson's catalogue frames it as a question: "How to atomically update the database and send messages to a message broker?" ^[R19] His listed drawback of the answer isn't mechanical. It's human: "Potentially error prone since the developer might forget to publish the message/event after updating the database." ^[R69]

Now the part most posts on this topic quietly skip. There is no measurement of how often dual writes actually lose messages in production. Not a small number, not a contested number. No number. The research literature and the vendor documentation reviewed for this piece publish no rate of message loss or inconsistency attributable to dual writes; the only quantitative work in the area is formal rather than observational. ^[F4] If you've seen a percentage on a conference slide, ask the speaker where it came from.

So this is a post about mechanism, not frequency.

The paper everyone cites is a position paper, not a measurement

Follow the citation chain under "don't use distributed transactions" far enough and you arrive at one document: Pat Helland's "Life beyond Distributed Transactions: an Apostate's Opinion," presented at the 3rd Biennial Conference on Innovative Data Systems Research, January 2007. ^[R1] Helland was at Amazon at the time. The paper is genuinely excellent, and it's the intellectual foundation of a generation of system design.

It's also printed with the literal subtitle "Position Paper" on its title page, directly beneath the title. ^[R2] Its first line is a disclaimer: "The positions expressed in this paper are personal opinions and do not in any way reflect the positions of my employer Amazon.com." ^[R3] And a few paragraphs later Helland says the quiet part out loud: "The nice thing about writing a position paper is that you can express wild opinions. Here are a few that we will be arguing in the corpus of this position paper". ^[R4]

This is not a debunk. The reasoning in that paper is better than most of what's been written since, and Helland had a vantage point almost nobody had. But it contains no measurements, no benchmark, no dataset. ^[F8] It's authoritative as the design position of a senior practitioner inside Amazon in 2007. It isn't evidence that distributed transactions fail at some rate, because it doesn't measure anything.

Two bits of provenance hygiene while we're here. Helland's paper never uses the word "outbox" and never describes writing messages to a table inside the business transaction, so the link between his argument and the pattern is inferential. ^[F9] And Richardson's catalogue page makes no origin claim of any kind: no inventor, no cited paper, no prior art, no mention of Helland. ^[R22] Nobody in the reachable literature claims to have invented this. ^[F3]

Standing on a well-argued opinion rather than a measurement doesn't make the ground worse. It tells you how big a building to put on it.

What actually forces the transactional outbox pattern

The usual story is that two-phase commit is a discredited protocol and the outbox replaced it. That story is wrong in a specific and useful way.

Gray and Lamport pinned down 2PC's actual defect in "Consensus on Transaction Commit," ACM Transactions on Database Systems 31(1), March 2006, pages 133–160. The abstract says it plainly: "The classic Two-Phase Commit protocol blocks if the coordinator fails." ^[R48] If the transaction manager dies right after every participant has sent its Prepared message, the survivors "have no way of knowing whether the TM committed or aborted the transaction." ^[R49]

Then they fix it. Paxos Commit uses multiple coordinators and "makes progress if a majority of them are working," with "Two-Phase Commit... isomorphic to Paxos Commit with a single coordinator." ^[R50] For a five-participant transaction that costs 17 messages instead of 12. ^[R51] 2PC is a blocking protocol, not a broken one.

Helland's position is narrower than the folklore version too. His abstract says that when projects "attempt to use distributed transactions, the projects founder because the performance costs and fragility make them impractical." ^[R5] But he leaves a carve-out: "we aren't doing transactions across machines except perhaps in the simple case where there is a tight cluster which looks like one machine." ^[R6] His technical objection is the one Gray and Lamport diagnosed, that 2PC "can easily block when nodes are unavailable." ^[R7] So no, two-phase commit is not always wrong. ^[F5] If you run a tight cluster that behaves as one machine, the man everyone cites against distributed transactions already gave you permission.

What actually forces the outbox is duller than the theory. Richardson lists it as a force: "2PC is not an option. The database and/or the message broker might not support 2PC." ^[R20] Your Postgres and your Kafka can't enlist in a common transaction, so atomicity across them isn't on sale at any price. That's a product-capability constraint, not a proof.

Helland's structural claim survives either way: "Atomic Transactions Cannot Span Entities," where an entity is data that "must live within a single scope of serializability (i.e. one machine or cluster)." ^{[R13] [R12]} Your database row is one entity. The broker's log is another. One transaction each, never one across both.

The outbox moves the dual write, it doesn't remove it

Here's the trick, and it's a good one. Instead of writing to the database and then to the broker, you write the message into the database, in the same transaction as the business data. Richardson: "store the message in the database as part of the transaction that updates the business entities. A separate process then sends the messages to the message broker." ^[R19] Microsoft states what that buys, and notice how narrow it is: "When you save the business object and its events within the same database transaction, the system guarantees no data loss." ^[R26] An atomicity guarantee about your database. Not a delivery guarantee.

The relay comes in two flavours, catalogued as peers. Polling Publisher "Works with any SQL database" but is "Tricky to publish events in order." ^[R23] Transaction Log Tailing reads the MySQL binlog, the Postgres WAL or DynamoDB table streams and publishes "each message/event inserted into the outbox," at the cost of being database-specific. ^[R24] AWS notes change data capture can skip the outbox table entirely, which "saves the overhead of creating another table to track the updates." ^[R43]

Now the result that should change how you talk about this in design reviews.

In August 2026, Andreas Andreakis published a preprint, "Machine-Checked Dual-Write Recovery from a Committed Log" (arXiv:2608.00501, submitted 1 August, revised to v4 on 13 August), with its core argument proved in Isabelle/HOL. Its statement about the pattern is unambiguous: "Transactional outboxes and change data capture move this dual write out of an application process, but relay delivery and checkpointing remain separate durable operations." ^[R52] The relay still has to deliver the message and then record that it delivered it. Two writes. Different place, same shape.

The main result is an information bound, not an observation. Two reachable post-crash states share "the same durable source-side state" but differ downstream, so "Any recovery policy based only on the source side must duplicate an effect in one state or leave it undelivered in the other." ^[R53] Read that operationally and it's brutal. After a crash, your outbox table cannot tell you whether that row was already delivered. No cleverness in the relay closes the gap, because the information isn't in your database. The only escape reads the far end: "An authoritative, complete, and current sink acceptance record lets recovery compute the missing operations." ^[R54]

Stated honestly: single-author preprint, not peer-reviewed, and it proves an impossibility rather than measuring a system. What it kills is the claim that the outbox makes the dual write go away. ^[F2] Worth knowing who wrote it, though. Andreakis co-authored Netflix's DBLog change-data-capture paper ^[R56], which reported DBLog in production across "tens of microservices at Netflix" as at October 2020 ^[R57]. The engineer who built the CDC framework later proved what it can't guarantee.

So what did the outbox buy? It relocated the unreliable step. In the naive version the failure happens inside your request handler, where the message is gone and nobody knows. In the outbox version it happens in a relay, where the undelivered row is still sitting in a table you can query, retry and alert on. Microsoft even tells you to size that table's retention for an outage: set the event time-to-live to "a time span of multiple days, like 10 days." ^[R28] A lost message becomes a visible backlog. That's the whole win, and it's a big one.

Does the outbox give exactly-once delivery? No — at-least-once

No. The transactional outbox pattern gives you at-least-once delivery, never exactly-once. ^[F1] The other half of the pattern is a consumer that absorbs duplicates without doing damage, and it usually lives in someone else's codebase.

The relay retries. That's its job. Microsoft's own outbox guidance concedes what retrying produces: "When reprocessing occurs, the application might have already sent some messages to Service Bus, which normally creates duplicate message processing." ^[R27]

Every vendor that ships this says so in its own documentation. Microsoft: "Most brokers, including Azure Service Bus, Azure Event Hubs, Apache Kafka, and RabbitMQ, provide at-least-once delivery," which "means that the broker can deliver the same message more than once." ^[R29] AWS, on the standard-queue implementation: SQS standard queues "guarantee that the message is delivered at least once and doesn't get lost," but "the same message or event might be delivered more than once, so you should ensure that the event notification service is idempotent." ^[R42]

Richardson puts the fix on the consumer: "a message consumer must be idempotent, perhaps by tracking the IDs of the messages that it has already processed." ^[R21] Debezium ships for that assumption. Its outbox router emits the event's unique ID as a message header so that "You can use this ID, for example, to remove duplicate messages" ^[R59], in service of a pattern the project describes as one that "avoids inconsistencies between a service's internal state... and state in events consumed by services that need the same data." ^[R58] Both quotes come from Debezium's own project documentation.

Microsoft's architecture guidance states the general case as plainly as vendors ever state anything: "It's impractical to guarantee exactly-once delivery across a distributed system. Even brokers that use exactly-once semantics can guarantee only operations they directly control... They can't control the side effects that consumers implement in external systems." ^[R30] Confluent and AWS say the same thing in their own words, which matters, because this is the sentence most design docs get wrong.

Which brings us to Kafka, because someone in the room always says Kafka has exactly-once. It does, within a scope. Confluent's documentation: Kafka "uses transactional producers and consumers to provide exactly-once delivery when transferring and processing data between Kafka topics." ^[R45] Between Kafka topics. On writing anywhere else, the same page says it "can be challenging to coordinate the data the consumer is receiving and the consumer's position in the data," and that this is typically done with a two-phase commit. ^[R46] The guarantee does not span your database write. ^[F6] Apache's own docs site yielded no text to automated fetching, so attribute those to Confluent, not to the Kafka project.

Helland got there first, and was warmer about it than the modern consensus. He calls himself "a big fan of 'exactly-once in-order' messaging," but says those facilities "are rarely available to the programmer building scalable applications." ^[R11] The plumbing settles for at-least-once "because its only other recourse is to occasionally lose messages." ^[R10]

At-least-once delivery plus a consumer that ignores duplicates gives you what they call effectively-once processing: "The durable solution isn't to eliminate duplicate delivery but to make the consumer process it correctly." ^[R31] That's the property to write on the design doc.

Building an idempotent consumer

Consumer idempotence is where this pattern gets half-built, because it's the half that isn't yours. The relay is in your service. The consumer is in another team's repo, on another roadmap, and the duplicate surfaces as a double charge months later.

Helland put the burden exactly where it still sits: the application "must implement mechanisms to ensure that the incoming message is idempotent." ^[R14] His definition is deliberately loose — processing is idempotent "if a subsequent execution of the processing does not perform a substantive change to the entity," leaving "open to the application" what counts as substantive. ^[R15] For messages that aren't naturally idempotent, "the entity must remember they have been processed. This knowledge is state." ^[R16]

Before you build any of that, try to make it unnecessary. Microsoft's documentation recommends operations that are idempotent by construction — "An upsert keyed on a business identifier, a write that sets an absolute value rather than an increment, or an HTTP PUT to a resource identifier" — and says to "Design for natural idempotency if possible." ^[R63] A lot of deduplication machinery exists because someone modelled an increment where an absolute value would have done.

When you do need bookkeeping, three vendors publish how they bound theirs, and each bound is a clock you set whether you meant to or not.

Stripe's API documentation describes result-caching against a client-supplied key of up to 255 characters: Stripe saves "the resulting status code and body of the first request made for any given idempotency key." ^{[R60] [R62]} The expiry is explicit: keys can be removed "after they're at least 24 hours old," and "We generate a new request if a key is reused after the original is pruned." ^[R61]

Microsoft's Azure Service Bus documentation gives duplicate detection a window that "defaults to 10 minutes for queues and topics, with a minimum value of 20 seconds and a maximum value of 7 days," comparing exactly one field: "No other parts of the message other than the MessageId are considered." ^{[R37] [R38]}

AWS's SQS developer guide bounds FIFO deduplication to five minutes: retry SendMessage "within the 5-minute deduplication interval" and "Amazon SQS doesn't introduce any duplicates into the queue." ^[R44]

None of those three is measurement. Each is a vendor documenting the shape of its own safety net, and the nets are smaller than people assume. Microsoft says so directly: broker-side deduplication "operates on the send side and within a bounded window, so it doesn't prevent a consumer from processing the same message twice after a redelivery. You still need idempotent consumer logic." ^{[R35] [F7]}

Three practical notes, again from vendor documentation rather than anyone's field data. Write the deduplication marker and the business effect in the same transaction. That's the consumer-side mirror of the outbox: "This transactional variant is the Inbox pattern, and is the consumer-side companion to the producer Transactional Outbox pattern." ^[R33] Don't check-then-write, either; under competing consumers that races, and the prescription is a uniqueness constraint that "makes the database the single arbiter of the conflict." ^[R34]

Size retention against the broker, not against a round number. Keep each record "at least as long as the broker can still redeliver the original message," because "Deleting records too early reopens the window for duplicates." ^[R36] The Andreakis preprint makes the same point formally, showing "how bounded deduplication state and truncated source history limit the lifetime of the guarantee." ^[R55] Your correctness has an expiry date. Someone should choose it deliberately.

And know its limits. Deduplication "removes duplicates, but doesn't guarantee order." ^[R70] Idempotence doesn't propagate on its own: "propagate the idempotency key so that each service tier can deduplicate its own work." ^[R66] And Microsoft argues against hand-rolling it at all, since "Correctly implementing deduplication storage, commit, and cleanup is error-prone." ^[R64]

Emit a signal, too. Microsoft suggests tracking detected duplicates, since "A rising duplicate rate can indicate producer misconfiguration, an undersized acknowledgment or lock window, or unhealthy consumers." ^[R65] What else belongs on that dashboard is a separate argument.

One last risk, and it isn't technical. Those four numbers — dedup window, key retention, outbox TTL, broker redelivery limit — have to be reconciled against each other, and in most teams that reconciliation lives in one engineer's head.

Task-shaped work that spans two teams

An outbox table, a relay with retry and backlog metrics, and idempotent consumers with a deliberately chosen retention window is well-specified work with a clear definition of done. It's also never the most urgent thing in the sprint, and it spans two teams' codebases, which is how it ends up sitting in a backlog for quarters.

That shape is what Dev On Demand is for: single stream $3,495/mo with 1 dedicated AI-augmented engineer, dual stream $6,795/mo with 2 in parallel, a 3-day task cycle, daily async updates, and a task-by-task approval gate. You get a 5-day first ship, a 5-day replacement guarantee, and you can cancel any time with no notice period required. If you'd rather judge before committing, take the Proof of Quality: one real task, and you judge the engineer before subscribing.

Either way, start with the sentence in your design doc: if it says exactly-once, it's wrong. Write down at-least-once plus an idempotent consumer, then go and find out what your deduplication window is actually set to.

Frequently asked questions

Does the transactional outbox pattern remove the dual write?

No. It moves it. The machine-checked preprint states it directly: "Transactional outboxes and change data capture move this dual write out of an application process, but relay delivery and checkpointing remain separate durable operations." ^[R52] The gain is that the unreliable step now happens in a relay, where an undelivered row is still sitting in a table you can query, retry and alert on, instead of inside a request handler where the message is gone. ^[F2]

Does the transactional outbox give exactly-once delivery?

No. The pattern gives at-least-once delivery, and it needs a consumer that is idempotent — one that absorbs duplicates without doing damage. ^{[F1] [R21]} The achievable target is what Microsoft calls effectively-once processing: "The durable solution isn't to eliminate duplicate delivery but to make the consumer process it correctly." ^[R31] Kafka's exactly-once is real but scoped: Confluent's documentation describes it as exactly-once "when transferring and processing data between Kafka topics," which does not span a write to your database. ^{[R45] [F6]}

Is two-phase commit always the wrong choice?

No. Gray and Lamport's diagnosis is that "The classic Two-Phase Commit protocol blocks if the coordinator fails" — blocking, not broken. ^[R48] Helland's own carve-out allows "the simple case where there is a tight cluster which looks like one machine." ^{[R6] [F5]} What rules 2PC out in most stacks is a product-capability constraint rather than a proof: "The database and/or the message broker might not support 2PC." ^[R20]

How often do dual writes actually lose messages in production?

Nobody knows, and nobody publishes a figure. The research literature and vendor documentation reviewed for this piece publish no rate of message loss or inconsistency attributable to dual writes; the only quantitative work in the area is formal rather than observational. ^[F4] Treat this as a question of mechanism, not frequency.

Is Pat Helland's 2007 paper evidence that distributed transactions fail?

Not in the measurement sense. "Life beyond Distributed Transactions: an Apostate's Opinion" is printed with the subtitle "Position Paper" on its title page and opens by stating that its positions "are personal opinions." ^{[R2] [R3]} It contains no measurements, no benchmark and no dataset. ^[F8] It is authoritative as the design position of a senior practitioner, not as evidence of a failure rate.

Do I still need an idempotent consumer if my broker deduplicates?

Yes. Broker-side deduplication "operates on the send side and within a bounded window, so it doesn't prevent a consumer from processing the same message twice after a redelivery. You still need idempotent consumer logic." ^{[R35] [F7]} The consumer-side companion is to write the deduplication marker and the business effect in the same transaction: "This transactional variant is the Inbox pattern, and is the consumer-side companion to the producer Transactional Outbox pattern." ^[R33]

Sources

• [F1] Folklore, refuted: "The transactional outbox gives you exactly-once delivery." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [F2] Folklore, refuted: "The outbox makes the dual write go away." — https://arxiv.org/abs/2608.00501
• [F3] Folklore, refuted: "Chris Richardson invented the transactional outbox pattern." — https://microservices.io/patterns/data/transactional-outbox.html
• [F4] Folklore, refuted: "N% of distributed systems lose messages", or any specific message-loss rate attributed to dual writes. — http://export.arxiv.org/api/query?search_query=all:%22transactional%20outbox%22%20OR%20all:%22dual%20write%22%20OR%20all:%22dual-write%22&start=0&max_results=30
• [F5] Folklore, refuted: "Two-phase commit is always wrong, and nobody uses it." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [F6] Folklore, refuted: "Kafka gives exactly-once, so publishing an event after a database write is safe." — https://docs.confluent.io/kafka/design/delivery-semantics.html
• [F7] Folklore, refuted: "Broker-side deduplication (Service Bus duplicate detection, SQS FIFO) makes the consumer safe." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [F8] Folklore, refuted: "Helland's CIDR 2007 paper is empirical evidence that distributed transactions fail at scale." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [F9] Unverified, do not assert: "The outbox pattern originated in Helland's CIDR 2007 paper." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R1] The paper "Life beyond Distributed Transactions: an Apostate's Opinion" by Pat Helland, then at Amazon.com, was published at the 3rd Biennial Conference on Innovative Data Systems Research (CIDR), 7–10 January 2007, Asilomar, California, appearing at page 132 of the proceedings. — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R2] The paper carries the literal subtitle "Position Paper" on its title page, directly beneath the title. — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R3] Helland's paper opens with the disclaimer: "The positions expressed in this paper are personal opinions and do not in any way reflect the positions of my employer Amazon.com." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R4] Helland writes: "The nice thing about writing a position paper is that you can express wild opinions. Here are a few that we will be arguing in the corpus of this position paper". — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R5] Helland's abstract states: "When they attempt to use distributed transactions, the projects founder because the performance costs and fragility make them impractical. Natural selection kicks in..." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R6] Helland does NOT argue that two-phase commit is always wrong. He writes: "Put simply, we aren't doing transactions across machines except perhaps in the simple case where there is a tight cluster which looks like one machine." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R7] Helland describes 2PC's specific technical weakness rather than a blanket prohibition: "This includes 2PC (two phase commit) which can easily block when nodes are unavailable and other protocols which do not block in the face of node failures such as the Paxos algorithm." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R8] Helland states the exact dual-write failure window under a section headed 'Most Applications Use "At-Least-Once" Messaging': "The message is consumed but not yet acknowledged. The database is updated and then the message is acknowledged. In a failure, this is restarted and the message is processed again." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R9] Helland names the root cause as the absence of coupling between messaging and durable state: "The dilemma derives from the fact that the message delivery is not directly coupled to the update of the durable data other than through application action... The absence of this coupling leads to failure windows in which the message is delivered more than once." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R10] Helland argues at-least-once is chosen because the alternative is worse: the messaging plumbing behaves this way "because its only other recourse is to occasionally lose messages ("at-most-once" messaging) and that is even more onerous to deal with". — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R11] Helland explicitly does not dismiss exactly-once, he says it is usually unavailable: "I am a big fan of "exactly-once in-order" messaging but to provide it for durable data requires a long-lived programmatic abstraction similar to a TCP connection. The assertion here is that these facilities are rarely available to the programmer building scalable applications. Hence, we are considering cases dealing with "at-least-once"." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R12] Helland defines an entity as the unit that bounds a transaction: "the upper layer code for each application must manipulate a single collection of data we are calling an entity... it must live within a single scope of serializability (i.e. one machine or cluster). Each entity has a unique identifier or key." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R13] Helland's central structural claim: "Atomic Transactions Cannot Span Entities... The programmer must always stick to the data contained inside a single entity for each transaction." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R14] Helland places the idempotence burden on application code, not infrastructure: "Typically, the scale-agnostic (higher-level) portion of the application must implement mechanisms to ensure that the incoming message is idempotent... So far, this is not yet available." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R15] Helland defines idempotence functionally rather than formally: "The processing of a message is idempotent if a subsequent execution of the processing does not perform a substantive change to the entity. This is an amorphous definition which leaves open to the application the specification of what is and what is not substantive." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R16] Helland distinguishes naturally idempotent messages (reads, non-substantive changes) from those needing bookkeeping; for non-naturally-idempotent messages "the entity must remember they have been processed. This knowledge is state. The state accumulates as messages are processed." — https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
• [R19] Chris Richardson's microservices.io pattern page states the problem as: "How to atomically update the database and send messages to a message broker?" and the solution as: "The solution is for the service that sends the message to first store the message in the database as part of the transaction that updates the business entities. A separate process then sends the messages to the message broker." — https://microservices.io/patterns/data/transactional-outbox.html
• [R20] Richardson's outbox page lists as a force: "2PC is not an option. The database and/or the message broker might not support 2PC." — https://microservices.io/patterns/data/transactional-outbox.html
• [R21] Richardson's outbox page requires consumer idempotence: "a message consumer must be idempotent, perhaps by tracking the IDs of the messages that it has already processed. Fortunately, since message Consumers usually need to be idempotent (because a message broker can deliver messages more than once) this is typically not a problem." — https://microservices.io/patterns/data/transactional-outbox.html
• [R22] Richardson's outbox catalogue page makes no origin claim whatsoever: it names no inventor, cites no research paper, credits no prior art and does not mention Pat Helland. Its only attribution is the page footer, "Copyright © 2026 Chris Richardson • All rights reserved • Supported by Kong." — https://microservices.io/patterns/data/transactional-outbox.html
• [R23] Richardson splits the relay into two named sub-patterns. Polling Publisher: "Publish messages by polling the database's outbox table," with benefit "Works with any SQL database" and drawbacks "Tricky to publish events in order" and "Not all NoSQL databases support this pattern." — https://microservices.io/patterns/data/polling-publisher.html
• [R24] Richardson's Transaction Log Tailing pattern is the CDC alternative: "Tail the database transaction log and publish each message/event inserted into the outbox to the message broker," naming MySQL binlog, Postgres WAL and AWS DynamoDB table streams as mechanisms. Its listed drawbacks include being database-specific and difficulty avoiding duplicate publishing. — https://microservices.io/patterns/data/transaction-log-tailing.html
• [R25] Microsoft's Azure Architecture Center documents the pattern as "Implement the Transactional Outbox Pattern by Using Azure Cosmos DB," describing the failure as: "This approach works until an error occurs between saving the order object and publishing the event," listing network error, message service outage and host failure as causes, and concluding "Lost events can cause data inconsistencies across the application." — https://learn.microsoft.com/en-us/azure/architecture/best-practices/transactional-outbox-cosmos
• [R26] The same Microsoft page states the outbox guarantee as an atomicity guarantee only: "When you save the business object and its events within the same database transaction, the system guarantees no data loss. The transaction either commits everything or rolls back everything if an error occurs." — https://learn.microsoft.com/en-us/azure/architecture/best-practices/transactional-outbox-cosmos
• [R27] The same Microsoft page concedes the relay produces duplicates: "When reprocessing occurs, the application might have already sent some messages to Service Bus, which normally creates duplicate message processing. To prevent this scenario, you can turn on duplicate message detection in Service Bus." — https://learn.microsoft.com/en-us/azure/architecture/best-practices/transactional-outbox-cosmos
• [R28] Microsoft recommends setting the outbox event time-to-live long enough to survive a relay outage: "In a production environment, set a time span of multiple days, like 10 days. This duration ensures that all components have sufficient time to process and publish changes within the application." — https://learn.microsoft.com/en-us/azure/architecture/best-practices/transactional-outbox-cosmos
• [R29] Microsoft's Idempotent Consumer pattern page states that at-least-once is the industry norm: "Most brokers, including Azure Service Bus, Azure Event Hubs, Apache Kafka, and RabbitMQ, provide at-least-once delivery. This guarantee ensures that a message reaches a consumer even when failures occur, but also means that the broker can deliver the same message more than once." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R30] Microsoft states plainly that exactly-once is not achievable end-to-end: "It's impractical to guarantee exactly-once delivery across a distributed system. Even brokers that use exactly-once semantics can guarantee only operations they directly control, such as delivering messages to consumers or writing data back to the broker. They can't control the side effects that consumers implement in external systems." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R31] Microsoft names the achievable target "effectively-once": "The durable solution isn't to eliminate duplicate delivery but to make the consumer process it correctly. When you combine at-least-once delivery with a consumer that ignores duplicates, you achieve effectively-once processing." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R33] Microsoft names the consumer-side mirror of the outbox as the Inbox pattern: "Avoid this failure window by writing the deduplication marker and the business side effects in the same transaction... This transactional variant is the Inbox pattern, and is the consumer-side companion to the producer Transactional Outbox pattern." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R34] Microsoft warns that check-then-write deduplication races under competing consumers and prescribes a database constraint: "Use a uniqueness constraint on the deduplication key such that two transactions can attempt to insert a key, but only one can succeed... This approach makes the database the single arbiter of the conflict." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R35] Microsoft warns that broker-level deduplication is not a substitute for consumer idempotence: "This feature operates on the send side and within a bounded window, so it doesn't prevent a consumer from processing the same message twice after a redelivery. You still need idempotent consumer logic. Use platform features to reduce duplicate volume, not as a replacement for the Idempotent Consumer pattern." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R36] Microsoft ties deduplication-record retention to the broker's redelivery window: "Retain each record at least as long as the broker can still redeliver the original message. The size of this window depends on the broker's maximum delivery attempts, its lock or visibility timeout, and the message time-to-live... Deleting records too early reopens the window for duplicates." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R37] Azure Service Bus duplicate detection retains message IDs for a configurable window that "defaults to 10 minutes for queues and topics, with a minimum value of 20 seconds and a maximum value of 7 days." — https://learn.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection
• [R38] Azure Service Bus duplicate detection compares only one field: "If any new message is sent with MessageId that was logged during the time window, Service Bus reports the message as accepted (the send operation succeeds), but the newly sent message is instantly ignored and dropped. No other parts of the message other than the MessageId are considered." — https://learn.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection
• [R39] AWS Prescriptive Guidance defines the problem the outbox solves: "A dual write operation occurs when an application writes to two different systems; for example, when a microservice needs to persist data in the database and send a message to notify other systems. A failure in one of these operations might result in inconsistent data." — https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
• [R40] AWS names both failure directions: "If the database update is successful but the event notification fails, the downstream service will not be aware of the change, and the system can enter an inconsistent state. If the database update fails but the event notification is sent, data could get corrupted, which might affect the reliability of the system." — https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
• [R42] AWS states the delivery guarantee of the standard-queue outbox explicitly: "Amazon SQS standard queues guarantee that the message is delivered at least once and doesn't get lost. However, when you use Amazon SQS standard queues, the same message or event might be delivered more than once, so you should ensure that the event notification service is idempotent." — https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
• [R43] AWS documents CDC as the second implementation of the same pattern, avoiding the outbox table entirely: "Some databases support the publishing of item-level modifications to capture changed data. You can identify the changed items and send an event notification accordingly. This saves the overhead of creating another table to track the updates." — https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
• [R44] Amazon SQS FIFO deduplication is bounded to five minutes: "If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue." — https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html
• [R45] Confluent's Kafka documentation scopes Kafka's exactly-once guarantee to Kafka itself: "Kafka supports exactly-once delivery in Kafka Streams and uses transactional producers and consumers to provide exactly-once delivery when transferring and processing data between Kafka topics." — https://docs.confluent.io/kafka/design/delivery-semantics.html
• [R46] The same Confluent page states that writing to an external system falls outside that guarantee: "When writing to an external system, it can be challenging to coordinate the data the consumer is receiving and the consumer's position in the data. Typically, this might be done with a two-phase commit, storing the consumer position, and then storing the consumed data." — https://docs.confluent.io/kafka/design/delivery-semantics.html
• [R48] Gray and Lamport, "Consensus on Transaction Commit," ACM Transactions on Database Systems, Volume 31, Issue 1, March 2006, pages 133-160 (Microsoft Research technical report MSR-TR-2003-96), states in its abstract: "The classic Two-Phase Commit protocol blocks if the coordinator fails." — https://lamport.azurewebsites.net/video/consensus-on-transaction-commit.pdf
• [R49] Gray and Lamport describe the precise 2PC failure mode: "the failure of the TM can cause the protocol to block until the TM is repaired. In particular, if the TM fails right after every RM has sent a Prepared message, then the other RMs have no way of knowing whether the TM committed or aborted the transaction." — https://lamport.azurewebsites.net/video/consensus-on-transaction-commit.pdf
• [R50] Gray and Lamport show 2PC is not irreparable but a degenerate case of a fault-tolerant protocol: "Two-Phase Commit is not fault tolerant because it uses a single coordinator whose failure can cause the protocol to block. We have introduced Paxos Commit, a new transaction commit protocol that uses multiple coordinators and makes progress if a majority of them are working. Hence, 2F + 1 coordinators can make progress even if F of them are faulty. Two-Phase Commit is isomorphic to Paxos Commit with a single coordinator." — https://lamport.azurewebsites.net/video/consensus-on-transaction-commit.pdf
• [R51] Gray and Lamport give the message-count cost of non-blocking commit for a transaction with five resource managers: "the Two-Phase Commit uses 12 messages, regular Paxos Commit uses 17, and Faster Paxos Commit uses 20 (with co-location)." — https://lamport.azurewebsites.net/video/consensus-on-transaction-commit.pdf
• [R52] An arXiv preprint, Andreas Andreakis, "Machine-Checked Dual-Write Recovery from a Committed Log," arXiv:2608.00501, submitted 1 August 2026 and revised to v4 on 13 August 2026, states that the outbox relocates rather than removes the dual write: "Transactional outboxes and change data capture move this dual write out of an application process, but relay delivery and checkpointing remain separate durable operations." — https://arxiv.org/abs/2608.00501
• [R53] The same preprint's main result, proved in Isabelle/HOL, is an information bound: "We construct two reachable post-crash states with the same durable source-side state and different sink acceptance records. Any recovery policy based only on the source side must duplicate an effect in one state or leave it undelivered in the other." — https://arxiv.org/abs/2608.00501
• [R54] The same preprint states the escape route requires reading the destination, not the source: "An authoritative, complete, and current sink acceptance record lets recovery compute the missing operations when source coordinates distinguish them." — https://arxiv.org/abs/2608.00501
• [R55] The same preprint states that deduplication state is finite and the guarantee therefore expires: "we show how bounded deduplication state and truncated source history limit the lifetime of the guarantee." — https://arxiv.org/abs/2608.00501
• [R56] Andreakis and Papapanagiotou, "DBLog: A Watermark Based Change-Data-Capture Framework," arXiv:2010.12597, submitted 23 October 2020, names dual writes as a known-limited approach: "We have observed a series of distinct patterns that have tried to solve this problem such as dual-writes and distributed transactions. However, these approaches have limitations with regard to feasibility, robustness, and maintenance." — https://arxiv.org/abs/2010.12597
• [R57] The DBLog paper reports production use at Netflix as at October 2020: "DBLog is currently used in production by tens of microservices at Netflix." — https://arxiv.org/abs/2010.12597
• [R58] Debezium's own documentation source describes the outbox pattern as: "The outbox pattern is a way to safely and reliably exchange data between multiple (micro) services. An outbox pattern implementation avoids inconsistencies between a service's internal state (as typically persisted in its database) and state in events consumed by services that need the same data." — https://raw.githubusercontent.com/debezium/debezium/main/documentation/modules/ROOT/pages/transformations/outbox-event-router.adoc
• [R59] Debezium's outbox event router expects an outbox table whose id column is emitted as a message header specifically so consumers can deduplicate: "Contains the unique ID of the event. In an outbox message, this value is a header. You can use this ID, for example, to remove duplicate messages." — https://raw.githubusercontent.com/debezium/debezium/main/documentation/modules/ROOT/pages/transformations/outbox-event-router.adoc
• [R60] Stripe's API documentation describes idempotency keys as client-generated and result-caching: "Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors." — https://docs.stripe.com/api/idempotent_requests
• [R61] Stripe prunes idempotency keys after a bounded window, after which a retry is treated as a new request: "You can remove keys from the system automatically after they're at least 24 hours old. We generate a new request if a key is reused after the original is pruned." — https://docs.stripe.com/api/idempotent_requests
• [R62] Stripe rejects reuse of a key with different parameters: "The idempotency layer compares incoming parameters to those of the original request and errors if they're not the same to prevent accidental misuse." Keys are "up to 255 characters long" and Stripe suggests "using V4 UUIDs, or another random string with enough entropy to avoid collisions." — https://docs.stripe.com/api/idempotent_requests
• [R63] Microsoft recommends preferring naturally idempotent operations over deduplication bookkeeping: "An upsert keyed on a business identifier, a write that sets an absolute value rather than an increment, or an HTTP PUT to a resource identifier produce the same results whether they run once or many times... Design for natural idempotency if possible, and use deduplication techniques only for operations that can't be made naturally idempotent." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R64] Microsoft recommends existing frameworks over hand-rolled deduplication: "Correctly implementing deduplication storage, commit, and cleanup is error-prone. Message-based frameworks provide this pattern as a built-in feature," naming NServiceBus and the MassTransit consumer outbox. — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R65] Microsoft recommends instrumenting duplicate rate as an operational signal: "Emit the deduplication key and a correlation identifier in structured logs, and track a metric for detected duplicates. A rising duplicate rate can indicate producer misconfiguration, an undersized acknowledgment or lock window, or unhealthy consumers." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R66] Microsoft warns that consumer idempotence does not propagate automatically: "Making the message consumer idempotent doesn't protect the services it calls. When a consumer invokes downstream services as part of processing, propagate the idempotency key so that each service tier can deduplicate its own work." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer
• [R69] Richardson's stated drawback of the outbox is human, not mechanical: "Potentially error prone since the developer might forget to publish the message/event after updating the database." — https://microservices.io/patterns/data/transactional-outbox.html
• [R70] Microsoft warns that deduplication does not preserve order: "Deduplication removes duplicates, but doesn't guarantee order. If the consumer depends on processing order, combine this pattern with an ordering mechanism such as Service Bus message sessions, or include sequence or version data so the consumer can reject stale messages." — https://learn.microsoft.com/en-us/azure/architecture/patterns/idempotent-consumer