The Transactional Outbox Pattern

Distributed systems rarely fail in obvious ways.

They tend to fail in small, inconvenient, almost invisible ways. A timeout here. A dropped connection there. A process that exits at exactly the wrong moment.

One of the most dangerous versions of this looks harmless at first:

You update your database successfully.
You attempt to publish an event.
The publish fails.
The process crashes.

From the outside, nothing dramatic has happened. There is no exception bubbling up to the user. There is no obvious corruption.

And yet, your system has split into two different versions of reality.

The database says the change happened.

The rest of the world never heard about it.

This is how silent drops occur. An event that should have been emitted simply vanishes between a database commit and a message publish.

The Transactional Outbox pattern exists to close that gap.

The Subtle Failure

Consider a typical piece of application code:

// ❌ Dangerous pattern
await db.SaveChangesAsync();
await messageBus.PublishAsync(new UserRegistered(user.Id));

The code is clean and readable. First persist the change. Then notify the rest of the system.

The problem is that those two operations live in different worlds.

The database commit is durable once it succeeds. The message publish depends on a network hop to a broker. If the commit succeeds and the publish fails, the user is registered in the database but no downstream service is aware of it.

There is no practical way to wrap a relational database and a message broker in a distributed two-phase commit without introducing serious complexity and fragility.

A different approach is needed.

Bringing the Message Inside the Transaction

The key insight behind the Transactional Outbox pattern is simple: treat the event as data.

Instead of publishing directly to the message broker, the application writes the event into a dedicated “outbox” table as part of the same database transaction that performs the business change.

The flow becomes:

Write your business change.
Write an outbox row describing the event.
Commit both within the same database transaction.
A background process reads from the outbox and publishes events to the message broker.

If the transaction commits, both the business state and the event record are safely stored. If it rolls back, neither exists.

The database becomes the single source of truth for both state and pending messages.

A Simple Outbox Schema

An outbox table often looks something like this:

CREATE TABLE outbox (
    outbox_id UUID PRIMARY KEY,
    event_type VARCHAR(200) NOT NULL,
    aggregate_id UUID NOT NULL,
    payload JSON NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    published_at TIMESTAMP NULL,
    publish_attempts INT NOT NULL DEFAULT 0
);

Each row represents an event waiting to be published. The presence of a row means the event exists. The published_at column indicates whether it has been successfully sent.

Writing to the Outbox

Application code changes slightly:

using var transaction = await db.Database.BeginTransactionAsync();

db.Users.Add(user);

db.Outbox.Add(new OutboxMessage
{
    OutboxId = Guid.NewGuid(),
    EventType = "UserRegistered",
    AggregateId = user.Id,
    Payload = JsonSerializer.Serialize(new { user.Id, user.Email })
});

await db.SaveChangesAsync();
await transaction.CommitAsync();

There is no message bus call here.

The transaction commits once. Both the user record and the outbox row are persisted together. If the process crashes before the commit, nothing is stored. If it crashes after the commit, both pieces of data remain.

At this point, the system is internally consistent.

The Outbox Relay

Publishing happens asynchronously.

A background worker periodically reads rows where published_at is null, publishes them to the message broker, and marks them as published.

var messages = db.Outbox
    .Where(x => x.PublishedAt == null)
    .OrderBy(x => x.CreatedAt)
    .Take(100)
    .ToList();

foreach (var message in messages)
{
    await messageBus.PublishAsync(message.Payload);

    message.PublishedAt = DateTime.UtcNow;
    message.PublishAttempts++;
}

await db.SaveChangesAsync();

If publishing fails, the row remains. The relay will attempt again later. The event is not lost because it already exists as durable data in the database.

What Happens During a Crash?

Different crash points produce different outcomes:

If the crash happens before the transaction commits, neither the business change nor the outbox row is stored.
If it happens after the commit but before publishing, the outbox row remains and will be picked up by the relay.
If it happens during publishing, the relay can retry based on the stored state.

The system may become eventually consistent, but it does not become inconsistent. The state and the intent to publish move forward together.

Idempotency Still Matters

The outbox guarantees that an event will be published. It does not guarantee that it will be published exactly once.

A relay might retry. A network issue might cause duplicate delivery. Consumers must therefore be designed to handle at-least-once delivery semantics.

Idempotency remains a requirement at the edges of the system.

Operational Considerations

The pattern introduces some overhead. There is an extra table. There must be a cleanup strategy. There is a slight delay between committing data and publishing the event. The relay requires monitoring.

Outbox rows are not permanent records. Common approaches include deleting rows after a retention window, partitioning by time, or archiving for audit purposes.

These are manageable tradeoffs when compared to silent event loss.

When It Fits

The Transactional Outbox pattern is particularly useful in microservices, event-driven systems, domain event publishing, and any architecture where losing an event would leave the system in an incorrect state.

In a simple monolith without messaging, the pattern adds little value. In systems where eventual consistency is unnecessary, it may be overkill.

As always, context determines complexity.

The Underlying Principle

The pattern rests on a simple idea:

Do not rely on the network inside your transaction boundary.

Commit your state once.

Persist your intent to publish.

Let publication happen eventually.

In distributed systems, durability begins at the database. The outbox ensures your events share that durability, instead of trusting a network call that might fail at the worst possible moment.