The Outbox Pattern
The Silent Hero

Every distributed system has its famous parts. Service meshes get discussed in conference keynotes. Kafka gets detailed architecture diagrams. Kubernetes even has its own job titles. But one component keeps your events from getting lost. It helps an order survive a broker outage and a pod eviction, even on the same difficult day. This component rarely gets any credit. It works in the background, and nobody thanks it.

This component is called the Outbox pattern. It never becomes a trending topic, and it never wins a design review because of its charm. Yet it supports more production systems than most of the technologies that people like to talk about.

01 — THE PROBLEMThe problem it solves

Every service eventually faces the same situation. It needs to update its own database, and it needs to tell the rest of the system what happened. For example: an order was placed, a payment was cleared, or a user signed up. Two things must happen together: a local database write, and a message sent to a broker. If these two things do not happen together, the system starts giving out wrong information.

The naive version is the one everyone writes first:

public async Task PlaceOrder(Order order)
{
    await _dbContext.Orders.AddAsync(order);
    await _dbContext.SaveChangesAsync();          // (1) DB commit

    await _serviceBus.PublishAsync(new OrderPlaced(order.Id)); // (2) broker publish
}

This code depends on two independent systems, so there are two independent ways it can fail, and there is no shared transaction between them. Suppose step (1) succeeds but step (2) fails — perhaps due to a temporary Service Bus timeout, a network issue, or a pod that gets killed mid-request. In this case, the database says the order exists, but the rest of the system never finds out. There is no invoice, no fulfilment, and no confirmation email. If you reverse the order and publish the message first, you get the opposite problem: an event announcing an order that was never saved.

You cannot combine a database transaction and a broker publish into one atomic operation. Distributed transactions can technically do this, but they are slow, fragile, and poorly supported across cloud brokers. Almost nobody wants to use them today. This is exactly the problem the Outbox pattern solves, and it solves it with one database table.

02 — THE SOLUTIONWhat the pattern does instead

Instead of writing to two systems, the Outbox pattern writes to only one. The outgoing message is stored in the same database as the business data, inside the same transaction.

using var tx = await _dbContext.Database.BeginTransactionAsync();

await _dbContext.Orders.AddAsync(order);

await _dbContext.OutboxMessages.AddAsync(new OutboxMessage
{
    Id          = Guid.NewGuid(),
    Type        = nameof(OrderPlaced),
    Payload     = JsonSerializer.Serialize(new OrderPlaced(order.Id)),
    OccurredOn  = DateTime.UtcNow,
    ProcessedOn = null
});

await _dbContext.SaveChangesAsync();
await tx.CommitAsync();

The business data and the message-to-publish now share a single atomic commit. Either both are saved, or neither is saved. There is no situation where one exists without the other. This turns a very difficult distributed-transaction problem into a simple single-database transaction — something every relational database engine has solved for decades.

A separate process, called the relay, reads the unprocessed rows and publishes them to the broker. Once a message is published successfully, the relay marks that row as done.

public async Task RelayLoop(CancellationToken ct)
{
    while (!ct.IsCancellationRequested)
    {
        var pending = await _dbContext.OutboxMessages
            .Where(m => m.ProcessedOn == null)
            .OrderBy(m => m.OccurredOn)
            .Take(50)
            .ToListAsync(ct);

        foreach (var msg in pending)
        {
            await _serviceBus.PublishAsync(msg.Type, msg.Payload);
            msg.ProcessedOn = DateTime.UtcNow;
            await _dbContext.SaveChangesAsync(ct);
        }

        await Task.Delay(TimeSpan.FromMilliseconds(500), ct);
    }
}

There is no drama and no exotic infrastructure involved. It is just a table, a transaction, and a loop — doing the one job that nobody else was willing to guarantee.

03 — THE BENEFITSBenefits you only notice when they are missing

The best way to measure this pattern's value is to notice what does not happen once it is in place. The Outbox pattern does not just fix correctness. It also reduces the load on every service it touches, and you notice its absence far more than its presence.

First, the request thread stops waiting on the broker. In the naive version, the request that handles PlaceOrder is blocked until the broker confirms the publish. When the broker has a bad day, your API latency also has a bad day. With the Outbox pattern, the request commits locally and returns immediately; the publish happens separately, in the background. Your P99 latency no longer depends on the broker's health, and nobody notices a problem, because nothing breaks.

Second, broker outages no longer become service outages. If the broker is unreachable for ninety seconds, a naive service either fails the requests or drops the events and reports success. A service using the Outbox pattern keeps accepting writes the whole time. Messages pile up in the table and get sent once the broker is back online. The outage is absorbed into the same database you were already writing to.

Third, retries no longer sit in the main request path. Temporary publish failures become the relay's problem, and it retries them in the background. The customer has already received their 201 Created response. Redelivery is no longer something a user has to wait for.

Fourth, you get back-pressure handling for free. During a traffic spike, the table grows, and the relay drains it at a steady, sustainable rate. The service absorbs the burst into durable storage, instead of struggling to publish every message synchronously at peak load.

Put all of this together, and the service does less work per request. It holds threads for less time, and it depends on fewer systems being healthy at the exact moment a user is waiting. The Outbox pattern carries this weight so the rest of the service does not have to.

04 — THE ONE RULEThe one condition it asks of you

Every reliable pattern comes with some conditions. The Outbox pattern gives you at-least-once delivery, not exactly-once delivery. If the relay publishes a message and then crashes before writing ProcessedOn, it will publish that message again after it restarts. That is a feature — it is why you never lose events. But it also means your consumers must be idempotent. You should deduplicate using a message ID, use upserts where possible, and make the downstream operation naturally repeatable. If you follow this one rule, the pattern will never let you down. If you ignore it, you will end up sending duplicate invoices the first time a relay pod restarts in the middle of a batch.

05 — THE PATROLHow the relay keeps watch

Polling is very simple, and it works anywhere: you query for unprocessed rows at a regular interval. It adds a small amount of latency and a small amount of steady querying. Add an index on ProcessedOn, and it will scale further than you might expect.

Change Data Capture (CDC) reads the database transaction log directly, instead of polling. On Azure, a common setup uses Debezium to read a SQL Server or PostgreSQL CDC feed into Event Hubs or Kafka. This approach gives lower latency and removes the polling load, but it also means more infrastructure to run. Use this approach only when your event volume is high enough that polling latency shows up in your metrics — not before.

06 — THE GEARRunning it well on Azure and Kubernetes

• Run the relay as a hosted service or a sidecar. An IHostedService / BackgroundService inside the app works well to begin with. As volume grows, split the relay into its own deployment, so it can scale and restart independently.
• Use leader election. If you run multiple relay replicas, they will compete over the same rows. Use FOR UPDATE SKIP LOCKED in PostgreSQL, or UPDLOCK, READPAST in SQL Server, to share the work safely. You can also elect a single active relay using a Kubernetes lease. Do not leave this to chance.
• Clean up processed rows. Rows that have already been processed are dead weight. Schedule a job to archive or delete any row with a ProcessedOn value past your retention window, so the table and its indexes stay small and efficient.
• Add observability. Set up alerts for outbox depth and the age of the oldest unprocessed row. A growing backlog is an early and clear sign that the broker or relay is unhealthy, often before any consumer even notices.

07 — KNOWING WHEN TO RESTWhen you do not need this pattern

This pattern should not be used everywhere. It requires a table, a relay, regular cleanup, and some operational effort. If a service publishes no events, or if an occasional lost notification is genuinely harmless, you can skip it. The pattern is worth using when a local state change absolutely must be reflected downstream — for example, in orders, payments, or provisioning, where a missing or incorrect record is a real business problem.

The key takeaway

The Outbox pattern will never be the reason someone praises your architecture in a design review. It has no dashboard worth showing off, and no logo worth framing. It replaces a hard, unsolved distributed-transaction problem with a well-understood single-database one. Then it runs in the background, letting your services respond quickly, survive broker outages, and absorb traffic spikes into durable storage instead of into your users' latency.

That is the whole idea behind this pattern: doing the difficult, load-bearing work so reliably that everyone eventually forgets it is even there.