The Dual Write Problem
Understanding and Solving the Atomic Write Challenge in Microservices
Jiuhe Wang | Created: Feb 22, 2026
What is the Dual Write Problem?
The Dual Write Problem occurs when a microservice needs to write to two external systems (like a database AND Apache Kafka) in an atomic fashion, but these systems cannot be wrapped in a single transaction.
When a user pays a bill, the system must:
1. Write to database: Update the account balance (money deducted)
2. Emit event to Kafka: Trigger the payment processing
The problem: If the database write succeeds but Kafka fails, money is gone but payment never happens!
Why Simple Solutions Do Not Work
Still has the problem! If database write fails after event is emitted, you have an event for a transaction that never happened. The system is still inconsistent.
Still does not work! There is no transaction spanning both database AND Kafka. If event is published but transaction is not committed, the event goes out but state rolls back!
Solutions to the Dual Write Problem
1. User places order -> write to orders table + outbox table in ONE transaction
2. Return "Order placed!" to user immediately
3. Background worker reads outbox -> publishes "OrderCreated" event to Kafka
4. Notification service consumes event -> sends email to user
1. Instead of storing balance = $1000, store: Deposited $500, Withdrew $200, Deposited $700
2. Current balance = $500 - $200 + $700 = $1000 (computed on read)
3. Any state can be reconstructed by replaying all events from the beginning
4. To transfer $100: record DebitEvent(acc A, $100) + CreditEvent(acc B, $100)
5. Event store is the single source of truth - Kafka holds all history
1. User clicks "Place Order" -> immediately publish OrderCreated event to Kafka
2. Return "Order confirmed!" to user (very fast, ~10ms)
3. A consumer within the same service reads the event and writes to local database
4. If DB write fails: consumer retries until it succeeds (idempotent)
5. User sees confirmation immediately, background syncs the state
1. Old monolith app writes to PostgreSQL (unchanged, no code modifications)
2. Debezium monitors the database binlog for changes
3. When inventory row updates, Debezium automatically streams to Kafka topic
4. New analytics service consumes from Kafka, builds real-time dashboards
5. Zero changes needed to the legacy app - just add Debezium connector
Real-World: Two-Phase Commit with Reconciliation
In production, the patterns above are not enough. A robust implementation uses Two-Phase Commit (2PC) or Multi-Phase Commit with reconciliation:
State Machine: Two-Phase Commit
Record created
Event emitting
Wait for ack
Consumed + txn complete
Consume failed
Phase 1: INITIALIZED -> PROCESSING (mark before emit)
Phase 2: PROCESSING -> EMITTED (event sent, wait for consumer ack)
Complete: EMITTED -> SUCCESS (consumer confirmed) or FAILED (consumer rejected)
Reconciliation Process
INIT/PROC > threshold
Query + decide
Trigger again
Final state
How It Works
Write to database with status = 'initialized'. The business operation is recorded but event is not yet emitted.
Before emitting the event (via CDC or outbox), atomically update status to processing. This prevents duplicate emission if the process crashes and restarts.
Send event to Kafka, then update status to emitted. Now wait for consumer acknowledgment.
When downstream consumer successfully processes the event and confirms, update to success. Transaction is complete.
If consumer rejects or fails to process, update to failed with error details. May trigger compensation logic.
Scheduled job finds records in initialized or processing beyond threshold. Either auto-retry (re-trigger from EMITTED step) or alert for manual review.
The Safety Net: Records that never make it to SUCCESS will appear in the reconciliation report. You can then investigate and replay safely because idempotency guarantees no duplicate side effects.
Database Schema Example
CREATE TABLE outbox (
id UUID PRIMARY KEY,
aggregate_id UUID NOT NULL,
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
status VARCHAR(20) DEFAULT 'initialized',
retry_count INT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW(),
processed_at TIMESTAMP,
error_message TEXT
);
-- Partial index for reconciliation: only index stuck records
CREATE INDEX idx_outbox_stuck
ON outbox(created_at)
WHERE status = 'initialized' OR status = 'processing';
Solution Comparison
| Pattern | Consistency | Complexity | Latency | Audit Trail | Reconciliation |
|---|---|---|---|---|---|
| Transactional Outbox | [+] Strong | Medium | Low | [-] | Recommended |
| Event Sourcing | [+] Strong | High | Medium | [+] Complete | Built-in |
| Listen to Yourself | [-] Eventual | Low | Very Low | [+] | Recommended |
| CDC | [+] Strong | Medium | Low | [-] | Recommended |
Which Pattern to Use?
Key Takeaways
1. The dual write problem is real and dangerous - do not ignore it!
2. Never assume both writes will succeed - plan for failure
3. Choose the pattern that fits your consistency requirements
4. At-least-once delivery is common - build idempotent consumers
5. Always implement reconciliation for production systems!
6. Test your failure scenarios!