Implementing Distributed Transactions
1. Understanding ACID Properties in Distributed Systems
| Property | Distributed Challenge |
|---|
| Atomicity | Cross-node commit requires coordination (2PC) |
| Consistency | Invariants across shards may need sagas |
| Isolation | Snapshot/serializable expensive cross-shard |
| Durability | Replicated WAL with quorum |
2. Implementing Two-Phase Commit (2PC)
2PC Protocol
- Prepare: Coordinator asks all participants "can you commit?"
- Participants write to WAL, vote YES/NO
- Commit/Abort: Coordinator decides based on votes
- Participants apply decision and ACK
| Property | Value |
|---|
| Blocking | If coordinator fails after prepare, participants stuck |
| Latency | 2 RTTs minimum |
| Used by | XA transactions, MSDTC |
3. Implementing Three-Phase Commit (3PC)
| Phase | Action |
|---|
| CanCommit | Voting phase |
| PreCommit | Coordinator broadcasts intent; non-blocking unblock point |
| DoCommit | Final commit |
| Limitation | Assumes synchronous network; vulnerable to partitions |
4. Understanding Distributed Deadlocks
| Detection | Mechanism |
|---|
| Wait-for graph | Centralized cycle detection |
| Edge chasing | Probe messages around graph |
| Timeout-based | Abort long-waiting txns (most DBs) |
| Wound-wait / wait-die | Priority by timestamp prevents cycles |
5. Implementing Saga Pattern (choreography)
| Property | Detail |
|---|
| Mechanism | Each service emits events; others react |
| No central coordinator | Loose coupling |
| Risk | Implicit flow hard to trace; cyclic dependencies |
| Compensation | On failure, reverse-direction events |
6. Implementing Saga Pattern (orchestration)
Example: Orchestrated saga (pseudo-code)
public class OrderSaga {
public void execute(Order o) {
try {
paymentSvc.charge(o);
inventorySvc.reserve(o);
shippingSvc.schedule(o);
} catch (Exception e) {
// Compensating transactions in reverse order
try { shippingSvc.cancelIfScheduled(o); } catch (Exception ignored) {}
try { inventorySvc.release(o); } catch (Exception ignored) {}
try { paymentSvc.refund(o); } catch (Exception ignored) {}
throw e;
}
}
}
| Property | Detail |
|---|
| Central coordinator | Explicit state machine |
| Tools | Camunda, Temporal, AWS Step Functions, Netflix Conductor |
| Pros | Visible, testable, easier debugging |
7. Implementing Compensating Transactions
| Action | Compensation |
|---|
| Reserve seat | Release seat |
| Charge card | Refund |
| Send email | Send retraction email |
| Decrement stock | Increment stock |
Note: Compensations must be idempotent and able to handle partial original execution.
8. Understanding Isolation Levels
| Level | Anomalies Prevented |
|---|
| Read Uncommitted | None |
| Read Committed | Dirty reads |
| Repeatable Read | + Non-repeatable reads |
| Snapshot Isolation | + Most anomalies; allows write skew |
| Serializable | All anomalies |
| Strict Serializable | Serializable + linearizable real-time order |
9. Implementing Optimistic Concurrency Control
Example: Version-based OCC
UPDATE accounts
SET balance = balance - 100, version = version + 1
WHERE id = 42 AND version = 7;
-- Affected rows = 0 means concurrent update; retry
| Best for | Avoid for |
|---|
| Low contention | High contention (retry storms) |
| Short txns | Long txns |
| Read-heavy workloads | Write-heavy hotspots |
10. Implementing Pessimistic Locking
| Lock Type | Use |
|---|
| Shared (S) | Multiple readers |
| Exclusive (X) | Single writer |
| Intent (IS, IX) | Hierarchical locking |
SELECT ... FOR UPDATE | Row-level X lock |
| Two-phase locking (2PL) | Acquire growing phase, release shrinking phase |
11. Understanding Transaction Coordinators
| System | Role |
|---|
| Spanner TrueTime | Globally consistent timestamps |
| Percolator (Bigtable) | Snapshot isolation via timestamps |
| CockroachDB | Serializable snapshot via HLC |
| YugabyteDB | HLC + Raft groups |