Understanding Distributed Systems Concepts

1. Understanding Distributed Computing Challenges

Challenge	Description	Mitigation
Partial failure	Some nodes fail, others continue	Timeouts, retries, idempotency
Network unreliability	Packets lost/delayed/reordered	TCP, retries, ack protocols
Concurrency	Multiple nodes act simultaneously	Locks, consensus, CRDTs
No global clock	Cannot order events globally	Logical clocks, NTP+TrueTime
Heterogeneity	Mixed HW/OS/lang versions	Standard protocols, schemas
Security	Untrusted network	mTLS, signed messages

2. Understanding the Fallacies of Distributed Computing

#	Fallacy	Reality
1	Network is reliable	Packets lost, partitions occur
2	Latency is zero	Light-speed lower bound; cross-region 100ms+
3	Bandwidth is infinite	Bottleneck under load
4	Network is secure	Always assume hostile
5	Topology doesn't change	Auto-scaling, deploys, failures
6	One administrator	Multi-team, multi-cloud
7	Transport cost is zero	Egress costs $$, serialization CPU
8	Network is homogeneous	Mixed protocols, devices, versions

3. Understanding Network Partitions and Failures

Failure Type	Symptom	Detection
Crash failure	Node stops responding	Heartbeat timeout
Omission failure	Some messages dropped	Acks + retries
Network partition	Subset cannot reach others	Quorum loss detection
Byzantine failure	Arbitrary/malicious behavior	BFT consensus
Gray failure	Slow but not failed	p99 latency, hedged requests

4. Understanding Clock Synchronization

Clock	Mechanism	Accuracy
NTP	Public time servers	~10–100ms
PTP	HW timestamps in DC	<1µs
GPS/Atomic	Direct hw reference	ns range
TrueTime (Spanner)	GPS+atomic+uncertainty bound	<7ms uncertainty

Warning: Never rely on wall clocks for ordering across nodes — clock skew, drift, and leap seconds make it unsafe.

5. Understanding Vector Clocks and Lamport Timestamps

Type	Captures	Limitation
Lamport timestamp	Single counter, total order	Cannot detect concurrent events
Vector clock	Per-node counters; partial order	O(N) size, GC complex
Version vector	Replica versioning for conflicts	Used in Dynamo, Riak
Hybrid Logical Clock	Physical + logical bits	Bounded skew, used in CockroachDB

Example: Vector clock causality

Node A: [A:1]  → send msg → Node B receives
Node B: merges = [A:1, B:1]  → send to C
Node C: [A:1, B:1, C:1]
Concurrent: [A:2, B:0] vs [A:0, B:2] (neither dominates)

6. Understanding Distributed Transaction Challenges

Challenge	Cause	Solution
Atomicity across nodes	Partial commits	2PC, 3PC, Saga
Coordinator failure	Blocks participants	Timeouts, 3PC, Paxos commit
Performance	Many round trips	Batch, optimistic concurrency
Cross-DB transactions	XA limited support	Sagas, outbox pattern

7. Understanding Two-Phase Commit (2PC)

2PC Protocol

Prepare: Coordinator sends PREPARE; participants vote YES (write to log) or NO
Commit: If all YES → COMMIT; any NO → ABORT; participants apply and ACK

Property	Behavior
Atomicity	All commit or all abort
Blocking	If coordinator dies after PREPARE, participants block
Latency	2 round trips + log fsyncs
Used in	XA transactions, distributed RDBMS

8. Understanding Three-Phase Commit (3PC)

Phase	Action
CanCommit	Coordinator asks participants if they can commit
PreCommit	If all yes, coordinator says "prepare to commit"
DoCommit	Final commit; participants ACK

Note: 3PC is non-blocking with synchronous network but unsafe under network partitions. Modern systems prefer Paxos/Raft commit.

9. Understanding Distributed Snapshots

Algorithm	Purpose	Use Case
Chandy-Lamport	Consistent global snapshot via markers	Deadlock detection, checkpointing
Asynchronous Barrier Snapshot	Snapshot in stream pipeline	Apache Flink checkpointing
Copy-on-write	Point-in-time view of state	ZFS, etcd snapshots

10. Understanding Byzantine Fault Tolerance

Algorithm	Tolerates	Use Case
PBFT	(N-1)/3 malicious nodes	Hyperledger, permissioned chains
Tendermint	1/3 byzantine	Cosmos SDK
PoW (Nakamoto)	<50% hash power	Bitcoin
PoS variants	<1/3 stake byzantine	Ethereum, Solana

Note: BFT requires 3f+1 nodes to tolerate f byzantine failures. Used where actors may be malicious (blockchains, multi-org systems). Not needed in trusted DCs.

Related Cheat Sheets

GRPC