Understanding Distributed Systems Fundamentals

1. Understanding Distributed System Definition

Property	Definition	Example
Distributed System	Collection of independent nodes appearing as single coherent system	Kubernetes cluster, Cassandra, Kafka
Node	Independent computing entity (process, VM, container)	Pod, EC2 instance, broker
Cluster	Group of nodes coordinating to provide service	3-node etcd cluster
Coordination	Mechanisms for nodes to agree on shared state	Consensus, locks, leader election
Shared-Nothing	Each node has independent CPU/memory/disk	Cassandra, BigTable

2. Understanding Network Communication

Aspect	Detail	Impact
Message Passing	Only way nodes interact	No shared memory across nodes
Latency	Time for round-trip; ~0.5ms LAN, 50-200ms WAN	Bounds responsiveness
Bandwidth	Throughput limit per link	Constrains data volume
Packet Loss	0.01-1% typical; higher under congestion	Requires retries/ACKs
Ordering	TCP per-connection only; UDP none	App must enforce global order
Protocols	TCP, UDP, QUIC, HTTP/2, HTTP/3, gRPC	Choose by latency/reliability

3. Understanding Partial Failures

Failure Type	Description	Detection
Crash-Stop	Node halts permanently	Heartbeat timeout
Crash-Recovery	Node halts, then resumes (possibly stale)	Epoch/term numbers
Omission	Messages lost in transit	Sequence numbers, ACKs
Network Partition	Subset of nodes unreachable	Quorum loss, gossip divergence
Byzantine	Arbitrary/malicious behavior	Cryptographic verification
Gray Failure	Slow but not down (degraded)	Latency percentiles, anomaly detection

4. Understanding Transparency Goals

Type	Hides From User	Example
Access	Local vs remote resource access	NFS, gRPC stubs
Location	Physical placement	DNS, service discovery
Migration	Resource moving between nodes	Pod rescheduling
Replication	Multiple copies exist	S3, DynamoDB
Concurrency	Other users sharing resource	DB transactions
Failure	Component failures recovered transparently	RAID, retries, failover
Scaling	System growth/shrinkage	Auto-scaling groups

5. Understanding Fallacies of Distributed Computing

#	Fallacy	Reality
1	Network is reliable	Packets drop, links fail
2	Latency is zero	Speed-of-light bounds; cross-continent ~150ms
3	Bandwidth is infinite	Costs and physical limits
4	Network is secure	Assume hostile; use TLS, mTLS
5	Topology doesn't change	Nodes join/leave constantly
6	One administrator	Multi-team, multi-cloud
7	Transport cost is zero	Egress/serialization expensive
8	Network is homogeneous	Mixed protocols, MTUs, vendors

6. Understanding Scalability Types

Type	Approach	Limit
Vertical (scale up)	Bigger CPU/RAM/disk on one node	Hardware ceiling, cost ∝ size²
Horizontal (scale out)	Add more nodes	Coordination overhead
Functional	Split by service/feature	Service boundaries
Data (sharding)	Partition data across nodes	Cross-shard queries
Geographic	Distribute across regions	Cross-region consistency

7. Understanding Latency vs Throughput Tradeoffs

Metric	Definition	Optimization
Latency	Time for single request (p50/p95/p99)	Caching, fewer hops, faster paths
Throughput	Requests/second handled	Batching, parallelism, async
Tail Latency	p99/p999 — outliers	Hedged requests, tied requests
Little's Law	L = λW (concurrency = rate × latency)	Bound queue depth

8. Understanding Reliability vs Availability

Metric	Formula	Target
Availability	Uptime / (Uptime + Downtime)	99.9% = 8.76h/yr down
99.99% (4 nines)	—	52.6 min/yr
99.999% (5 nines)	—	5.26 min/yr
MTBF	Mean Time Between Failures	Higher = more reliable
MTTR	Mean Time To Recover	Lower = better availability
Reliability	P(no failure in time T)	Function of MTBF

9. Understanding Distributed System Challenges

Challenge	Cause	Mitigation
Partial Failure	Some nodes down, others up	Replication, retries, circuit breakers
Network Partitions	Split connectivity	Quorum, partition-tolerant protocols
Clock Skew	No global time	Logical clocks, HLC, NTP
Concurrency	Simultaneous updates	Locks, MVCC, CRDTs
Ordering	Messages arrive out-of-order	Sequence numbers, vector clocks
Consensus	Agreement under failure	Paxos, Raft
Debuggability	State spread across nodes	Tracing, structured logs

10. Understanding When to Use Distributed Systems

Use When	Avoid When
Single node cannot meet load/storage	Single node suffices
HA > 99.9% required	Best-effort SLA acceptable
Geographic locality needed	Single region OK
Independent team/service deploys	Monolith team
Fault isolation between components	Coupled lifecycles fine