Fault-Tolerant Systems Cheat Sheet

Fault tolerance starts from an uncomfortable premise: components will fail at different times and in different ways, often while reporting incomplete information. A robust system is not one that avoids failure. It is one that limits blast radius, keeps critical work flowing, and makes recovery routine instead of improvisational.

Design around failure domains

The first question is where a fault can spread. A single process crash is a small failure domain. A shared database, message broker, region, or identity provider can become a much larger one. Systems become fragile when multiple important paths depend on the same hidden component.

Replication helps, but only when replicas fail independently. Running three copies of a service on the same host does not create meaningful resilience. The same principle applies to data. Backups stored in the same account or region as the primary system reduce convenience risk but not disaster risk.

Timeouts, retries, and backpressure need discipline

Distributed systems often fail by waiting too long. Every network call should have a timeout sized to the caller's purpose. Without one, a small downstream stall can pin threads, exhaust connection pools, and create a larger outage upstream.

Retries can help with transient faults, but they are dangerous when multiplied across service layers. A five-deep call stack where each layer retries three times can turn one failing request into hundreds of extra calls. Use bounded retries, exponential backoff, and jitter. For non-idempotent operations, require a request key or another mechanism that prevents duplicate side effects.

Backpressure is the companion control. When work arrives faster than the system can process it, queues should be bounded and rejection should be explicit. Silent buffering often delays the failure until recovery is harder.

Prefer degradation over collapse

Not every feature needs the same availability target. If recommendations fail, checkout should still work. If analytics lag, the write path should continue. Fault-tolerant systems classify dependencies by criticality and decide in advance what can be disabled, served stale, or approximated.

This is where circuit breakers, cached responses, and read-only modes earn their keep. They do not solve the root cause, but they stop one broken dependency from dragging the whole product down.

Durability and correctness are separate concerns

High availability does not guarantee correct state. Replication lag, split-brain conditions, and partial commits can all keep a service online while returning inconsistent data. Critical workflows need clear write ordering, transactional boundaries, or compensating actions when a distributed transaction cannot be completed atomically.

Event-driven systems should assume duplicates and reordering. Batch systems should assume replays. Recovery paths should be tested, not just described.

Operability is part of resilience

A system is only fault tolerant if operators can see what failed and act safely. Health checks, metrics, structured logs, dependency dashboards, and runbooks are not accessories. They are the tools that convert a fault from a mystery into a bounded repair task.

The practical goal is simple: a single fault should stay single. That outcome depends less on any one pattern than on disciplined boundaries, explicit operational limits, and recovery procedures that have already been exercised before the real outage arrives.