Node Failure Detection in Distributed Systems

Failure detection in a distributed system is an exercise in informed suspicion, not certainty. If one node stops responding, the observer cannot immediately know whether the process crashed, the machine froze, the network path broke, the remote node is overloaded, or a long garbage-collection pause delayed the reply. Good failure detectors are designed around this ambiguity.

Heartbeats are the starting point

The simplest detector uses heartbeats. A node periodically sends a small signal to a peer or monitoring service. If the signal does not arrive before a timeout, the observer marks the node as suspect. Push heartbeats are easy to implement and give low-latency detection, but they can create noisy traffic in large fleets. Pull-based checks move the scheduling burden to the observer, which can simplify senders but often lengthens detection time.

A heartbeat should usually carry more than "I am alive". Term, epoch, software version, or coarse health state can help other nodes decide whether the sender is merely reachable or actually fit to serve. Even then, be careful not to overload heartbeat messages with heavy diagnostics. Large or slow heartbeat handling can become its own failure mode.

Timeouts are a policy choice

Every detector balances speed against false positives. Short timeouts catch failures quickly but can mistake congestion or temporary pauses for death. Long timeouts reduce false alarms but delay failover. The right threshold depends on network behaviour, runtime pauses, and the cost of a wrong decision.

That is why production systems often use suspicion rather than immediate eviction. A suspect node may stop receiving new work while the cluster waits for stronger evidence. Some systems use accrual detectors, such as phi-based approaches, which estimate how unusual the current silence is instead of applying one fixed timeout. This adapts better to changing latency patterns.

Leases and acknowledgements tighten the signal

Two-way heartbeats improve confidence because they prove both reachability and a working return path. Leases go further. A node is considered leader or owner of a resource only while it can renew a time-bounded lease. If renewal stops, others may take over after the lease expires. This is safer than assuming ownership lasts until explicit release.

Leases need fencing tokens or monotonically increasing terms. Without them, a paused old leader might resume and continue writing after a new leader has already taken over. The cluster would then have two actors that both believe they are authoritative.

Membership is a system-wide problem

In small clusters, a central coordinator may track node liveness. In larger or more dynamic fleets, gossip protocols spread membership information gradually between peers. Gossip scales well and avoids one monitoring bottleneck, but it accepts that different nodes will briefly disagree about who is alive. Consensus systems such as Raft layer stricter rules on top: a node can be reachable and still not be allowed to lead because it lacks quorum.

Quorum changes the question from "is this node up?" to "does the cluster still have enough healthy members to make safe progress?" That distinction is essential for databases, lock services, and schedulers.

Design for bad detections

Failure detectors will be wrong sometimes. Build the rest of the system so a wrong suspicion is survivable. Make operations idempotent where possible, avoid duplicate leadership, and ensure rejoining nodes reconcile state before serving traffic again. Log suspicion transitions clearly so operators can distinguish real node crashes from network partitions.

This approach fits almost every distributed system, but the detector should match the consequences of failure. A chat presence service can tolerate occasional false suspicions. A payment ledger or consensus cluster cannot.