Latency Numbers for System Design

The most useful latency numbers to know are not exact constants. They are order of magnitude anchors that train your intuition about where time goes in a system. Engineers make better design decisions when they can tell, almost automatically, whether an operation is measured in nanoseconds, microseconds, milliseconds, or seconds. That gap matters because adding two remote calls is not like adding two in memory operations.

A good starting scale looks like this. CPU cache access is extremely fast, often a few nanoseconds. Main memory access is slower but still usually measured in tens to a few hundreds of nanoseconds. Reading from local SSD storage tends to land in microseconds or low milliseconds depending on the access pattern. A mechanical disk seek is dramatically slower again. A network round trip inside one datacentre may be measured in tens or hundreds of microseconds, while cross region traffic is usually measured in milliseconds to tens of milliseconds. A full request that touches browser, internet, edge, application, and database layers can easily reach hundreds of milliseconds if not designed carefully.

The precise numbers vary by hardware, topology, queueing, and load, which is why memorising one canonical table is less important than understanding the relative gaps. Memory is slower than cache by an order of magnitude. Storage is slower than memory by more than that. Remote network calls are slower again. Human perception adds another perspective: around 100 milliseconds often feels instant, 300 milliseconds is noticeable, and multi second delays start to feel broken unless the interface gives clear progress feedback.

These mental anchors help in practical ways. If a code path moves from one local lookup to five sequential network calls, the likely latency impact is enormous even before benchmarking. If a service needs 99th percentile response times under load, you learn to care about queueing, retries, and tail amplification rather than only average case timings.

Percentiles matter as much as raw numbers. A system with a 20 millisecond median and a 2 second 99th percentile may feel unreliable because the tail dominates user trust. Shared resources such as disks, caches, and thread pools often look fine at low load and then degrade sharply once contention begins. That is why latency tables are a design aid, not a performance guarantee.

Another common mistake is mixing throughput thinking with latency thinking. Batching can improve throughput but increase per item wait time. Compression can reduce network transfer cost but increase CPU work. Caching removes slow work on a hit and does nothing on a miss. Every performance technique shifts costs somewhere.

So which latency numbers should you know? Know enough to recognise scale changes instantly. Local memory is not the network. Same region is not cross continent. Median latency is not user experience if the tail is bad. Those instincts are more valuable than a memorised chart because they help you predict where the expensive boundary is before production teaches the lesson the hard way.