Cache Design Considerations

I only add a cache after deciding what kind of inconsistency the system can tolerate. That is the real design question. Lower latency is the reward, but the cost is another copy of data with its own eviction rules, failure modes, and coherence problems.

1. Make sure the workload is actually cacheable

A cache works best when many requests ask for the same data, backend access is expensive relative to memory access, and stale reads are acceptable for a short period. It fits badly when requests are mostly one-off, values change constantly, or read-after-write semantics are strict. In those cases the cache becomes an extra hop without improving tail latency.

Before introducing a cache, I want simple evidence:

• repeated access to the same keys
• backend latency high enough that a hit matters
• object sizes small enough for a useful in-memory working set
• a defined staleness budget for each key space

2. Choose the cache pattern from the consistency model

Different cache patterns move complexity to different layers.

Cache-aside

The application reads from cache first, then falls back to the database on a miss and populates the result. This is flexible and common, but invalidation lives in application code.

Read-through

The application reads from the cache layer, which loads misses from storage. This centralises miss handling, but can hide backend load if miss amplification is not monitored.

Write-through

The application writes to cache and backing store synchronously. This reduces stale reads for recently written data, but the write path is now more expensive.

Write-around

Writes go directly to the database and bypass the cache. This avoids polluting memory with write-heavy keys, but the first read after a write will miss by design.

Write-back

Writes are acknowledged once the cache accepts them and are flushed later. This gives excellent write latency, but it turns the cache into part of the durability story.

3. Design keys, TTLs, and invalidation before launch

Most cache bugs are not caused by lookup code. They come from vague key design, oversized values, or invalidation paths that were never fully mapped.

Good keys are deterministic, versioned when schema changes matter, and scoped tightly enough to avoid accidental collisions. Good TTLs are tied to business freshness, not guesswork. A stock price, a user profile, and a feature flag should not share the same expiry policy just because one cache client makes that easy.

Invalidation is harder than population. If an object is derived from several tables, events, or upstream services, every mutation path must either refresh or invalidate the same key. Miss one path and stale data can survive until TTL expiry.

I also add jitter to TTLs for hot keys. If thousands of popular entries expire at the same second, the backend sees a coordinated miss storm.

4. Expect contention and failure under real traffic

Cold start, thundering herd, and hot key imbalance are normal production problems.

When the cache is empty after a deploy or failover, many callers may race to rebuild the same object. The usual mitigations are request coalescing, single-flight loading per key, background warming for known hot sets, and admission control on misses. Hot keys create a different problem. One popular object can dominate traffic and saturate a cache shard. A more subtle failure mode is dependency inversion, where the database can no longer carry direct traffic when the cache cluster has trouble.

5. Measure the cache as a system

Hit ratio alone is not enough. A high hit rate can still hide poor user experience if hits are slow or the remaining misses are extremely expensive.

I watch hit and miss latency, hit ratio by key space, backend load before and after cache introduction, memory usage, eviction rate, and stale-read incidents.

The final rule is simple. A cache is a performance optimisation with explicit correctness boundaries. If the team cannot describe those boundaries in plain language, the cache is not ready for production.