← Back to Labs

KV Cache & FlashAttention Optimization

Step through memory bandwidth bottlenecks, KV cache VRAM footprint, vLLM PagedAttention virtual paging, and FlashAttention SRAM tiling

HBM Read: 16384 MB/tokSRAM Tile: 128 KBVRAM Total: 16.0 GBWaste/Frag: 0%GPU HBM (2.0 TB/s)Model Weights (16 GB)KV Cache MemoryEmpty (Step 0)Activation BuffersLow Intensity (O(1))GPU Compute Cores & SRAM (20 TB/s)Tensor Cores (312 TFLOPS)Waiting on HBM memory transfer overhead...Autoregressive Decoding LoopToken t₁ → Read 16GB → Gen t₂ → Read 16.5GB → Gen t₃Memory Bottleneck: Compute utilization < 5%
STEP 1 OF 6

Autoregressive Token Generation Bottleneck

LLM token generation is strictly memory-bandwidth bound. For every single token generated, the model must stream all weight tensors (e.g. 16 GB for an 8B FP16 model) from GPU HBM into fast SRAM.

Arrow keys to navigate · R to reset

Tap dots to jump to any step

Read the full article →Take the quiz →